docling-eval/docs/Docling-DP-Bench_benchmarks.md

Docling-DP-Bench Benchmarks

Docling-DP-Bench on HuggingFace Docling-DP-Bench is a re-annotated version of the original upstage/dpbench dataset with Docling-native labels.

Create Docling-DPBench evaluation datasets:

# Download the GT straight from HuggingFace
huggingface-cli download --repo-type dataset --local-dir ./benchmarks/Docling-DPBench-gt/gt_dataset ds4sd/docling-dpbench
# Make predictions for different modalities.
docling-eval create-eval \
  --benchmark DPBench \
  --gt-dir ./benchmarks/Docling-DPBench-gt/gt_dataset/ \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ \
  --prediction-provider Docling # use full-document predictions from docling
  
docling-eval create-eval \
  --benchmark DPBench \
  --gt-dir ./benchmarks/Docling-DPBench-gt/gt_dataset/ \
  --output-dir ./benchmarks/Docling-DPBench-tables/ \
  --prediction-provider TableFormer # use tableformer predictions only

Layout Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality layout \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ 

Layout evaluation json

Visualize the report:

docling-eval visualize \
  --modality layout \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ 

mAP[0.5:0.95] report

TableFormer Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality table_structure \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-tables/ 

Visualize the report:

Tableformer evaluation json

Visualize the report:

docling-eval visualize \
  --modality table_structure \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-tables/ 

TEDS struct only report

TEDS struct with text report

Reading order Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality reading_order \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ 

Reading order json

Visualize the report:

docling-eval visualize \
  --modality reading_order \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ 

ARD report

Weighted ARD report

Markdown text Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality markdown_text \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ 

Markdown text json

Visualize the report:

docling-eval visualize \
  --modality markdown_text \
  --benchmark DPBench \
  --output-dir ./benchmarks/Docling-DPBench-e2e/ 

Markdown text report