docling-eval/docs/DocLayNetv1_benchmarks.md

DocLayNet V1.2 Benchmarks

Create and evaluate DocLayNetv1.2 datasets using the following commands. This downloads the DocLayNetv1.2_OTSL from HuggingFace and runs the evaluations using the PDF Docling converter for all supported modalities.

Create evaluation datasets

# Make the ground-truth
docling-eval create-gt --benchmark DocLayNetV1 --output-dir ./benchmarks/DocLayNetV1/ 

# Make predictions for different modalities.
docling-eval create-eval \
  --benchmark DocLayNetV1 \
  --output-dir ./benchmarks/DocLayNetV1/ \
  --prediction-provider Docling # use full-document predictions from docling
  
docling-eval create-eval \
  --benchmark DPBench \
  --output-dir ./benchmarks/DocLayNetV1/ \
  --prediction-provider TableFormer # use tableformer predictions only

Layout Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality layout \
  --benchmark DocLayNetV1 \
  --output-dir ./benchmarks/DocLayNetV1/ 

Layout evaluation json

Visualize the report:

docling-eval visualize \
  --modality layout \
  --benchmark DocLayNetV1 \
  --output-dir ./benchmarks/DocLayNetV1/ 

mAP[0.5:0.95] report

Markdown text Evaluation

Create the report:

docling-eval evaluate \
  --modality markdown_text \
  --benchmark DocLayNetV1 \
  --output-dir ./benchmarks/DocLayNetV1/ 

Markdown text json

Visualize the report:

docling-eval visualize \
  --modality markdown_text \
  --benchmark DocLayNetV1 \
  --output-dir ./benchmarks/DocLayNetV1/ 

Markdown text report