mirror of
https://github.com/docling-project/docling-eval.git
synced 2026-05-17 13:10:47 +00:00
Christoph Auer
629a451d7b
* Misc fixes Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Make DatasetRecord tolerant to old parquet files Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Make DatasetRecord tolerant to old parquet files (2) Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fix docvqa test, more cleanup Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Important fixes for layout mAP computation Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Adding modes for missing_prediction_strategy and label_filtering_strategy Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fixes for mismatched docs Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Add F1 no_picture metrics to layout evaluator Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fixed commands on all READMEs Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Remove extract_images ambiguity, use utility and fix errors on visualizer Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Upgrade to latest docling_core Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fix ocrmac dep, upgrade uv.lock Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fix for tableformer provider Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Remove code redundancy Signed-off-by: Christoph Auer <cau@zurich.ibm.com> --------- Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
1.4 KiB
1.4 KiB
Pub1M Benchmarks
Create Pub1M evaluation datasets. This downloads from Huggingface the Pub1M_OTSL and runs the evaluations for TableFormer using the first 1000 samples.
# Make the ground-truth
docling-eval create-gt --benchmark Pub1M --output-dir ./benchmarks/Pub1M/
# Make predictions for tables.
docling-eval create-eval \
--benchmark DPBench \
--output-dir ./benchmarks/Pub1M/ \
--end-index 1000 \
--prediction-provider TableFormer # use tableformer predictions only
Tableformer Evaluation
Create the evaluation report:
docling-eval evaluate \
--modality table_structure \
--benchmark Pub1M \
--output-dir ./benchmarks/Pub1M/
Visualize the report:
docling-eval visualize \
--modality table_structure \
--benchmark Pub1M \
--output-dir ./benchmarks/Pub1M/
