mirror of
https://github.com/docling-project/docling-eval.git
synced 2026-05-17 13:10:47 +00:00
8a6d2fce9d
* correct mpy Signed-off-by: Peter Staar <taa@zurich.ibm.com> * reformatting Signed-off-by: Peter Staar <taa@zurich.ibm.com> * adding the script to make an initial dataset from pdf's Signed-off-by: Peter Staar <taa@zurich.ibm.com> * before switching to specific docling-core branch Signed-off-by: Peter Staar <taa@zurich.ibm.com> * rebased on kv-items and updated the create script in CVAT Signed-off-by: Peter Staar <taa@zurich.ibm.com> * fixed the cvat Signed-off-by: Peter Staar <taa@zurich.ibm.com> * added the annotation description on CVAT Signed-off-by: Peter Staar <taa@zurich.ibm.com> * added the annotation description on CVAT (2) Signed-off-by: Peter Staar <taa@zurich.ibm.com> * added the annotation description on CVAT (3) Signed-off-by: Peter Staar <taa@zurich.ibm.com> * feat: Add new reading-order model evaluator, re-factor to BaseReadingOrderEvaluator Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Rename New Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix: Fix calling DoclingDocument methods. Pin latest versions of docling and docling-core. Remove commented out code, remove unused imports, code formatting. Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * docs: Re-run the evaluations for DPBench and update the docs Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * docs: Re-run the evaluations for OmniDocBench and update the docs Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * docs: Re-run the evaluations for DPBench and update the docs Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * docs: Re-run the evaluations for OmniDocBench and update the docs Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * docs: Minor improvements in DPBench OmniDocBench documentation Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> --------- Signed-off-by: Peter Staar <taa@zurich.ibm.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> Co-authored-by: Peter Staar <taa@zurich.ibm.com> Co-authored-by: Nikos Livathinos <nli@zurich.ibm.com>
159 lines
9.2 KiB
Plaintext
159 lines
9.2 KiB
Plaintext
DPBench size: 200
|
|
|
|
DPBench markdown_text BLEU: mean=0.88 median=0.93 std=0.16
|
|
|
|
| BLEU | prob [%] | acc [%] | 1-acc [%] | total |
|
|
|----------------|------------|-----------|-------------|---------|
|
|
| (0.000, 0.050] | 1 | 0 | 100 | 2 |
|
|
| (0.050, 0.100] | 0 | 1 | 99 | 0 |
|
|
| (0.100, 0.150] | 0 | 1 | 99 | 0 |
|
|
| (0.150, 0.200] | 0 | 1 | 99 | 0 |
|
|
| (0.200, 0.250] | 0 | 1 | 99 | 0 |
|
|
| (0.250, 0.300] | 0 | 1 | 99 | 0 |
|
|
| (0.300, 0.350] | 1 | 1 | 99 | 2 |
|
|
| (0.350, 0.400] | 0.5 | 2 | 98 | 1 |
|
|
| (0.400, 0.450] | 0 | 2.5 | 97.5 | 0 |
|
|
| (0.450, 0.500] | 1 | 2.5 | 97.5 | 2 |
|
|
| (0.500, 0.550] | 1 | 3.5 | 96.5 | 2 |
|
|
| (0.550, 0.600] | 0.5 | 4.5 | 95.5 | 1 |
|
|
| (0.600, 0.650] | 2 | 5 | 95 | 4 |
|
|
| (0.650, 0.700] | 1.5 | 7 | 93 | 3 |
|
|
| (0.700, 0.750] | 4.5 | 8.5 | 91.5 | 9 |
|
|
| (0.750, 0.800] | 3.5 | 13 | 87 | 7 |
|
|
| (0.800, 0.850] | 8 | 16.5 | 83.5 | 16 |
|
|
| (0.850, 0.900] | 14.5 | 24.5 | 75.5 | 29 |
|
|
| (0.900, 0.950] | 21 | 39 | 61 | 42 |
|
|
| (0.950, 1.000] | 40 | 60 | 40 | 80 |
|
|
|
|
|
|
DPBench markdown_text F1: mean=0.96 median=0.98 std=0.07
|
|
|
|
| F1 | prob [%] | acc [%] | 1-acc [%] | total |
|
|
|----------------|------------|-----------|-------------|---------|
|
|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
|
|
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
|
|
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
|
|
| (0.150, 0.200] | 0.5 | 0 | 100 | 1 |
|
|
| (0.200, 0.250] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.250, 0.300] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.300, 0.350] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.350, 0.400] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.400, 0.450] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.450, 0.500] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.500, 0.550] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.550, 0.600] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.600, 0.650] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.650, 0.700] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.700, 0.750] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.750, 0.800] | 0.5 | 0.5 | 99.5 | 1 |
|
|
| (0.800, 0.850] | 1.5 | 1 | 99 | 3 |
|
|
| (0.850, 0.900] | 7 | 2.5 | 97.5 | 14 |
|
|
| (0.900, 0.950] | 24 | 9.5 | 90.5 | 48 |
|
|
| (0.950, 1.000] | 66.5 | 33.5 | 66.5 | 133 |
|
|
|
|
|
|
DPBench markdown_text precision: mean=0.97 median=0.98 std=0.04
|
|
|
|
| precision | prob [%] | acc [%] | 1-acc [%] | total |
|
|
|----------------|------------|-----------|-------------|---------|
|
|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
|
|
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
|
|
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
|
|
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
|
|
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
|
|
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
|
|
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
|
|
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
|
|
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
|
|
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
|
|
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
|
|
| (0.550, 0.600] | 0 | 0 | 100 | 0 |
|
|
| (0.600, 0.650] | 0 | 0 | 100 | 0 |
|
|
| (0.650, 0.700] | 0 | 0 | 100 | 0 |
|
|
| (0.700, 0.750] | 0 | 0 | 100 | 0 |
|
|
| (0.750, 0.800] | 0 | 0 | 100 | 0 |
|
|
| (0.800, 0.850] | 1 | 0 | 100 | 2 |
|
|
| (0.850, 0.900] | 5.5 | 1 | 99 | 11 |
|
|
| (0.900, 0.950] | 16 | 6.5 | 93.5 | 32 |
|
|
| (0.950, 1.000] | 77.5 | 22.5 | 77.5 | 155 |
|
|
|
|
|
|
DPBench markdown_text recall: mean=0.95 median=0.98 std=0.08
|
|
|
|
| recall | prob [%] | acc [%] | 1-acc [%] | total |
|
|
|----------------|------------|-----------|-------------|---------|
|
|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
|
|
| (0.050, 0.100] | 0.5 | 0 | 100 | 1 |
|
|
| (0.100, 0.150] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.150, 0.200] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.200, 0.250] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.250, 0.300] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.300, 0.350] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.350, 0.400] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.400, 0.450] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.450, 0.500] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.500, 0.550] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.550, 0.600] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.600, 0.650] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.650, 0.700] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.700, 0.750] | 0.5 | 0.5 | 99.5 | 1 |
|
|
| (0.750, 0.800] | 2.5 | 1 | 99 | 5 |
|
|
| (0.800, 0.850] | 4 | 3.5 | 96.5 | 8 |
|
|
| (0.850, 0.900] | 10.5 | 7.5 | 92.5 | 21 |
|
|
| (0.900, 0.950] | 19.5 | 18 | 82 | 39 |
|
|
| (0.950, 1.000] | 62.5 | 37.5 | 62.5 | 125 |
|
|
|
|
|
|
DPBench markdown_text edit_distance: mean=0.10 median=0.05 std=0.14
|
|
|
|
| edit_distance | prob [%] | acc [%] | 1-acc [%] | total |
|
|
|-----------------|------------|-----------|-------------|---------|
|
|
| (0.000, 0.050] | 49 | 0 | 100 | 98 |
|
|
| (0.050, 0.100] | 17 | 49 | 51 | 34 |
|
|
| (0.100, 0.150] | 12.5 | 66 | 34 | 25 |
|
|
| (0.150, 0.200] | 5 | 78.5 | 21.5 | 10 |
|
|
| (0.200, 0.250] | 5 | 83.5 | 16.5 | 10 |
|
|
| (0.250, 0.300] | 4.5 | 88.5 | 11.5 | 9 |
|
|
| (0.300, 0.350] | 1.5 | 93 | 7 | 3 |
|
|
| (0.350, 0.400] | 2 | 94.5 | 5.5 | 4 |
|
|
| (0.400, 0.450] | 0 | 96.5 | 3.5 | 0 |
|
|
| (0.450, 0.500] | 2 | 96.5 | 3.5 | 4 |
|
|
| (0.500, 0.550] | 0 | 98.5 | 1.5 | 0 |
|
|
| (0.550, 0.600] | 0 | 98.5 | 1.5 | 0 |
|
|
| (0.600, 0.650] | 0.5 | 98.5 | 1.5 | 1 |
|
|
| (0.650, 0.700] | 0 | 99 | 1 | 0 |
|
|
| (0.700, 0.750] | 0 | 99 | 1 | 0 |
|
|
| (0.750, 0.800] | 0 | 99 | 1 | 0 |
|
|
| (0.800, 0.850] | 0.5 | 99 | 1 | 1 |
|
|
| (0.850, 0.900] | 0 | 99.5 | 0.5 | 0 |
|
|
| (0.900, 0.950] | 0 | 99.5 | 0.5 | 0 |
|
|
| (0.950, 1.000] | 0.5 | 99.5 | 0.5 | 1 |
|
|
|
|
|
|
DPBench markdown_text meteor: mean=0.94 median=0.98 std=0.11
|
|
|
|
| meteor | prob [%] | acc [%] | 1-acc [%] | total |
|
|
|----------------|------------|-----------|-------------|---------|
|
|
| (0.000, 0.050] | 0.5 | 0 | 100 | 1 |
|
|
| (0.050, 0.100] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.100, 0.150] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.150, 0.200] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.200, 0.250] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.250, 0.300] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.300, 0.350] | 0 | 0.5 | 99.5 | 0 |
|
|
| (0.350, 0.400] | 0.5 | 0.5 | 99.5 | 1 |
|
|
| (0.400, 0.450] | 0 | 1 | 99 | 0 |
|
|
| (0.450, 0.500] | 0 | 1 | 99 | 0 |
|
|
| (0.500, 0.550] | 0 | 1 | 99 | 0 |
|
|
| (0.550, 0.600] | 1 | 1 | 99 | 2 |
|
|
| (0.600, 0.650] | 0.5 | 2 | 98 | 1 |
|
|
| (0.650, 0.700] | 2 | 2.5 | 97.5 | 4 |
|
|
| (0.700, 0.750] | 2.5 | 4.5 | 95.5 | 5 |
|
|
| (0.750, 0.800] | 1 | 7 | 93 | 2 |
|
|
| (0.800, 0.850] | 2 | 8 | 92 | 4 |
|
|
| (0.850, 0.900] | 6.5 | 10 | 90 | 13 |
|
|
| (0.900, 0.950] | 16 | 16.5 | 83.5 | 32 |
|
|
| (0.950, 1.000] | 67.5 | 32.5 | 67.5 | 135 |
|
|
|
|
|