Add README for Docling-DPBench (#60)

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
This commit is contained in:
Christoph Auer
2025-04-07 14:50:14 +02:00
committed by GitHub
parent ea8c41f1b2
commit ddf40241a9
26 changed files with 17733 additions and 4 deletions
+1 -1
View File
@@ -257,7 +257,7 @@ def get_prediction_provider(
elif provider_type == PredictionProviderType.TABLEFORMER:
return TableFormerPredictionProvider(
do_visualization=True,
do_visualization=False,
ignore_missing_predictions=True,
)
+1 -1
View File
@@ -36,7 +36,7 @@ class DatasetRecord(
alias="GroundTruthPictures", default=[]
)
mime_type: str = Field(default="")
mime_type: str = Field(default="application/pdf")
modalities: List[EvaluationModality] = Field(default=[])
model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True)
+2 -2
View File
@@ -12,14 +12,14 @@ docling_eval create-gt --benchmark DPBench --output-dir ./benchmarks/DPBench-gt/
docling_eval create-eval \
--modality end-to-end \
--benchmark DPBench \
--gt-dir ./benchmarks/DPBench-gt/ \
--gt-dir ./benchmarks/DPBench-gt/gt_dataset/ \
--output-dir ./benchmarks/DPBench-e2e/ \
--prediction-provider docling # use full-document predictions from docling
docling_eval create-eval \
--modality table_structure \
--benchmark DPBench \
--gt-dir ./benchmarks/DPBench-gt/ \
--gt-dir ./benchmarks/DPBench-gt/gt_dataset/ \
--output-dir ./benchmarks/DPBench-tables/ \
--prediction-provider tableformer # use tableformer predictions only
```
+159
View File
@@ -0,0 +1,159 @@
# Docling-DP-Bench Benchmarks
[Docling-DP-Bench on HuggingFace](https://huggingface.co/datasets/ds4sd/docling-dpbench)
Docling-DP-Bench is a re-annotated version of the original `upstage/dpbench` dataset with Docling-native labels.
Create Docling-DPBench evaluation datasets:
```sh
# Download the GT straight from HuggingFace
huggingface-cli download --repo-type dataset --local-dir ./benchmarks/Docling-DPBench-gt/gt_dataset ds4sd/docling-dpbench
# Make predictions for different modalities.
docling_eval create-eval \
--modality end-to-end \
--benchmark DPBench \
--gt-dir ./benchmarks/Docling-DPBench-gt/gt_dataset/ \
--output-dir ./benchmarks/Docling-DPBench-e2e/ \
--prediction-provider docling # use full-document predictions from docling
docling_eval create-eval \
--modality table_structure \
--benchmark DPBench \
--gt-dir ./benchmarks/Docling-DPBench-gt/gt_dataset/ \
--output-dir ./benchmarks/Docling-DPBench-tables/ \
--prediction-provider tableformer # use tableformer predictions only
```
## Layout Evaluation
Create the evaluation report:
```sh
docling_eval evaluate \
--modality layout \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-e2e/
```
[Layout evaluation json](evaluations/Docling-DPBench/evaluation_DPBench_layout.json)
Visualize the report:
```sh
docling_eval visualize \
--modality layout \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-e2e/
```
[mAP[0.5:0.95] report](evaluations/Docling-DPBench/evaluation_DPBench_layout_mAP_0.5_0.95.txt)
![mAP[0.5:0.95] plot](evaluations/Docling-DPBench/evaluation_DPBench_layout_mAP_0.5_0.95.png)
## TableFormer Evaluation
Create the evaluation report:
```sh
docling_eval evaluate \
--modality table_structure \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-tables/
```
Visualize the report:
[Tableformer evaluation json](evaluations/Docling-DPBench/evaluation_DPBench_tableformer.json)
Visualize the report:
```sh
docling_eval visualize \
--modality table_structure \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-tables/
```
![TEDS plot](evaluations/Docling-DPBench/evaluation_DPBench_tableformer-delta_row_col.png)
![TEDS struct only plot](evaluations/Docling-DPBench/evaluation_DPBench_table_structure_TEDS_struct-only.png)
[TEDS struct only report](evaluations/Docling-DPBench/evaluation_DPBench_table_structure_TEDS_struct-only.txt)
![TEDS struct with text plot](evaluations/Docling-DPBench/evaluation_DPBench_table_structure_TEDS_struct-with-text.png)
[TEDS struct with text report](evaluations/Docling-DPBench/evaluation_DPBench_table_structure_TEDS_struct-with-text.txt)
## Reading order Evaluation
Create the evaluation report:
```sh
docling_eval evaluate \
--modality reading_order \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-e2e/
```
[Reading order json](evaluations/Docling-DPBench/evaluation_DPBench_reading_order.json)
Visualize the report:
```sh
docling_eval visualize \
--modality reading_order \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-e2e/
```
![ARD plot](evaluations/Docling-DPBench/evaluation_DPBench_reading_order_ARD_norm.png)
[ARD report](evaluations/Docling-DPBench/evaluation_DPBench_reading_order_ARD_norm.txt)
![Weighted ARD plot](evaluations/Docling-DPBench/evaluation_DPBench_reading_order_weighted_ARD.png)
[Weighted ARD report](evaluations/Docling-DPBench/evaluation_DPBench_reading_order_weighted_ARD.txt)
## Markdown text Evaluation
Create the evaluation report:
```sh
docling_eval evaluate \
--modality markdown_text \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-e2e/
```
[Markdown text json](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text.json)
Visualize the report:
```sh
docling_eval visualize \
--modality markdown_text \
--benchmark DPBench \
--output-dir ./benchmarks/Docling-DPBench-e2e/
```
[Markdown text report](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text.txt)
![BLEU plot](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text_BLEU.png)
![Edit distance plot](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text_edit_distance.png)
![F1 plot](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text_F1.png)
![Meteor plot](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text_meteor.png)
![Precision plot](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text_precision.png)
![Recall plot](evaluations/Docling-DPBench/evaluation_DPBench_markdown_text_recall.png)
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

@@ -0,0 +1,44 @@
DPBench layout mAP_0.5_0.95: mean=0.70 median=0.67 std=0.21
| mAP_0.5_0.95 | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0.5 | 0 | 100 | 1 |
| (0.200, 0.250] | 0 | 0.5 | 99.5 | 0 |
| (0.250, 0.300] | 0.5 | 0.5 | 99.5 | 1 |
| (0.300, 0.350] | 2.5 | 1 | 99 | 5 |
| (0.350, 0.400] | 1.5 | 3.5 | 96.5 | 3 |
| (0.400, 0.450] | 6.5 | 5 | 95 | 13 |
| (0.450, 0.500] | 4.5 | 11.5 | 88.5 | 9 |
| (0.500, 0.550] | 11.5 | 16 | 84 | 23 |
| (0.550, 0.600] | 7 | 27.5 | 72.5 | 14 |
| (0.600, 0.650] | 9 | 34.5 | 65.5 | 18 |
| (0.650, 0.700] | 10 | 43.5 | 56.5 | 20 |
| (0.700, 0.750] | 4 | 53.5 | 46.5 | 8 |
| (0.750, 0.800] | 7 | 57.5 | 42.5 | 14 |
| (0.800, 0.850] | 5.5 | 64.5 | 35.5 | 11 |
| (0.850, 0.900] | 7 | 70 | 30 | 14 |
| (0.900, 0.950] | 4 | 77 | 23 | 8 |
| (0.950, 1.000] | 19 | 81 | 19 | 38 |
AP[0.5:0.05:0.95] per class (reported as %):
| label | Class mAP[0.5:0.95] |
|----------------|-----------------------|
| document_index | 100 |
| picture | 92.58 |
| table | 90.55 |
| text | 68.87 |
| list_item | 52.87 |
| formula | 51.32 |
| section_header | 47.21 |
| footnote | 37.72 |
| caption | 14.43 |
Total mAP[0.5:0.05:0.95] (reported as %): 61.73
File diff suppressed because one or more lines are too long
@@ -0,0 +1,158 @@
DPBench size: 200
DPBench markdown_text BLEU: mean=0.92 median=0.96 std=0.13
| BLEU | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0.5 | 0 | 100 | 1 |
| (0.050, 0.100] | 0 | 0.5 | 99.5 | 0 |
| (0.100, 0.150] | 0 | 0.5 | 99.5 | 0 |
| (0.150, 0.200] | 0.5 | 0.5 | 99.5 | 1 |
| (0.200, 0.250] | 0 | 1 | 99 | 0 |
| (0.250, 0.300] | 0 | 1 | 99 | 0 |
| (0.300, 0.350] | 0 | 1 | 99 | 0 |
| (0.350, 0.400] | 0 | 1 | 99 | 0 |
| (0.400, 0.450] | 0.5 | 1 | 99 | 1 |
| (0.450, 0.500] | 0.5 | 1.5 | 98.5 | 1 |
| (0.500, 0.550] | 0.5 | 2 | 98 | 1 |
| (0.550, 0.600] | 0.5 | 2.5 | 97.5 | 1 |
| (0.600, 0.650] | 0.5 | 3 | 97 | 1 |
| (0.650, 0.700] | 2 | 3.5 | 96.5 | 4 |
| (0.700, 0.750] | 0.5 | 5.5 | 94.5 | 1 |
| (0.750, 0.800] | 3 | 6 | 94 | 6 |
| (0.800, 0.850] | 3.5 | 9 | 91 | 7 |
| (0.850, 0.900] | 11 | 12.5 | 87.5 | 22 |
| (0.900, 0.950] | 21.5 | 23.5 | 76.5 | 43 |
| (0.950, 1.000] | 55 | 45 | 55 | 110 |
DPBench markdown_text F1: mean=0.97 median=0.99 std=0.03
| F1 | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
| (0.550, 0.600] | 0 | 0 | 100 | 0 |
| (0.600, 0.650] | 0 | 0 | 100 | 0 |
| (0.650, 0.700] | 0 | 0 | 100 | 0 |
| (0.700, 0.750] | 0 | 0 | 100 | 0 |
| (0.750, 0.800] | 0.5 | 0 | 100 | 1 |
| (0.800, 0.850] | 0.5 | 0.5 | 99.5 | 1 |
| (0.850, 0.900] | 1 | 1 | 99 | 2 |
| (0.900, 0.950] | 19 | 2 | 98 | 38 |
| (0.950, 1.000] | 79 | 21 | 79 | 158 |
DPBench markdown_text precision: mean=0.97 median=0.99 std=0.04
| precision | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
| (0.550, 0.600] | 0 | 0 | 100 | 0 |
| (0.600, 0.650] | 0.5 | 0 | 100 | 1 |
| (0.650, 0.700] | 0 | 0.5 | 99.5 | 0 |
| (0.700, 0.750] | 0.5 | 0.5 | 99.5 | 1 |
| (0.750, 0.800] | 0 | 1 | 99 | 0 |
| (0.800, 0.850] | 1 | 1 | 99 | 2 |
| (0.850, 0.900] | 2 | 2 | 98 | 4 |
| (0.900, 0.950] | 7.5 | 4 | 96 | 15 |
| (0.950, 1.000] | 88.5 | 11.5 | 88.5 | 177 |
DPBench markdown_text recall: mean=0.98 median=0.99 std=0.04
| recall | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
| (0.550, 0.600] | 0 | 0 | 100 | 0 |
| (0.600, 0.650] | 0 | 0 | 100 | 0 |
| (0.650, 0.700] | 0 | 0 | 100 | 0 |
| (0.700, 0.750] | 0 | 0 | 100 | 0 |
| (0.750, 0.800] | 0 | 0 | 100 | 0 |
| (0.800, 0.850] | 0 | 0 | 100 | 0 |
| (0.850, 0.900] | 5.5 | 0 | 100 | 11 |
| (0.900, 0.950] | 13 | 5.5 | 94.5 | 26 |
| (0.950, 1.000] | 81.5 | 18.5 | 81.5 | 163 |
DPBench markdown_text edit_distance: mean=0.07 median=0.03 std=0.10
| edit_distance | prob [%] | acc [%] | 1-acc [%] | total |
|-----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 62.5 | 0 | 100 | 125 |
| (0.050, 0.100] | 15 | 62.5 | 37.5 | 30 |
| (0.100, 0.150] | 8.5 | 77.5 | 22.5 | 17 |
| (0.150, 0.200] | 2.5 | 86 | 14 | 5 |
| (0.200, 0.250] | 4 | 88.5 | 11.5 | 8 |
| (0.250, 0.300] | 2 | 92.5 | 7.5 | 4 |
| (0.300, 0.350] | 1.5 | 94.5 | 5.5 | 3 |
| (0.350, 0.400] | 1 | 96 | 4 | 2 |
| (0.400, 0.450] | 1 | 97 | 3 | 2 |
| (0.450, 0.500] | 2 | 98 | 2 | 4 |
| (0.500, 0.550] | 0 | 100 | 0 | 0 |
| (0.550, 0.600] | 0 | 100 | 0 | 0 |
| (0.600, 0.650] | 0 | 100 | 0 | 0 |
| (0.650, 0.700] | 0 | 100 | 0 | 0 |
| (0.700, 0.750] | 0 | 100 | 0 | 0 |
| (0.750, 0.800] | 0 | 100 | 0 | 0 |
| (0.800, 0.850] | 0 | 100 | 0 | 0 |
| (0.850, 0.900] | 0 | 100 | 0 | 0 |
| (0.900, 0.950] | 0 | 100 | 0 | 0 |
| (0.950, 1.000] | 0 | 100 | 0 | 0 |
DPBench markdown_text meteor: mean=0.97 median=0.99 std=0.06
| meteor | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
| (0.550, 0.600] | 0.5 | 0 | 100 | 1 |
| (0.600, 0.650] | 0 | 0.5 | 99.5 | 0 |
| (0.650, 0.700] | 0.5 | 0.5 | 99.5 | 1 |
| (0.700, 0.750] | 1 | 1 | 99 | 2 |
| (0.750, 0.800] | 0 | 2 | 98 | 0 |
| (0.800, 0.850] | 1.5 | 2 | 98 | 3 |
| (0.850, 0.900] | 5.5 | 3.5 | 96.5 | 11 |
| (0.900, 0.950] | 9 | 9 | 91 | 18 |
| (0.950, 1.000] | 82 | 18 | 82 | 164 |
Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

@@ -0,0 +1,26 @@
DPBench reading_order ARD_norm: mean=0.98 median=1.00 std=0.04
| ARD_norm | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
| (0.550, 0.600] | 0 | 0 | 100 | 0 |
| (0.600, 0.650] | 0 | 0 | 100 | 0 |
| (0.650, 0.700] | 0 | 0 | 100 | 0 |
| (0.700, 0.750] | 0 | 0 | 100 | 0 |
| (0.750, 0.800] | 0.57 | 0 | 100 | 1 |
| (0.800, 0.850] | 1.14 | 0.57 | 99.43 | 2 |
| (0.850, 0.900] | 4 | 1.71 | 98.29 | 7 |
| (0.900, 0.950] | 7.43 | 5.71 | 94.29 | 13 |
| (0.950, 1.000] | 86.86 | 13.14 | 86.86 | 152 |
Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

@@ -0,0 +1,26 @@
DPBench reading_order weighted_ARD: mean=1.00 median=1.00 std=0.01
| weighted_ARD | prob [%] | acc [%] | 1-acc [%] | total |
|----------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 0 | 0 | 100 | 0 |
| (0.450, 0.500] | 0 | 0 | 100 | 0 |
| (0.500, 0.550] | 0 | 0 | 100 | 0 |
| (0.550, 0.600] | 0 | 0 | 100 | 0 |
| (0.600, 0.650] | 0 | 0 | 100 | 0 |
| (0.650, 0.700] | 0 | 0 | 100 | 0 |
| (0.700, 0.750] | 0 | 0 | 100 | 0 |
| (0.750, 0.800] | 0 | 0 | 100 | 0 |
| (0.800, 0.850] | 0 | 0 | 100 | 0 |
| (0.850, 0.900] | 0 | 0 | 100 | 0 |
| (0.900, 0.950] | 1.71 | 0 | 100 | 3 |
| (0.950, 1.000] | 98.29 | 1.71 | 98.29 | 172 |
Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

@@ -0,0 +1,718 @@
{
"TEDS": {
"bins": [
0.0,
0.05,
0.1,
0.15000000000000002,
0.2,
0.25,
0.30000000000000004,
0.35000000000000003,
0.4,
0.45,
0.5,
0.55,
0.6000000000000001,
0.65,
0.7000000000000001,
0.75,
0.8,
0.8500000000000001,
0.9,
0.9500000000000001,
1.0
],
"hist": [
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
1.0,
0.0,
1.0,
1.0,
1.0,
0.0,
6.0,
6.0,
7.0,
15.0,
8.0,
5.0
],
"mean": 0.8315098039215686,
"median": 0.861,
"std": 0.11816401695085008,
"total": 51
},
"TEDS_complex": {
"bins": [
0.0,
0.05,
0.1,
0.15000000000000002,
0.2,
0.25,
0.30000000000000004,
0.35000000000000003,
0.4,
0.45,
0.5,
0.55,
0.6000000000000001,
0.65,
0.7000000000000001,
0.75,
0.8,
0.8500000000000001,
0.9,
0.9500000000000001,
1.0
],
"hist": [
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
1.0,
0.0,
0.0,
0.0,
2.0,
3.0,
0.0,
1.0
],
"mean": 0.8462857142857143,
"median": 0.874,
"std": 0.11662291696705568,
"total": 7
},
"TEDS_simple": {
"bins": [
0.0,
0.05,
0.1,
0.15000000000000002,
0.2,
0.25,
0.30000000000000004,
0.35000000000000003,
0.4,
0.45,
0.5,
0.55,
0.6000000000000001,
0.65,
0.7000000000000001,
0.75,
0.8,
0.8500000000000001,
0.9,
0.9500000000000001,
1.0
],
"hist": [
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
1.0,
0.0,
1.0,
1.0,
0.0,
0.0,
6.0,
6.0,
5.0,
12.0,
8.0,
4.0
],
"mean": 0.8291590909090909,
"median": 0.859,
"std": 0.119568982363071,
"total": 44
},
"TEDS_struct": {
"bins": [
0.0,
0.05,
0.1,
0.15000000000000002,
0.2,
0.25,
0.30000000000000004,
0.35000000000000003,
0.4,
0.45,
0.5,
0.55,
0.6000000000000001,
0.65,
0.7000000000000001,
0.75,
0.8,
0.8500000000000001,
0.9,
0.9500000000000001,
1.0
],
"hist": [
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
1.0,
0.0,
0.0,
0.0,
1.0,
1.0,
4.0,
6.0,
7.0,
17.0,
9.0,
5.0
],
"mean": 0.8471176470588235,
"median": 0.871,
"std": 0.10569534465790319,
"total": 51
},
"evaluations": [
{
"TEDS": 0.919,
"filename": "doc_00f0adaaa8358a28b4b4e83bc97dcd83a01f7283605b140c2be8e8d47bba8b6b_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 9,
"table_id": 0,
"true_ncols": 3,
"true_nrows": 9
},
{
"TEDS": 0.874,
"filename": "doc_d276ca9a5ecb8d6d11359f515e50c8f78395548de4e3e2c49e38f5500ee40ebe_page_000001.png",
"is_complex": true,
"pred_ncols": 7,
"pred_nrows": 12,
"table_id": 0,
"true_ncols": 7,
"true_nrows": 12
},
{
"TEDS": 0.848,
"filename": "doc_934fbf534914863f6431eef38f5bf66fa91afd439ddf20fb1af0cf3225159ac1_page_000001.png",
"is_complex": true,
"pred_ncols": 7,
"pred_nrows": 10,
"table_id": 0,
"true_ncols": 7,
"true_nrows": 10
},
{
"TEDS": 0.913,
"filename": "doc_4682941b1a9a3ec96599d8188b673e8c6d1c4f2a555b5dc0d739c6c706815553_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 9,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 9
},
{
"TEDS": 0.932,
"filename": "doc_f4dea73d2b75fbc7d590e2085b14e9bee7e766da73e14f723a2580fb0f36c707_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 12,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 12
},
{
"TEDS": 0.933,
"filename": "doc_0ce1ded16152ad18c45203c1dcba0287235b888266fbcd112396fdf7cd094fab_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 9,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 9
},
{
"TEDS": 0.896,
"filename": "doc_bdf0e09acee7a07f7e60c6055128967b36b44564842983ad3b0f273f1c1c5914_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 12,
"table_id": 0,
"true_ncols": 3,
"true_nrows": 11
},
{
"TEDS": 0.893,
"filename": "doc_c4416f86c1db7fee5dccc1a10e85b1aade1cd77803f4ee626a090e8bc26aa9a9_page_000001.png",
"is_complex": true,
"pred_ncols": 7,
"pred_nrows": 10,
"table_id": 0,
"true_ncols": 7,
"true_nrows": 10
},
{
"TEDS": 0.769,
"filename": "doc_24ab9416df440de9d792a4034e177eb71933a6eed72d4cf1ead17d22a787702f_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 3,
"table_id": 0,
"true_ncols": 3,
"true_nrows": 3
},
{
"TEDS": 0.897,
"filename": "doc_24ab9416df440de9d792a4034e177eb71933a6eed72d4cf1ead17d22a787702f_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 7,
"table_id": 1,
"true_ncols": 3,
"true_nrows": 7
},
{
"TEDS": 0.889,
"filename": "doc_0ec2813813e8e7496d97aa4fa15dd414aa66cae2d6f62db818d30d9856ff0439_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 7,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 7
},
{
"TEDS": 0.871,
"filename": "doc_0ec2813813e8e7496d97aa4fa15dd414aa66cae2d6f62db818d30d9856ff0439_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 6,
"table_id": 1,
"true_ncols": 4,
"true_nrows": 6
},
{
"TEDS": 0.897,
"filename": "doc_970804c04c81dd9cc3ba80f0063b9551d290e83934c4e2c258e376845b3d180f_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 8,
"table_id": 0,
"true_ncols": 5,
"true_nrows": 8
},
{
"TEDS": 0.857,
"filename": "doc_970804c04c81dd9cc3ba80f0063b9551d290e83934c4e2c258e376845b3d180f_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 4,
"table_id": 1,
"true_ncols": 4,
"true_nrows": 4
},
{
"TEDS": 0.903,
"filename": "doc_970804c04c81dd9cc3ba80f0063b9551d290e83934c4e2c258e376845b3d180f_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 6,
"table_id": 2,
"true_ncols": 4,
"true_nrows": 6
},
{
"TEDS": 0.857,
"filename": "doc_39a0d95abd6dc977dfc7986b57fbbd9a8331af6b5b603e023f41dfcbc67003c7_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 4,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 4
},
{
"TEDS": 0.871,
"filename": "doc_39a0d95abd6dc977dfc7986b57fbbd9a8331af6b5b603e023f41dfcbc67003c7_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 6,
"table_id": 1,
"true_ncols": 4,
"true_nrows": 6
},
{
"TEDS": 0.865,
"filename": "doc_1039bce8b948fac48b0b9a42f8266700a104aca355c3681a544d65e6b136b8d5_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 5,
"true_nrows": 6
},
{
"TEDS": 0.861,
"filename": "doc_1cf54980192667fd7a6977b301a175e98be02b11651bbf78b6cae338858db2aa_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 5,
"true_nrows": 6
},
{
"TEDS": 0.884,
"filename": "doc_d1437ae1c6bedfa2bbf72e4d04dee3a01896dd045024438ee7c4e775e0ab6e9c_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 7,
"table_id": 0,
"true_ncols": 5,
"true_nrows": 7
},
{
"TEDS": 0.846,
"filename": "doc_a5528f4dd6f5ba0d52e6bf97a0aae99ede782fe97f4a0b4f96029e5ed7fe18b5_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 5,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 5
},
{
"TEDS": 0.571,
"filename": "doc_10f75857bb705a5441260399c3fca1080262cc0c271270825caf141cf7b31770_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 3,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 4
},
{
"TEDS": 0.92,
"filename": "doc_b9e5879b4dcd812e854cc06c6c3cb4f54231af28c919be63e27bef4c639551c9_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 3,
"true_nrows": 6
},
{
"TEDS": 0.734,
"filename": "doc_6916893bac49ba4fefa0a83eaf039b73c4adf00ed81ef97d4aad8f1a5cb90994_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 3,
"table_id": 0,
"true_ncols": 5,
"true_nrows": 3
},
{
"TEDS": 0.71,
"filename": "doc_f151ca1888000160dc6ce2e46c3438bc13f9a034200b1ec5d81b03c77606eed6_page_000001.png",
"is_complex": false,
"pred_ncols": 2,
"pred_nrows": 2,
"table_id": 0,
"true_ncols": 2,
"true_nrows": 2
},
{
"TEDS": 0.913,
"filename": "doc_603d71e12f52d9801a9d82995babf69681d923f493b8d49abfbab8662a88b376_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 9,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 9
},
{
"TEDS": 0.839,
"filename": "doc_603d71e12f52d9801a9d82995babf69681d923f493b8d49abfbab8662a88b376_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 5,
"table_id": 1,
"true_ncols": 5,
"true_nrows": 5
},
{
"TEDS": 0.839,
"filename": "doc_603d71e12f52d9801a9d82995babf69681d923f493b8d49abfbab8662a88b376_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 5,
"table_id": 2,
"true_ncols": 5,
"true_nrows": 5
},
{
"TEDS": 0.956,
"filename": "doc_a0edae1fa147c7bb78ebc493743a68ba4372b5ead31f2a2b146c35119462379e_page_000001.png",
"is_complex": false,
"pred_ncols": 6,
"pred_nrows": 16,
"table_id": 0,
"true_ncols": 6,
"true_nrows": 16
},
{
"TEDS": 0.88,
"filename": "doc_95f808815969717e4ef6390b2b789a8ea552a000a2b936c3c9371f695a71bd9e_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 3,
"true_nrows": 6
},
{
"TEDS": 1.0,
"filename": "doc_ba0e025d53c091e8d4bb87499ff69ed3428dcee325c8895ecdb40e973b4c835c_page_000001.png",
"is_complex": true,
"pred_ncols": 2,
"pred_nrows": 9,
"table_id": 0,
"true_ncols": 2,
"true_nrows": 9
},
{
"TEDS": 0.81,
"filename": "doc_b862cd0d6f06c06ee5ab7729ed4e8ce58e6964eb0f1ab98b3865b57a4808216f_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 4,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 4
},
{
"TEDS": 0.998,
"filename": "doc_750091cec2c31ca61ffaa40c282148258f40bf5db7356a7ed3f918a07c81ef0f_page_000001.png",
"is_complex": false,
"pred_ncols": 1,
"pred_nrows": 8,
"table_id": 0,
"true_ncols": 1,
"true_nrows": 8
},
{
"TEDS": 0.886,
"filename": "doc_a46e40b2b04dc83e4014fca656dae6df20af1fe7a420df306406409ea8c0db31_page_000001.png",
"is_complex": true,
"pred_ncols": 2,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 2,
"true_nrows": 6
},
{
"TEDS": 1.0,
"filename": "doc_ed046862e68a27a7260da60f3984484577f83500dea61020d043c7af8b76731f_page_000001.png",
"is_complex": false,
"pred_ncols": 2,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 2,
"true_nrows": 6
},
{
"TEDS": 0.773,
"filename": "doc_2e4f2561d8624cab6140564d53aed11862cf5979572fc3d1205af105486e0b1f_page_000001.png",
"is_complex": false,
"pred_ncols": 2,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 2,
"true_nrows": 7
},
{
"TEDS": 0.763,
"filename": "doc_4d5b682effaf9580928a7073980e3916687d168238bc96b6c17222f747eacde6_page_000001.png",
"is_complex": false,
"pred_ncols": 6,
"pred_nrows": 7,
"table_id": 0,
"true_ncols": 6,
"true_nrows": 8
},
{
"TEDS": 0.764,
"filename": "doc_4d5b682effaf9580928a7073980e3916687d168238bc96b6c17222f747eacde6_page_000001.png",
"is_complex": false,
"pred_ncols": 5,
"pred_nrows": 9,
"table_id": 1,
"true_ncols": 5,
"true_nrows": 9
},
{
"TEDS": 0.549,
"filename": "doc_62c4b0340519f5963a37534d3abeb0546766cb53bc90504f663eb7cc321a6a6d_page_000001.png",
"is_complex": false,
"pred_ncols": 3,
"pred_nrows": 7,
"table_id": 0,
"true_ncols": 3,
"true_nrows": 7
},
{
"TEDS": 0.749,
"filename": "doc_4b5f23f97fd273cacb8de95fc1799ed4dc4d5c15e9ed86d707cdf4be53eb7c7c_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 3,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 3
},
{
"TEDS": 0.742,
"filename": "doc_c31d842c665085cb1379d441ef0b65869f29537b9252956d6105e6b4cc972311_page_000001.png",
"is_complex": false,
"pred_ncols": 4,
"pred_nrows": 4,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 4
},
{
"TEDS": 0.619,
"filename": "doc_29d883eb26e7b9d08250b4bab1c51092d74e310e6acfa955f8fe28f1008accf5_page_000001.png",
"is_complex": true,
"pred_ncols": 7,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 7,
"true_nrows": 6
},
{
"TEDS": 0.916,
"filename": "doc_61070c2fe64a690c7c9cc97d12b76380e7a134feda3c5315f779306699a89a1b_page_000001.png",
"is_complex": false,
"pred_ncols": 10,
"pred_nrows": 14,
"table_id": 0,
"true_ncols": 10,
"true_nrows": 14
},
{
"TEDS": 0.795,
"filename": "doc_2f972daaa81cb3949db12a39cf6910c4f9952135f7eb3850e1f60ef7d8b5aaf9_page_000001.png",
"is_complex": false,
"pred_ncols": 11,
"pred_nrows": 6,
"table_id": 0,
"true_ncols": 11,
"true_nrows": 6
},
{
"TEDS": 0.756,
"filename": "doc_2f972daaa81cb3949db12a39cf6910c4f9952135f7eb3850e1f60ef7d8b5aaf9_page_000001.png",
"is_complex": false,
"pred_ncols": 10,
"pred_nrows": 4,
"table_id": 1,
"true_ncols": 10,
"true_nrows": 4
},
{
"TEDS": 0.71,
"filename": "doc_2f972daaa81cb3949db12a39cf6910c4f9952135f7eb3850e1f60ef7d8b5aaf9_page_000001.png",
"is_complex": false,
"pred_ncols": 9,
"pred_nrows": 3,
"table_id": 2,
"true_ncols": 9,
"true_nrows": 3
},
{
"TEDS": 0.714,
"filename": "doc_eac01901729786f15ba498716c592370d50f00658c291495086ff49a8538c5c6_page_000001.png",
"is_complex": false,
"pred_ncols": 8,
"pred_nrows": 3,
"table_id": 0,
"true_ncols": 8,
"true_nrows": 3
},
{
"TEDS": 0.822,
"filename": "doc_eac01901729786f15ba498716c592370d50f00658c291495086ff49a8538c5c6_page_000001.png",
"is_complex": false,
"pred_ncols": 9,
"pred_nrows": 5,
"table_id": 1,
"true_ncols": 9,
"true_nrows": 5
},
{
"TEDS": 0.4,
"filename": "doc_39f32d6a01b8e434ad2fb16ff3896931153cdf4f85bca09194b71e1514711355_page_000001.png",
"is_complex": false,
"pred_ncols": 1,
"pred_nrows": 2,
"table_id": 0,
"true_ncols": 2,
"true_nrows": 1
},
{
"TEDS": 1.0,
"filename": "doc_39f32d6a01b8e434ad2fb16ff3896931153cdf4f85bca09194b71e1514711355_page_000001.png",
"is_complex": false,
"pred_ncols": 6,
"pred_nrows": 2,
"table_id": 1,
"true_ncols": 6,
"true_nrows": 2
},
{
"TEDS": 0.804,
"filename": "doc_646f014ab615ac22266a43be2691010032d9fd62368737d488ffcb912cd68fd2_page_000001.png",
"is_complex": true,
"pred_ncols": 4,
"pred_nrows": 10,
"table_id": 0,
"true_ncols": 4,
"true_nrows": 10
}
]
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

@@ -0,0 +1,26 @@
DPBench table_structure TEDS_struct-only: mean=0.85 median=0.87 std=0.11
| TEDS_struct-only | prob [%] | acc [%] | 1-acc [%] | total |
|--------------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 1.96 | 0 | 100 | 1 |
| (0.450, 0.500] | 0 | 1.96 | 98.04 | 0 |
| (0.500, 0.550] | 0 | 1.96 | 98.04 | 0 |
| (0.550, 0.600] | 0 | 1.96 | 98.04 | 0 |
| (0.600, 0.650] | 1.96 | 1.96 | 98.04 | 1 |
| (0.650, 0.700] | 1.96 | 3.92 | 96.08 | 1 |
| (0.700, 0.750] | 7.84 | 5.88 | 94.12 | 4 |
| (0.750, 0.800] | 11.76 | 13.73 | 86.27 | 6 |
| (0.800, 0.850] | 13.73 | 25.49 | 74.51 | 7 |
| (0.850, 0.900] | 33.33 | 39.22 | 60.78 | 17 |
| (0.900, 0.950] | 17.65 | 72.55 | 27.45 | 9 |
| (0.950, 1.000] | 9.8 | 90.2 | 9.8 | 5 |
Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

@@ -0,0 +1,26 @@
DPBench table_structure TEDS_struct-with-text: mean=0.83 median=0.86 std=0.12
| TEDS_struct-with-text | prob [%] | acc [%] | 1-acc [%] | total |
|-------------------------|------------|-----------|-------------|---------|
| (0.000, 0.050] | 0 | 0 | 100 | 0 |
| (0.050, 0.100] | 0 | 0 | 100 | 0 |
| (0.100, 0.150] | 0 | 0 | 100 | 0 |
| (0.150, 0.200] | 0 | 0 | 100 | 0 |
| (0.200, 0.250] | 0 | 0 | 100 | 0 |
| (0.250, 0.300] | 0 | 0 | 100 | 0 |
| (0.300, 0.350] | 0 | 0 | 100 | 0 |
| (0.350, 0.400] | 0 | 0 | 100 | 0 |
| (0.400, 0.450] | 1.96 | 0 | 100 | 1 |
| (0.450, 0.500] | 0 | 1.96 | 98.04 | 0 |
| (0.500, 0.550] | 1.96 | 1.96 | 98.04 | 1 |
| (0.550, 0.600] | 1.96 | 3.92 | 96.08 | 1 |
| (0.600, 0.650] | 1.96 | 5.88 | 94.12 | 1 |
| (0.650, 0.700] | 0 | 7.84 | 92.16 | 0 |
| (0.700, 0.750] | 11.76 | 7.84 | 92.16 | 6 |
| (0.750, 0.800] | 11.76 | 19.61 | 80.39 | 6 |
| (0.800, 0.850] | 13.73 | 31.37 | 68.63 | 7 |
| (0.850, 0.900] | 29.41 | 45.1 | 54.9 | 15 |
| (0.900, 0.950] | 15.69 | 74.51 | 25.49 | 8 |
| (0.950, 1.000] | 9.8 | 90.2 | 9.8 | 5 |