mirror of
https://github.com/docling-project/docling-core.git
synced 2026-11-18 20:27:00 +00:00
* Added ruff to dev dependencies Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added ruff settings to pyproject.toml as in docling Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Cleanup uf pyproject.toml Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Copied settings for ruff pre-commit hooks from docling Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Excluded test/data/** from ruff formatting / linting Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * ruff format Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added some ignore statements to pyproject.toml such that ruff check raises fewer issues Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * ruff check --fix Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Ignored some more rules Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Fixed the rest of the errors that would only concern 1 - 3 files Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added another ignore related to df for DataFrame names Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Modified CONTRIBUTING.md such that black / isort are replaced by ruff Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added UP045 to ignore list such that Optional[...] does not raise Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Moved .flake8 configs to pyproject.toml Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Moved autoflake to be used with ruff Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Removed flake8 from .pre-commit hooks Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Adding PD901 to ignore list such that pre-commit hooks run fully again Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Replaced dtype | None syntax by Optional[dtype] in remaining places Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * chore: fix 'test' ref in pyproject Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove typing List, Set, Tuple, Dict Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove UP015 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove UP034 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: normalize dashes in comments and docstrings Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove PD901 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove C403 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove C403, C413, C416 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove E203, F811 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> --------- Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com> Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
27 KiB
27 KiB
In [1]:
from docling_core.types.doc.document import DoclingDocument
file_path = "2408.09869v3.json"
pages = {5} # pages to serialize (for output brevity)
doc = DoclingDocument.load_from_json(file_path)In [2]:
from typing import Optional
from rich.console import Console
from rich.panel import Panel
def print_excerpt(
txt: str,
*,
limit: int = 2000,
title: Optional[str] = None,
min_width: int = 80,
table_end: str = "--|",
):
excerpt = txt[:limit]
width = max(
max([ln.rfind(table_end) for ln in excerpt.splitlines()]) + len(table_end) + 4,
min_width,
)
console = Console(width=width)
console.print(Panel(f"{excerpt}{'...' if len(txt) > limit else ''}", title=title))In [3]:
from docling_core.types.doc.document import DescriptionAnnotation, MiscAnnotation
assert doc.tables, "No table available in this document"
table = doc.tables[0]
table.add_annotation(
annotation=DescriptionAnnotation(
text="A typical Docling setup runtime characterization.",
provenance="model-foo",
),
)
table.add_annotation(
annotation=MiscAnnotation(
content={
"type": "performance data",
"sentiment": 0.85,
# ...
},
),
)In [4]:
from docling_core.transforms.serializer.markdown import (
MarkdownDocSerializer,
MarkdownParams,
)
ser = MarkdownDocSerializer(
doc=doc,
params=MarkdownParams(
pages=pages,
),
)
ser_out = ser.serialize()
ser_txt = ser_out.text
print_excerpt(ser_txt, title=f"{pages=}")╭────────────────────────────────────────────────────────────────────────────────── pages={5} ───────────────────────────────────────────────────────────────────────────────────╮
│ torch runtimes backing the Docling pipeline. We will deliver updates on this topic at in a future version of this report. │
│ │
│ Table 1: Runtime characteristics of Docling with the standard model pipeline and settings, on our test dataset of 225 pages, on two different systems. OCR is disabled. We │
│ show the time-to-solution (TTS), computed throughput in pages per second, and the peak memory used (resident set size) for both the Docling-native PDF backend and for the │
│ pypdfium backend, using 4 and 16 threads. │
│ │
│ A typical Docling setup runtime characterization. │
│ │
│ | CPU | Thread budget | native backend | native backend | native backend | pypdfium backend | pypdfium backend | pypdfium backend | │
│ |----------------------------------|-----------------|------------------|------------------|------------------|--------------------|--------------------|--------------------| │
│ | | | TTS | Pages/s | Mem | TTS | Pages/s | Mem | │
│ | Apple M3 Max | 4 | 177 s 167 s | 1.27 1.34 | 6.20 GB | 103 s 92 s | 2.18 2.45 | 2.56 GB | │
│ | (16 cores) Intel(R) Xeon E5-2690 | 16 4 16 | 375 s 244 s | 0.60 0.92 | 6.16 GB | 239 s 143 s | 0.94 1.57 | 2.42 GB | │
│ │
│ ## 5 Applications │
│ │
│ Thanks to the high-quality, richly structured document conversion achieved by Docling, its output qualifies for numerous downstream applications. For example, Docling can │
│ provide a base for detailed enterprise document search, passage retrieval or classification use-cases, or support knowledge extraction pipelines, allowing specific treatment │
│ of different structures in the document, such as tables, figures, section structure or references. For popular generative AI application patterns, such as retrieval-augmented │
│ generation (RAG), we provi... │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
In [5]:
from typing import Any
from docling_core.transforms.serializer.base import SerializationResult
from docling_core.transforms.serializer.common import create_ser_result
from docling_core.transforms.serializer.markdown import MarkdownAnnotationSerializer
from docling_core.types.doc.document import MiscAnnotation, DocItem
class CustomAnnotationSerializer(MarkdownAnnotationSerializer):
def serialize(
self,
*,
item: DocItem,
doc: DoclingDocument,
**kwargs: Any,
) -> SerializationResult:
text_parts: list[str] = []
# reusing result from parent serializer:
parent_res = super().serialize(
item=item,
doc=doc,
**kwargs,
)
text_parts.append(parent_res.text)
# custom serialization logic (appending misc annotation result):
for ann in item.get_annotations():
if isinstance(ann, MiscAnnotation):
out_txt = "".join([f"- {k}: {ann.content[k]}\n" for k in ann.content])
text_parts.append(out_txt)
text_res = "\n\n".join(text_parts)
return create_ser_result(text=text_res, span_source=item)In [6]:
ser = MarkdownDocSerializer(
doc=doc,
annotation_serializer=CustomAnnotationSerializer(),
params=MarkdownParams(
pages=pages,
),
)
ser_out = ser.serialize()
ser_txt = ser_out.text
print_excerpt(ser_txt, title=f"{pages=}")╭────────────────────────────────────────────────────────────────────────────────── pages={5} ───────────────────────────────────────────────────────────────────────────────────╮
│ torch runtimes backing the Docling pipeline. We will deliver updates on this topic at in a future version of this report. │
│ │
│ Table 1: Runtime characteristics of Docling with the standard model pipeline and settings, on our test dataset of 225 pages, on two different systems. OCR is disabled. We │
│ show the time-to-solution (TTS), computed throughput in pages per second, and the peak memory used (resident set size) for both the Docling-native PDF backend and for the │
│ pypdfium backend, using 4 and 16 threads. │
│ │
│ A typical Docling setup runtime characterization. │
│ │
│ - type: performance data │
│ - sentiment: 0.85 │
│ │
│ │
│ | CPU | Thread budget | native backend | native backend | native backend | pypdfium backend | pypdfium backend | pypdfium backend | │
│ |----------------------------------|-----------------|------------------|------------------|------------------|--------------------|--------------------|--------------------| │
│ | | | TTS | Pages/s | Mem | TTS | Pages/s | Mem | │
│ | Apple M3 Max | 4 | 177 s 167 s | 1.27 1.34 | 6.20 GB | 103 s 92 s | 2.18 2.45 | 2.56 GB | │
│ | (16 cores) Intel(R) Xeon E5-2690 | 16 4 16 | 375 s 244 s | 0.60 0.92 | 6.16 GB | 239 s 143 s | 0.94 1.57 | 2.42 GB | │
│ │
│ ## 5 Applications │
│ │
│ Thanks to the high-quality, richly structured document conversion achieved by Docling, its output qualifies for numerous downstream applications. For example, Docling can │
│ provide a base for detailed enterprise document search, passage retrieval or classification use-cases, or support knowledge extraction pipelines, allowing specific treatment │
│ of different structures in the document, such as tables, figures, section structure or references. For popular generative AI application patterns, such as r... │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
In [ ]: