mirror of
https://github.com/docling-project/docling-core.git
synced 2026-11-18 20:27:00 +00:00
* fix(doclang): preserve leading text before a lone formatting tag `_get_children_simple_text_block` guarded against a second *text* fragment but the Element branch just overwrote `result`, so `<text>2<superscript>nd</superscript></text>` returned "nd" and the leading `2` was silently discarded. Guard both branches so the element is routed to `_parse_inline_group`, which already handles mixed children. * feat: inline runs carry their own whitespace; serializers concatenate faithfully The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`, `O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never be removed while the separator lives in the serializer. Serializers: - join inline parts with `""`; DocLang keeps its delimiter between the element head and the body only, where `<content>` is the whitespace channel - hoist edge whitespace around the *complete* decoration stack (formatting and hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim emits `**bold ** tail`, which CommonMark renders as literal asterisks; hoisting inside one formatter would let the hyperlink recapture it. DocLang opts out: it keeps whitespace inside the markup, in `<content>`. DocLang deserializer: - `_get_text` stripped every fragment and joined with `""`, collapsing `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer block boundary only, and never trims whitespace delivered by `<content>`. - bare text nodes use the standard XML heuristic: a whitespace run containing a newline is pretty-print indentation, one without it is content. This is what hand-written and VLM-emitted DocLang needs, since it will not use `<content>`. - element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`) no longer bleed into body text, where they were also double-modelled in `item.meta`. Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator re-injects the legacy separator for documents stamped below it, preserving their previous *visible* output including its extra-space bugs (`H 2 O` stays `H 2 O`). It must run before `check_version_is_compatible`, which rewrites `version` and destroys the evidence. Versionless input is left alone. The validator works on a copy, so the caller's dict is untouched and re-validating the same raw value cannot compound separators. A separator can never land inside a delimiter. Code and formula runs are wrapped in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when both neighbours are delimited it becomes a plain run of its own, because `` `a``b` `` and `$a$$b$` are both corrupt. Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join -- anyone with a persisted vector index gets silent retrieval drift, so that needs its own line in the release notes. DocTags code is untouched. Its golden gains one empty `<text></text>` for the legacy fixture because that document now carries an extra whitespace run, which DocTags renders empty by its own pre-existing strip-per-run behaviour. Ships as a minor, not a major: `feat!:` would make python-semantic-release (7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package. * DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com> I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 64404b3709d9e3ba7caf08957cf2ef19df29459f I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 66040cb7b96f77699c9d74051500d645e7e758ed Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat(compat): register 1.11->1.10 projector for the whitespace-contract bump Integrates the 1.11.0 schema bump with the downgrade-projector framework (#703) landed upstream. 1.11 and 1.10 are structurally identical; the only difference is that 1.11 inline runs own their boundary whitespace, so the projector just re-stamps the version (a 1.10 client re-applies its own join, yielding the documented harmless double space). - add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10 - add the auto-generated 1.11 JSON Schema snapshot - re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff) - update the stale dclq CLI expectation (old join produced 'public class'; faithful join now yields 'public class') Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(dclq): use typer's vendored click; require typer>=0.25 typer >=0.25 vendors its own copy of click (typer._click) and raises those exceptions instead of the external click package's. The dclq CLI referenced the external click, so on typer >=0.25 bad input escaped the exit-code handlers and the context-flag callback (-C/-A/-B/--context-scope) silently no-op'd because the isinstance(param, click.Option) check no longer matched. Import the exception and Parameter types from typer._click, duck-type the option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused external click dependency. Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(doclang): keep inherited formatting on <content> runs in inline groups <content> is the explicit whitespace-preserving text wrapper, so inside an inline group it is a run, not a nested item: it must inherit the formatting accumulated while descending through <bold>/<italic>/... . Dispatching it through _dispatch_element dropped that formatting, which surfaced after the rebase as <bold><content>x </content></bold> coming back unformatted and broke the doclang round-trip for whitespace_inside_bold / _link. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com> * chore(compat): point schema 1.11.0 at the next minor release The whitespace-contract bump was first written against v2.86.0; main is now at v2.104.0, so record v2.105.0 as the placeholder release that ships schema 1.11.0. Adjust to the actual version at release time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * test: rebaseline goldens for the inline whitespace contract Regenerated with DOCLING_GEN_TEST_DATA=1. The only change is the schema version stamp 1.10.0 -> 1.11.0 in goldens added or moved on main (including the six test/ -> tests/ renames and misplaced_list_items.out.yaml). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix: join inline runs faithfully in the plain-text reader _collect_subtree_text (TextItem._get_plain_text, FloatingItem.caption_text, nested rich table cells) still joined the runs of an InlineGroup with " ", so mixed captions read "Table 1 : results". It now concatenates the runs of an inline group; every other join, including a host text item's own text and its children, keeps its space. Children of a text item are block structure (producers put a section's paragraphs under its heading), so the host text stays apart from its inline group, as on main; add_inline_group documents this. Tests that asserted the invented run spacing now expect the contract output, fixtures that built runs without boundary whitespace now carry it, and a new test pins the host-text boundary for headings and captions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * docs: note that load_from_doctags cannot restore inline run whitespace DocTags stores inline runs trimmed, so the boundary whitespace the inline whitespace contract expects is not in the source. Document the limitation instead of guessing separators back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(md): keep the host separator before a list item's inline group The Markdown list serializer glued an inline group onto the preceding list item with no separator, so a list item "Intro" whose child group starts with "more" exported "- Intromore". Captions, headings and the plain-text reader keep a space between host text and its inline group; the list serializer now does the same when the group is the child of a text item with its own text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): keep migrated separators between code or formula runs in page filters A legacy document with two adjacent code or formula runs gets a standalone " " run between them on load. It had no provenance, so filter(page_nrs=...) and export_to_markdown(page_no=...) dropped it and rendered `$a$$b$`. The separator now takes the left run's location when both runs are on the same page, so it survives a page filter exactly when both runs do. Runs on different pages keep the unlocated separator, which matches the old join. Inline group locations in DocLang and DocTags are a union of the runs, so they do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): count empty inline hosts and nested inline groups as legacy runs The legacy separator migration counted only direct text children with non-empty text as runs. An empty-text item whose content is a child inline group (a formula or footnote host) and an inline group nested directly in another were skipped, so `left`, host(`x`), `right` loaded as `left $x$right` instead of the old `left $x$ right`. The DocLang deserializer writes both shapes for nested mixed-content elements. A child now counts as a run when it or its subtree has text. Only a plain run with its own text takes the separator; an empty host must keep its empty text to render its group, so it is treated like a code or formula run and the separator stays outside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * docs(doclang): note that pre-contract archives load without legacy separators DocLang has no schema version, so a .dclx written before schema 1.11.0 cannot be told apart from a current one and is not migrated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * chore(dclq): move the vendored-click fix to its own PR This reverts 6a8b8374; the change is unrelated to the whitespace contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): don't run the legacy inline migration in project_to project_to validated the projected dict under its target stamp. For 1.10.0 that ran the load-time separator migration on runs that already carry their whitespace, so 'Normal ', 'bold', ' tail' came back as 'Normal ', 'bold ', ' tail'. Validate under the source stamp and restore the target stamp afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): strip inline runs when projecting 1.11 to 1.10 A 1.10 client joins inline runs with " ", so runs that keep their boundary whitespace rendered with double spaces and with the space inside the markup (**bold **, which CommonMark does not render as bold). The projector now strips each inline-group child's text to the shape old producers emitted. Whitespace-only runs become "", which old serializers skip, so no refs move. orig and text outside inline groups are left alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): reuse the stripped separator run when migrating Between two code or formula runs the migration inserts a plain " " run. The 1.11 -> 1.10 projector strips it to "", and the next migration skipped the empty run and inserted another, so every projection round trip added a run. Refill an empty plain run between the two neighbours instead. Old serializers skipped empty runs, so legacy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com> I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit:7600953bc0I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit:f0e26b6722Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat(doclang): read and write whitespace per DocLang 0.9 - Deserializer applies spec rules to the DOM before parsing: a whitespace sequence reads as one space, none at block edges, <content> is literal. - Serializer keeps each text block body on one line; text with non-single-space whitespace goes in <content>, unwrapped where the rules read it the same bare. - Emit version 0.9; validate against doclang 0.9. Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * fix(doclang): keep inline code and formula in virtual text In a list item or cell with text of its own, <code> and <formula> are inline, so whitespace around them reads as a space under DocLang 0.9. - Reader applies the whitespace rules across them instead of treating them as blocks. - Pretty-printer writes such a virtual text on one line, so it no longer adds spaces next to inline code or formulas. Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * chore(compat): point schema 1.11.0 at v2.106.0 v2.105.0 was released without the whitespace contract, so record the next minor release as the one that ships schema 1.11.0. Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * docs(compat): name schema 1.11 as current in the projectors header Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * fix(doclang): harden the DocLang 0.9 whitespace reader and writer - read list items with leading inline code or formula as virtual text - keep whitespace-only content between inline code runs - keep blank lines in content when pretty printing - keep inline handwriting and rtl text as runs - trim only XML whitespace when reading text - keep content literal in headings, captions and list items - read table cells with inline elements as virtual text - keep text beside a whitespace-only formatted run - read spaces around an empty content as one sequence - read a checkbox as part of virtual text - keep a whitespace-only inline code run Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(doclang): preserve checkbox spaces and formatted whitespace runs Signed-off-by: Christoph Auer <cau@zurich.ibm.com> --------- Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com> Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com> Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2899 lines
98 KiB
Python
2899 lines
98 KiB
Python
import re
|
||
import xml.etree.ElementTree as ET
|
||
from collections.abc import Callable
|
||
from pathlib import Path
|
||
|
||
import pytest
|
||
|
||
from docling_core.transforms.deserializer.doclang import DocLangDocDeserializer
|
||
from docling_core.transforms.serializer.doclang import (
|
||
DocLangDocSerializer,
|
||
DocLangParams,
|
||
EscapeMode,
|
||
LabelMode,
|
||
)
|
||
from docling_core.transforms.serializer.doctags import DocTagsDocSerializer
|
||
from docling_core.transforms.serializer.html import HTMLDocSerializer
|
||
from docling_core.transforms.serializer.latex import LaTeXDocSerializer
|
||
from docling_core.transforms.serializer.markdown import MarkdownDocSerializer
|
||
from docling_core.transforms.serializer.plain_text import PlainTextDocSerializer
|
||
from docling_core.types.doc import (
|
||
BoundingBox,
|
||
ContentLayer,
|
||
CoordOrigin,
|
||
DescriptionMetaField,
|
||
DocItemLabel,
|
||
DoclingDocument,
|
||
Formatting,
|
||
ImageRefMode,
|
||
MoleculeMetaField,
|
||
PictureClassificationMetaField,
|
||
PictureClassificationPrediction,
|
||
PictureItem,
|
||
PictureMeta,
|
||
ProvenanceItem,
|
||
RichTableCell,
|
||
Script,
|
||
Size,
|
||
SummaryMetaField,
|
||
TableCell,
|
||
TableData,
|
||
TableItem,
|
||
TabularChartMetaField,
|
||
TextItem,
|
||
)
|
||
from docling_core.types.doc.document import GroupLabel
|
||
from docling_core.types.doc.labels import CodeLanguageLabel, PictureClassificationLabel
|
||
from tests.doclang_validation import assert_valid_dclg_xml, doclang_validator
|
||
from tests.test_data_gen_flag import GEN_TEST_DATA
|
||
from tests.test_serialization_doclang import (
|
||
_doc_cross_column_list,
|
||
_doc_cross_page_list,
|
||
_doc_cross_page_paragraph,
|
||
_doc_cross_page_table,
|
||
_doc_multi_prov_text,
|
||
_verify_doc,
|
||
add_list_section,
|
||
add_texts_section,
|
||
verify_doclang,
|
||
)
|
||
from tests.test_serialization_doctag import verify
|
||
|
||
DO_PRINT: bool = False
|
||
|
||
|
||
def _serialize(doc: DoclingDocument) -> str:
|
||
ser = DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False),
|
||
)
|
||
text = ser.serialize().text
|
||
if not GEN_TEST_DATA:
|
||
assert_valid_dclg_xml(text)
|
||
return text
|
||
|
||
|
||
def _deserialize(text: str, *, validate: bool = True) -> DoclingDocument:
|
||
if validate and not GEN_TEST_DATA:
|
||
assert_valid_dclg_xml(text)
|
||
return DocLangDocDeserializer().deserialize_str(text)
|
||
|
||
|
||
def _add_default_page(doc: DoclingDocument):
|
||
doc.add_page(page_no=0, size=Size(width=1000, height=1000))
|
||
|
||
|
||
def _default_prov() -> ProvenanceItem:
|
||
return ProvenanceItem(
|
||
page_no=0,
|
||
bbox=BoundingBox(l=100, t=100, r=300, b=200),
|
||
charspan=(0, 0),
|
||
)
|
||
|
||
|
||
class _VirtualTextMixedBboxFactory:
|
||
"""Assign distinct page bboxes for the virtual-text mixed fixture."""
|
||
|
||
def __init__(self, *, page_no: int = 0, page_w: float = 1000, page_h: float = 1000) -> None:
|
||
self.page_no = page_no
|
||
self.page_w = page_w
|
||
self.page_h = page_h
|
||
self._slot = 0
|
||
|
||
def next_prov(self) -> ProvenanceItem:
|
||
row, col = divmod(self._slot, 4)
|
||
self._slot += 1
|
||
cell_w = self.page_w / 5
|
||
cell_h = self.page_h / 20
|
||
left = 20 + col * cell_w
|
||
top = self.page_h - 40 - row * cell_h
|
||
return ProvenanceItem(
|
||
page_no=self.page_no,
|
||
bbox=BoundingBox.from_tuple(
|
||
(left, top - cell_h, left + cell_w - 10, top),
|
||
origin=CoordOrigin.BOTTOMLEFT,
|
||
),
|
||
charspan=(0, 0),
|
||
)
|
||
|
||
def next_bbox(self) -> BoundingBox:
|
||
return self.next_prov().bbox
|
||
|
||
|
||
def _serialize_virtual_text_mixed(doc: DoclingDocument, *, add_location: bool = True) -> str:
|
||
ser = DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False, add_table_cell_location=True, add_location=add_location),
|
||
)
|
||
text = ser.serialize().text
|
||
if not GEN_TEST_DATA:
|
||
assert_valid_dclg_xml(text)
|
||
return text
|
||
|
||
|
||
def test_roundtrip_text():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_text(label=DocItemLabel.TEXT, text="Hello world")
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt = """
|
||
<doclang>
|
||
<text>Hello world</text>
|
||
</doclang>
|
||
"""
|
||
assert dt2.strip() == exp_dt.strip()
|
||
|
||
|
||
def test_deserialize_include_namespace_and_version():
|
||
"""Deserialize DocLang XML with namespace and version, then roundtrip."""
|
||
exp_file = Path("./tests/data/doc/deserialize_include_namespace_and_version.gt.dclg.xml")
|
||
xml = exp_file.read_text(encoding="utf-8")
|
||
|
||
doc = _deserialize(xml)
|
||
assert len(doc.texts) == 1
|
||
assert doc.texts[0].text == "Hello world"
|
||
|
||
reserialized = (
|
||
DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_namespace=True, include_version=True),
|
||
)
|
||
.serialize()
|
||
.text
|
||
)
|
||
verify_doclang(exp_file=exp_file, actual=reserialized)
|
||
|
||
|
||
def test_roundtrip_title():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_title(text="My Title")
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt = """
|
||
<doclang>
|
||
<heading>My Title</heading>
|
||
</doclang>
|
||
"""
|
||
assert dt2.strip() == exp_dt.strip()
|
||
|
||
|
||
def test_roundtrip_heading():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_heading(text="Section A", level=2)
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt = """
|
||
<doclang>
|
||
<heading level="3">Section A</heading>
|
||
</doclang>
|
||
"""
|
||
assert dt2.strip() == exp_dt.strip()
|
||
|
||
|
||
def test_roundtrip_caption():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_text(label=DocItemLabel.CAPTION, text="Cap text")
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt = """
|
||
<doclang>
|
||
<text>Cap text</text>
|
||
</doclang>
|
||
"""
|
||
assert dt2.strip() == exp_dt.strip()
|
||
|
||
|
||
def test_roundtrip_footnote():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_text(label=DocItemLabel.FOOTNOTE, text="Foot note")
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.FOOTNOTE
|
||
assert doc2.texts[0].text == "Foot note"
|
||
|
||
|
||
def test_roundtrip_page_header():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_text(label=DocItemLabel.PAGE_HEADER, text="Header")
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.PAGE_HEADER
|
||
assert doc2.texts[0].text == "Header"
|
||
|
||
|
||
def test_roundtrip_page_footer():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_text(label=DocItemLabel.PAGE_FOOTER, text="Footer")
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.PAGE_FOOTER
|
||
assert doc2.texts[0].text == "Footer"
|
||
|
||
|
||
def test_roundtrip_code():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_code(text="print('hi')", code_language=CodeLanguageLabel.PYTHON)
|
||
doc.add_code(text='disp("Hello world!")', code_language=CodeLanguageLabel.OCTAVE)
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt = """
|
||
<doclang>
|
||
<code>
|
||
<label value="Python"/>
|
||
<![CDATA[print('hi')]]>
|
||
</code>
|
||
<code>
|
||
<label value="MATLAB"/>
|
||
<![CDATA[disp("Hello world!")]]>
|
||
</code>
|
||
</doclang>
|
||
"""
|
||
assert dt2.strip() == exp_dt.strip()
|
||
|
||
|
||
def test_roundtrip_code_python_and_unknown():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_code(text="x = 1", code_language=CodeLanguageLabel.PYTHON)
|
||
doc.add_code(text="y = 2") # CodeLanguageLabel.UNKNOWN by default
|
||
|
||
xml = _serialize(doc)
|
||
assert '<label value="Python"/>' in xml
|
||
assert '<label value="other"/>' not in xml
|
||
assert '<label value="undefined"/>' not in xml
|
||
assert '<label value="unknown"/>' not in xml
|
||
|
||
doc2 = _deserialize(xml)
|
||
codes = [t for t in doc2.texts if t.label == DocItemLabel.CODE]
|
||
assert len(codes) == 2
|
||
assert codes[0].code_language == CodeLanguageLabel.PYTHON
|
||
assert codes[1].code_language == CodeLanguageLabel.UNKNOWN
|
||
assert codes[0].text.strip() == "x = 1"
|
||
assert codes[1].text.strip() == "y = 2"
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("docling_lang", "linguist_label", "roundtrip_lang"),
|
||
[
|
||
(CodeLanguageLabel.BASH, "Shell", CodeLanguageLabel.BASH),
|
||
(CodeLanguageLabel.LATEX, "TeX", CodeLanguageLabel.LATEX),
|
||
(CodeLanguageLabel.LISP, "Common Lisp", CodeLanguageLabel.LISP),
|
||
(CodeLanguageLabel.OBJECTIVEC, "Objective-C", CodeLanguageLabel.OBJECTIVEC),
|
||
(CodeLanguageLabel.SML, "Standard ML", CodeLanguageLabel.SML),
|
||
(CodeLanguageLabel.VISUALBASIC, "Visual Basic .NET", CodeLanguageLabel.VISUALBASIC),
|
||
(CodeLanguageLabel.OCTAVE, "MATLAB", CodeLanguageLabel.MATLAB),
|
||
(CodeLanguageLabel.BC, "other", CodeLanguageLabel.UNKNOWN),
|
||
(CodeLanguageLabel.DOCLANG, "XML", CodeLanguageLabel.XML),
|
||
],
|
||
)
|
||
def test_code_language_linguist_mapping(docling_lang, linguist_label, roundtrip_lang):
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_code(text="snippet", code_language=docling_lang)
|
||
|
||
xml = DocLangDocSerializer(doc=doc, params=DocLangParams(include_version=False)).serialize().text
|
||
assert f'<label value="{linguist_label}"/>' in xml
|
||
|
||
doc2 = _deserialize(xml)
|
||
assert doc2.texts[0].code_language == roundtrip_lang
|
||
|
||
|
||
def test_roundtrip_code_unknown_as_other_when_enabled():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_code(text="y = 2")
|
||
xml = (
|
||
DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False, interpret_code_unknown_as_other=True),
|
||
)
|
||
.serialize()
|
||
.text
|
||
)
|
||
assert '<label value="other"/>' in xml
|
||
doc2 = _deserialize(xml)
|
||
assert doc2.texts[0].code_language == CodeLanguageLabel.UNKNOWN
|
||
|
||
|
||
def test_roundtrip_picture_other_and_unknown_labels():
|
||
doc = DoclingDocument(name="t")
|
||
classified = doc.add_picture()
|
||
classified.meta = PictureMeta(
|
||
classification=PictureClassificationMetaField(
|
||
predictions=[
|
||
PictureClassificationPrediction(
|
||
class_name=PictureClassificationLabel.OTHER.value,
|
||
confidence=1.0,
|
||
)
|
||
]
|
||
)
|
||
)
|
||
doc.add_picture()
|
||
|
||
xml = _serialize(doc)
|
||
assert '<label value="other"/>' in xml
|
||
assert '<label value="unknown"/>' not in xml
|
||
|
||
doc2 = _deserialize(xml)
|
||
pics = [item for item, _ in doc2.iterate_items() if isinstance(item, PictureItem)]
|
||
assert pics[0].meta.classification.get_main_prediction().class_name == "other"
|
||
assert pics[1].meta is None or pics[1].meta.classification is None
|
||
|
||
xml_always = (
|
||
DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False, label_mode=LabelMode.ALWAYS, add_location=False),
|
||
)
|
||
.serialize()
|
||
.text
|
||
)
|
||
assert xml_always.count('<label value="undefined"/>') == 1
|
||
|
||
|
||
def test_code_language_unmapped_linguist_deserializes_to_unknown():
|
||
xml = '<doclang><code><label value="CoffeeScript"/>foo</code></doclang>'
|
||
doc = _deserialize(xml, validate=False)
|
||
assert doc.texts[0].code_language == CodeLanguageLabel.UNKNOWN
|
||
|
||
|
||
def test_roundtrip_formula():
|
||
doc = DoclingDocument(name="t")
|
||
doc.add_formula(text="E=mc^2")
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.FORMULA
|
||
assert doc2.texts[0].text == "E=mc^2"
|
||
|
||
|
||
def test_roundtrip_list_unordered():
|
||
doc = DoclingDocument(name="t")
|
||
lg = doc.add_list_group()
|
||
doc.add_list_item("A", parent=lg, enumerated=False)
|
||
doc.add_list_item("B", parent=lg, enumerated=False)
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
# Ensure group created and items present
|
||
assert len(doc2.groups) == 1
|
||
assert len(doc2.texts) == 2
|
||
assert [it.text for it in doc2.texts] == ["A", "B"]
|
||
|
||
|
||
def test_roundtrip_list_ordered():
|
||
doc = DoclingDocument(name="t")
|
||
lg = doc.add_list_group()
|
||
doc.add_list_item("1", parent=lg, enumerated=True)
|
||
doc.add_list_item("2", parent=lg, enumerated=True)
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.groups) == 1
|
||
assert len(doc2.texts) == 2
|
||
assert [it.text for it in doc2.texts] == ["1", "2"]
|
||
|
||
|
||
def test_roundtrip_picture_with_caption():
|
||
doc = DoclingDocument(name="t")
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Fig 1")
|
||
doc.add_picture(caption=cap)
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.pictures) == 1
|
||
pic = doc2.pictures[0]
|
||
assert len(pic.captions) == 1
|
||
cap_item = pic.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION and cap_item.text == "Fig 1"
|
||
|
||
|
||
def test_roundtrip_table_simple():
|
||
doc = DoclingDocument(name="t")
|
||
td = TableData(num_rows=0, num_cols=2)
|
||
td.add_row(["H1", "H2"]) # header row semantics not required here
|
||
td.add_row(["C1", "C2"]) # data row
|
||
doc.add_table(data=td)
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.tables) == 1
|
||
t2 = doc2.tables[0].data
|
||
assert t2.num_rows == 2 and t2.num_cols == 2
|
||
grid_texts = [[cell.text for cell in row] for row in t2.grid]
|
||
assert grid_texts == [["H1", "H2"], ["C1", "C2"]]
|
||
|
||
|
||
def test_roundtrip_table_with_caption():
|
||
doc = DoclingDocument(name="t")
|
||
# Create a caption and a simple 2x2 table
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Tbl 1")
|
||
td = TableData(num_rows=0, num_cols=2)
|
||
td.add_row(["H1", "H2"]) # header row
|
||
td.add_row(["C1", "C2"]) # data row
|
||
doc.add_table(data=td, caption=cap)
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print(dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# One table reconstructed with same grid
|
||
assert len(doc2.tables) == 1
|
||
t2 = doc2.tables[0]
|
||
assert t2.data.num_rows == 2 and t2.data.num_cols == 2
|
||
grid_texts = [[cell.text for cell in row] for row in t2.data.grid]
|
||
assert grid_texts == [["H1", "H2"], ["C1", "C2"]]
|
||
|
||
# Caption preserved and linked to the table
|
||
assert len(t2.captions) == 1
|
||
cap_item = t2.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION and cap_item.text == "Tbl 1"
|
||
|
||
|
||
def test_roundtrip_text_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="Hello world", prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\n", dt2)
|
||
print(f"`{doc2.texts[0].text}`")
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.TEXT
|
||
assert doc2.texts[0].text == "Hello world"
|
||
|
||
|
||
def test_roundtrip_title_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_title(
|
||
text="My Title",
|
||
prov=_default_prov(),
|
||
)
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt = """
|
||
<doclang>
|
||
<heading>
|
||
<location value="51"/>
|
||
<location value="51"/>
|
||
<location value="154"/>
|
||
<location value="102"/>
|
||
My Title
|
||
</heading>
|
||
</doclang>
|
||
"""
|
||
assert dt2.strip() == exp_dt.strip()
|
||
|
||
|
||
def test_roundtrip_heading_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_heading(text="Section A", level=2, prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
h = doc2.texts[0]
|
||
assert h.label == DocItemLabel.SECTION_HEADER and getattr(h, "level", 0) == 2
|
||
assert h.text == "Section A"
|
||
|
||
|
||
def test_roundtrip_caption_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="Cap text", prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.TEXT
|
||
assert doc2.texts[0].text == "Cap text"
|
||
|
||
|
||
def test_roundtrip_footnote_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_text(label=DocItemLabel.FOOTNOTE, text="Foot note", prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.FOOTNOTE
|
||
assert doc2.texts[0].text == "Foot note"
|
||
|
||
|
||
def test_roundtrip_page_header_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_text(label=DocItemLabel.PAGE_HEADER, text="Header", prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.PAGE_HEADER
|
||
assert doc2.texts[0].text == "Header"
|
||
|
||
|
||
def test_roundtrip_page_footer_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_text(label=DocItemLabel.PAGE_FOOTER, text="Footer", prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.PAGE_FOOTER
|
||
assert doc2.texts[0].text == "Footer"
|
||
|
||
|
||
def test_roundtrip_code_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_code(text="print('hi')", code_language=CodeLanguageLabel.PYTHON, prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.CODE
|
||
assert doc2.texts[0].text.strip() == "print('hi')"
|
||
assert getattr(doc2.texts[0], "code_language", None) == CodeLanguageLabel.PYTHON
|
||
|
||
|
||
def test_roundtrip_formula_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_formula(text="E=mc^2", prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.texts) == 1
|
||
assert doc2.texts[0].label == DocItemLabel.FORMULA
|
||
assert doc2.texts[0].text == "E=mc^2"
|
||
|
||
|
||
def test_roundtrip_list_unordered_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
lg = doc.add_list_group()
|
||
prov = _default_prov()
|
||
doc.add_list_item("A", parent=lg, prov=prov)
|
||
doc.add_list_item("B", parent=lg, prov=prov)
|
||
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
assert dt2.strip() == dt.strip()
|
||
|
||
|
||
def test_roundtrip_list_ordered_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
lg = doc.add_list_group()
|
||
prov = _default_prov()
|
||
doc.add_list_item("1", parent=lg, enumerated=True, prov=prov)
|
||
doc.add_list_item("2", parent=lg, enumerated=True, prov=prov)
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.groups) == 1
|
||
assert len(doc2.texts) == 2
|
||
assert [it.text for it in doc2.texts] == ["1", "2"]
|
||
|
||
|
||
def test_roundtrip_picture_with_caption_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Fig 1", prov=_default_prov())
|
||
doc.add_picture(caption=cap, prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.pictures) == 1
|
||
pic = doc2.pictures[0]
|
||
assert len(pic.captions) == 1
|
||
cap_item = pic.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION and cap_item.text == "Fig 1"
|
||
|
||
|
||
def test_roundtrip_table_simple_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
td = TableData(num_rows=0, num_cols=2)
|
||
td.add_row(["H1", "H2"]) # header row semantics not required here
|
||
td.add_row(["C1", "C2"]) # data row
|
||
doc.add_table(data=td, prov=_default_prov())
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
assert len(doc2.tables) == 1
|
||
t2 = doc2.tables[0].data
|
||
assert t2.num_rows == 2 and t2.num_cols == 2
|
||
grid_texts = [[cell.text for cell in row] for row in t2.grid]
|
||
assert grid_texts == [["H1", "H2"], ["C1", "C2"]]
|
||
|
||
|
||
def test_roundtrip_table_with_caption_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Tbl 1", prov=_default_prov())
|
||
td = TableData(num_rows=0, num_cols=2)
|
||
td.add_row(["H1", "H2"]) # header row
|
||
td.add_row(["C1", "C2"]) # data row
|
||
doc.add_table(data=td, caption=cap, prov=_default_prov())
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print(dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
assert len(doc2.tables) == 1
|
||
t2 = doc2.tables[0]
|
||
assert t2.data.num_rows == 2 and t2.data.num_cols == 2
|
||
grid_texts = [[cell.text for cell in row] for row in t2.data.grid]
|
||
assert grid_texts == [["H1", "H2"], ["C1", "C2"]]
|
||
|
||
assert len(t2.captions) == 1
|
||
cap_item = t2.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION and cap_item.text == "Tbl 1"
|
||
|
||
|
||
def test_roundtrip_complex_table_with_caption_prov():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Tbl 1", prov=_default_prov())
|
||
td = TableData(num_rows=3, num_cols=4)
|
||
|
||
cell_0_0 = TableCell(
|
||
row_span=1,
|
||
col_span=1,
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
text="H 0-0",
|
||
)
|
||
td.table_cells.append(cell_0_0)
|
||
|
||
cell_0_1 = TableCell(
|
||
row_span=1,
|
||
col_span=3,
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=4,
|
||
text="H 1-4",
|
||
)
|
||
td.table_cells.append(cell_0_1)
|
||
|
||
cell_1_0 = TableCell(
|
||
row_span=2,
|
||
col_span=1,
|
||
start_row_offset_idx=1,
|
||
end_row_offset_idx=3,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
text="R 1-3",
|
||
)
|
||
td.table_cells.append(cell_1_0)
|
||
|
||
cell_1_1 = TableCell(
|
||
row_span=2,
|
||
col_span=3,
|
||
start_row_offset_idx=1,
|
||
end_row_offset_idx=3,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=4,
|
||
text="R 2-2",
|
||
)
|
||
td.table_cells.append(cell_1_1)
|
||
|
||
doc.add_table(data=td, caption=cap, prov=_default_prov())
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print(dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
assert len(doc2.tables) == 1
|
||
t2 = doc2.tables[0]
|
||
assert t2.data.num_rows == 3 and t2.data.num_cols == 4
|
||
grid_texts = [[cell.text for cell in row] for row in t2.data.grid]
|
||
assert grid_texts == [
|
||
["H 0-0", "H 1-4", "H 1-4", "H 1-4"],
|
||
["R 1-3", "R 2-2", "R 2-2", "R 2-2"],
|
||
["R 1-3", "R 2-2", "R 2-2", "R 2-2"],
|
||
]
|
||
|
||
assert len(t2.captions) == 1
|
||
cap_item = t2.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION and cap_item.text == "Tbl 1"
|
||
|
||
|
||
def test_roundtrip_nested_list_unordered_in_unordered():
|
||
"""Test nested unordered list within unordered list."""
|
||
doc = DoclingDocument(name="t")
|
||
lg_outer = doc.add_list_group()
|
||
doc.add_list_item("Outer Item 1", parent=lg_outer, enumerated=False)
|
||
|
||
# Create nested list
|
||
lg_inner = doc.add_list_group(parent=lg_outer)
|
||
doc.add_list_item("Inner Item 1.1", parent=lg_inner, enumerated=False)
|
||
doc.add_list_item("Inner Item 1.2", parent=lg_inner, enumerated=False)
|
||
|
||
doc.add_list_item("Outer Item 2", parent=lg_outer, enumerated=False)
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Should have 2 groups (outer and inner)
|
||
assert len(doc2.groups) == 2
|
||
# Should have 4 text items total (1 ListItem for "Outer Item 1" + 2 inner ListItems + 1 outer ListItem)
|
||
assert len(doc2.texts) == 4
|
||
# Verify text content
|
||
text_contents = [it.text for it in doc2.texts]
|
||
assert "Outer Item 1" in text_contents
|
||
assert "Inner Item 1.1" in text_contents
|
||
assert "Inner Item 1.2" in text_contents
|
||
assert "Outer Item 2" in text_contents
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_deserialize_locationless_list_with_empty_item_and_formula():
|
||
doc = _deserialize(
|
||
"""
|
||
<doclang>
|
||
<list>
|
||
<ldiv/>
|
||
<ldiv/>
|
||
A.
|
||
<formula>f(x)</formula>
|
||
is odd
|
||
</list>
|
||
</doclang>
|
||
"""
|
||
)
|
||
|
||
assert [item.text for item in doc.texts] == ["", "", "A. ", "f(x)", " is odd"]
|
||
assert doc.texts[3].label == DocItemLabel.FORMULA
|
||
assert all(not item.prov for item in doc.texts)
|
||
assert "<location" not in _serialize(doc)
|
||
|
||
|
||
def test_roundtrip_nested_list_ordered_in_ordered():
|
||
"""Test nested ordered list within ordered list."""
|
||
doc = DoclingDocument(name="t")
|
||
lg_outer = doc.add_list_group()
|
||
doc.add_list_item("Step 1", parent=lg_outer, enumerated=False)
|
||
|
||
# Create nested ordered list
|
||
lg_inner = doc.add_list_group(parent=lg_outer)
|
||
doc.add_list_item("Step 1.1", parent=lg_inner, enumerated=True)
|
||
doc.add_list_item("Step 1.2", parent=lg_inner, enumerated=True)
|
||
|
||
doc.add_list_item("Step 2", parent=lg_outer, enumerated=True)
|
||
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
assert dt2.strip() == dt.strip()
|
||
|
||
|
||
def test_roundtrip_nested_list_ordered_in_unordered():
|
||
"""Test nested ordered list within unordered list."""
|
||
doc = DoclingDocument(name="t")
|
||
lg_outer = doc.add_list_group()
|
||
doc.add_list_item("Bullet A", parent=lg_outer, enumerated=False)
|
||
|
||
# Create nested ordered list
|
||
lg_inner = doc.add_list_group(parent=lg_outer)
|
||
doc.add_list_item("Numbered 1", parent=lg_inner, enumerated=True)
|
||
doc.add_list_item("Numbered 2", parent=lg_inner, enumerated=True)
|
||
|
||
doc.add_list_item("Bullet B", parent=lg_outer, enumerated=False)
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
assert len(doc2.groups) == 2
|
||
assert len(doc2.texts) == 4
|
||
text_contents = [it.text for it in doc2.texts]
|
||
assert "Bullet A" in text_contents
|
||
assert "Numbered 1" in text_contents
|
||
assert "Numbered 2" in text_contents
|
||
assert "Bullet B" in text_contents
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_roundtrip_nested_list_unordered_in_ordered():
|
||
"""Test nested unordered list within ordered list."""
|
||
doc = DoclingDocument(name="t")
|
||
lg_outer = doc.add_list_group()
|
||
doc.add_list_item("Step 1", parent=lg_outer, enumerated=True)
|
||
|
||
# Create nested unordered list
|
||
lg_inner = doc.add_list_group(parent=lg_outer)
|
||
doc.add_list_item("Bullet point", parent=lg_inner, enumerated=False)
|
||
doc.add_list_item("Another bullet", parent=lg_inner, enumerated=False)
|
||
|
||
doc.add_list_item("Step 2", parent=lg_outer, enumerated=True)
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
assert len(doc2.groups) == 2
|
||
assert len(doc2.texts) == 4
|
||
text_contents = [it.text for it in doc2.texts]
|
||
assert "Step 1" in text_contents
|
||
assert "Bullet point" in text_contents
|
||
assert "Another bullet" in text_contents
|
||
assert "Step 2" in text_contents
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_roundtrip_deeply_nested_list():
|
||
"""Test deeply nested lists (3 levels)."""
|
||
doc = DoclingDocument(name="t")
|
||
|
||
# Level 1
|
||
lg_level1 = doc.add_list_group()
|
||
doc.add_list_item("Level 1 Item 1", parent=lg_level1, enumerated=False)
|
||
|
||
# Level 2 (nested in level 1)
|
||
lg_level2 = doc.add_list_group(parent=lg_level1)
|
||
doc.add_list_item("Level 2 Item 1", parent=lg_level2, enumerated=False)
|
||
|
||
# Level 3 (nested in level 2)
|
||
lg_level3 = doc.add_list_group(parent=lg_level2)
|
||
doc.add_list_item("Level 3 Item 1", parent=lg_level3, enumerated=False)
|
||
doc.add_list_item("Level 3 Item 2", parent=lg_level3, enumerated=False)
|
||
|
||
doc.add_list_item("Level 2 Item 2", parent=lg_level2, enumerated=False)
|
||
|
||
doc.add_list_item("Level 1 Item 2", parent=lg_level1, enumerated=False)
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Should have 3 groups (3 levels)
|
||
assert len(doc2.groups) == 3
|
||
# Should have 6 text items total
|
||
assert len(doc2.texts) == 6
|
||
text_contents = [it.text for it in doc2.texts]
|
||
assert "Level 1 Item 1" in text_contents
|
||
assert "Level 2 Item 1" in text_contents
|
||
assert "Level 3 Item 1" in text_contents
|
||
assert "Level 3 Item 2" in text_contents
|
||
assert "Level 2 Item 2" in text_contents
|
||
assert "Level 1 Item 2" in text_contents
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_roundtrip_multiple_nested_lists_same_level():
|
||
"""Test multiple nested lists at the same level."""
|
||
doc = DoclingDocument(name="t")
|
||
|
||
lg_outer = doc.add_list_group()
|
||
doc.add_list_item("Item 1", parent=lg_outer, enumerated=False)
|
||
|
||
# First nested list
|
||
lg_inner1 = doc.add_list_group(parent=lg_outer)
|
||
doc.add_list_item("Nested 1.1", parent=lg_inner1, enumerated=False)
|
||
doc.add_list_item("Nested 1.2", parent=lg_inner1, enumerated=False)
|
||
|
||
doc.add_list_item("Item 2", parent=lg_outer, enumerated=False)
|
||
|
||
# Second nested list
|
||
lg_inner2 = doc.add_list_group(parent=lg_outer)
|
||
doc.add_list_item("Nested 2.1", parent=lg_inner2, enumerated=False)
|
||
doc.add_list_item("Nested 2.2", parent=lg_inner2, enumerated=False)
|
||
|
||
doc.add_list_item("Item 3", parent=lg_outer, enumerated=False)
|
||
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Should have 3 groups (1 outer, 2 inner)
|
||
assert len(doc2.groups) == 3
|
||
# Should have 7 text items total
|
||
assert len(doc2.texts) == 7
|
||
text_contents = [it.text for it in doc2.texts]
|
||
assert "Item 1" in text_contents
|
||
assert "Nested 1.1" in text_contents
|
||
assert "Nested 1.2" in text_contents
|
||
assert "Item 2" in text_contents
|
||
assert "Nested 2.1" in text_contents
|
||
assert "Nested 2.2" in text_contents
|
||
assert "Item 3" in text_contents
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_roundtrip_list_item_with_inline_group():
|
||
doc = DoclingDocument(name="t")
|
||
|
||
add_texts_section(doc)
|
||
add_list_section(doc)
|
||
|
||
dt = _serialize(doc)
|
||
exp_ser_file = Path(__file__).parent / "data" / "doc" / "roundtrip_list_item_with_inline_serialized.dclg.xml"
|
||
verify(exp_ser_file, dt)
|
||
doc2 = _deserialize(dt)
|
||
deser_yaml_file = Path(__file__).parent / "data" / "doc" / "roundtrip_list_item_with_inline_deserialized.yaml"
|
||
doc2.save_as_yaml(deser_yaml_file)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_dt2_file = Path(__file__).parent / "data" / "doc" / "roundtrip_list_item_with_inline_reserialized.dclg.xml"
|
||
verify(exp_dt2_file, dt2)
|
||
|
||
|
||
def test_deserialize_bare_picture():
|
||
dt = "<doclang><picture></picture></doclang>"
|
||
doc = _deserialize(dt)
|
||
|
||
assert len(doc.pictures) == 1
|
||
|
||
|
||
def test_deserialize_bare_table():
|
||
dt = "<doclang><table></table></doclang>"
|
||
doc = _deserialize(dt)
|
||
|
||
assert len(doc.tables) == 1
|
||
assert doc.tables[0].data.num_rows == 0
|
||
assert doc.tables[0].data.num_cols == 0
|
||
|
||
|
||
def test_roundtrip_table_with_caption_and_footnotes():
|
||
"""Test table with caption and multiple footnotes."""
|
||
doc = DoclingDocument(name="t")
|
||
|
||
# Create caption and footnotes
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Table 1: Sample Data")
|
||
|
||
# Create a simple 2x2 table
|
||
td = TableData(num_rows=0, num_cols=2)
|
||
td.add_row(["Header 1", "Header 2"])
|
||
td.add_row(["Data 1", "Data 2"])
|
||
|
||
# Add table with caption
|
||
table = doc.add_table(data=td, caption=cap)
|
||
|
||
footnote1 = doc.add_text(label=DocItemLabel.FOOTNOTE, text="First footnote")
|
||
footnote2 = doc.add_text(label=DocItemLabel.FOOTNOTE, text="Second footnote")
|
||
footnote3 = doc.add_text(label=DocItemLabel.FOOTNOTE, text="Third footnote")
|
||
|
||
# Add footnotes to the table
|
||
table.footnotes.append(footnote1.get_ref())
|
||
table.footnotes.append(footnote2.get_ref())
|
||
table.footnotes.append(footnote3.get_ref())
|
||
|
||
# Serialize and deserialize
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt2:", dt2)
|
||
|
||
# Verify table structure
|
||
assert len(doc2.tables) == 1
|
||
t2 = doc2.tables[0]
|
||
assert t2.data.num_rows == 2 and t2.data.num_cols == 2
|
||
grid_texts = [[cell.text for cell in row] for row in t2.data.grid]
|
||
assert grid_texts == [["Header 1", "Header 2"], ["Data 1", "Data 2"]]
|
||
|
||
# Verify caption
|
||
assert len(t2.captions) == 1
|
||
cap_item = t2.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION
|
||
assert cap_item.text == "Table 1: Sample Data"
|
||
|
||
# Verify footnotes
|
||
assert len(t2.footnotes) == 3
|
||
footnote_texts = [fn.resolve(doc2).text for fn in t2.footnotes]
|
||
assert footnote_texts == ["First footnote", "Second footnote", "Third footnote"]
|
||
|
||
# Verify all footnotes have the correct label
|
||
for fn_ref in t2.footnotes:
|
||
fn_item = fn_ref.resolve(doc2)
|
||
assert fn_item.label == DocItemLabel.FOOTNOTE
|
||
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_roundtrip_picture_with_caption_and_footnotes():
|
||
"""Test picture with caption and multiple footnotes."""
|
||
doc = DoclingDocument(name="t")
|
||
|
||
# Create caption and footnotes
|
||
cap = doc.add_text(label=DocItemLabel.CAPTION, text="Figure 1: Sample Image")
|
||
footnote1 = doc.add_text(label=DocItemLabel.FOOTNOTE, text="Image source: Dataset A")
|
||
footnote2 = doc.add_text(label=DocItemLabel.FOOTNOTE, text="Resolution: 1024x768")
|
||
|
||
# Add picture with caption
|
||
picture = doc.add_picture(caption=cap)
|
||
|
||
# Add footnotes to the picture
|
||
picture.footnotes.append(footnote1.get_ref())
|
||
picture.footnotes.append(footnote2.get_ref())
|
||
|
||
# Serialize and deserialize
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Verify picture exists
|
||
assert len(doc2.pictures) == 1
|
||
pic = doc2.pictures[0]
|
||
|
||
# Verify caption
|
||
assert len(pic.captions) == 1
|
||
cap_item = pic.captions[0].resolve(doc2)
|
||
assert cap_item.label == DocItemLabel.CAPTION
|
||
assert cap_item.text == "Figure 1: Sample Image"
|
||
|
||
# Verify footnotes
|
||
assert len(pic.footnotes) == 2
|
||
footnote_texts = [fn.resolve(doc2).text for fn in pic.footnotes]
|
||
assert footnote_texts == ["Image source: Dataset A", "Resolution: 1024x768"]
|
||
|
||
# Verify all footnotes have the correct label
|
||
for fn_ref in pic.footnotes:
|
||
fn_item = fn_ref.resolve(doc2)
|
||
assert fn_item.label == DocItemLabel.FOOTNOTE
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_roundtrip_table_with_rich_cells():
|
||
"""Test table with RichTableCells containing paragraphs, lists, and nested tables."""
|
||
doc = DoclingDocument(name="t")
|
||
|
||
# Create the main table (3x2 grid)
|
||
table = doc.add_table(data=TableData(num_rows=3, num_cols=2))
|
||
|
||
# Create content for rich cells
|
||
# Cell (0, 0): Multiple paragraphs
|
||
para1 = doc.add_text(parent=table, label=DocItemLabel.TEXT, text="First paragraph in cell.")
|
||
doc.add_text(parent=table, label=DocItemLabel.TEXT, text="Second paragraph in cell.")
|
||
|
||
# Cell (1, 0): A list
|
||
list_group = doc.add_list_group(parent=table)
|
||
doc.add_list_item(parent=list_group, text="List item 1", enumerated=False)
|
||
doc.add_list_item(parent=list_group, text="List item 2", enumerated=False)
|
||
doc.add_list_item(parent=list_group, text="List item 3", enumerated=False)
|
||
|
||
# Cell (2, 1): A nested table
|
||
nested_table_data = TableData(num_rows=0, num_cols=2)
|
||
nested_table_data.add_row(["A1", "A2"])
|
||
nested_table_data.add_row(["B1", "B2"])
|
||
nested_table = doc.add_table(data=nested_table_data, parent=table)
|
||
|
||
# Create the table cells
|
||
# Row 0: Rich cell with paragraphs in (0,0), simple cell in (0,1)
|
||
rich_cell_0_0 = RichTableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
ref=para1.get_ref(), # Note: In a real scenario, we'd want a group containing both paragraphs
|
||
text="cell 0,0",
|
||
)
|
||
doc.add_table_cell(table_item=table, cell=rich_cell_0_0)
|
||
|
||
simple_cell_0_1 = TableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=1,
|
||
text="Simple cell 0,1",
|
||
)
|
||
doc.add_table_cell(table_item=table, cell=simple_cell_0_1)
|
||
|
||
# Row 1: Rich cell with list in (1,0), simple cell in (1,1)
|
||
rich_cell_1_0 = RichTableCell(
|
||
start_row_offset_idx=1,
|
||
end_row_offset_idx=2,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
ref=list_group.get_ref(),
|
||
text="cell 1,0",
|
||
)
|
||
doc.add_table_cell(table_item=table, cell=rich_cell_1_0)
|
||
|
||
simple_cell_1_1 = TableCell(
|
||
start_row_offset_idx=1,
|
||
end_row_offset_idx=2,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=1,
|
||
text="Simple cell 1,1",
|
||
)
|
||
doc.add_table_cell(table_item=table, cell=simple_cell_1_1)
|
||
|
||
# Row 2: Simple cell in (2,0), rich cell with nested table in (2,1)
|
||
simple_cell_2_0 = TableCell(
|
||
start_row_offset_idx=2,
|
||
end_row_offset_idx=3,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
text="Simple cell 2,0",
|
||
)
|
||
doc.add_table_cell(table_item=table, cell=simple_cell_2_0)
|
||
|
||
rich_cell_2_1 = RichTableCell(
|
||
start_row_offset_idx=2,
|
||
end_row_offset_idx=3,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=1,
|
||
ref=nested_table.get_ref(),
|
||
text="cell 2,1",
|
||
)
|
||
doc.add_table_cell(table_item=table, cell=rich_cell_2_1)
|
||
|
||
# Serialize and deserialize
|
||
dt = _serialize(doc)
|
||
if DO_PRINT:
|
||
print("\n", dt)
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Verify main table structure
|
||
assert len(doc2.tables) >= 1
|
||
main_table = doc2.tables[0]
|
||
assert main_table.data.num_rows == 3
|
||
assert main_table.data.num_cols == 2
|
||
|
||
# Verify that we have rich table cells
|
||
rich_cells = [cell for cell in main_table.data.table_cells if isinstance(cell, RichTableCell)]
|
||
assert len(rich_cells) >= 1 # At least one rich cell should be preserved
|
||
|
||
# Verify round-trip serialization
|
||
dt2 = _serialize(doc2)
|
||
if DO_PRINT:
|
||
print("\ndt:", dt)
|
||
print("\ndt2:", dt2)
|
||
assert dt2 == dt
|
||
|
||
|
||
def _create_virtual_text_list_doc() -> DoclingDocument:
|
||
"""Document exercising virtual vs explicit ``<text>`` in list items."""
|
||
doc = DoclingDocument(name="virtual_text_list")
|
||
_add_default_page(doc)
|
||
bbox = _VirtualTextMixedBboxFactory()
|
||
|
||
lg = doc.add_list_group()
|
||
doc.add_list_item(text="plain list item", parent=lg, prov=bbox.next_prov())
|
||
|
||
li_inline = doc.add_list_item(text="", parent=lg, prov=bbox.next_prov())
|
||
inline = doc.add_inline_group(parent=li_inline)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="this is a ", parent=inline)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="bold",
|
||
parent=inline,
|
||
formatting=Formatting(bold=True),
|
||
)
|
||
doc.add_text(label=DocItemLabel.TEXT, text=" text", parent=inline)
|
||
|
||
li_nested = doc.add_list_item(
|
||
text="list item with nested list",
|
||
parent=lg,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
lg_sub = doc.add_list_group(parent=li_nested)
|
||
doc.add_list_item(text="nested list item", parent=lg_sub, prov=bbox.next_prov())
|
||
|
||
li_picture = doc.add_list_item(
|
||
text="list item with picture",
|
||
parent=lg,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
doc.add_picture(parent=li_picture, prov=bbox.next_prov())
|
||
|
||
return doc
|
||
|
||
|
||
def _assert_virtual_text_list_dclg(dclg: str) -> None:
|
||
"""Sanity-check virtual vs explicit ``<text>`` in list output."""
|
||
assert "<ldiv/>" in dclg
|
||
assert "plain list item" in dclg
|
||
assert "<text>plain list item</text>" not in dclg
|
||
assert "<text>" in dclg
|
||
assert "this is a " in dclg
|
||
assert "<bold>bold</bold>" in dclg
|
||
assert " text" in dclg
|
||
assert re.search(
|
||
r"<ldiv/>\s*(?:<location[^>]*/>\s*)*this is a <bold>bold</bold> text",
|
||
dclg,
|
||
)
|
||
assert not re.search(
|
||
r"<text>\s*(?:<location[^>]*/>\s*)*this is a <bold>bold</bold>",
|
||
dclg,
|
||
)
|
||
assert "list item with nested list" in dclg
|
||
assert "list item with picture" in dclg
|
||
assert re.search(
|
||
r"<text>\s*(?:<location[^>]*/>\s*)*list item with picture\s*</text>",
|
||
dclg,
|
||
)
|
||
assert "<picture" in dclg
|
||
assert "<location value=" in dclg
|
||
|
||
|
||
@doclang_validator
|
||
def test_virtual_text_list_roundtrip():
|
||
"""Round-trip list virtual-text edge cases through DocLang."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / "virtual_text_list"
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = _create_virtual_text_list_doc()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
|
||
dt = _serialize_virtual_text_mixed(doc)
|
||
verify(serialized_dclg, dt)
|
||
_assert_virtual_text_list_dclg(dt)
|
||
assert_valid_dclg_xml(dt)
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize_virtual_text_mixed(doc2, add_location=True)
|
||
verify(reserialized_dclg, dt2)
|
||
_assert_virtual_text_list_dclg(dt2)
|
||
assert_valid_dclg_xml(dt2)
|
||
|
||
|
||
@doclang_validator
|
||
def test_virtual_text_table_roundtrip():
|
||
"""Round-trip table virtual-text edge cases through DocLang."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / "virtual_text_table"
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = _create_virtual_text_table_doc()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
|
||
dt = _serialize_virtual_text_mixed(doc)
|
||
verify(serialized_dclg, dt)
|
||
_assert_virtual_text_table_dclg(dt)
|
||
assert_valid_dclg_xml(dt)
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize_virtual_text_mixed(doc2, add_location=True)
|
||
verify(reserialized_dclg, dt2)
|
||
_assert_virtual_text_table_dclg(dt2)
|
||
assert_valid_dclg_xml(dt2)
|
||
|
||
|
||
@doclang_validator
|
||
def test_virtual_text_index_roundtrip():
|
||
"""Round-trip index virtual-text edge cases through DocLang."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / "virtual_text_index"
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = _create_virtual_text_index_doc()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
|
||
dt = _serialize_virtual_text_mixed(doc)
|
||
verify(serialized_dclg, dt)
|
||
_assert_virtual_text_index_dclg(dt)
|
||
assert_valid_dclg_xml(dt)
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize_virtual_text_mixed(doc2, add_location=True)
|
||
verify(reserialized_dclg, dt2)
|
||
_assert_virtual_text_index_dclg(dt2)
|
||
assert_valid_dclg_xml(dt2)
|
||
|
||
|
||
@doclang_validator
|
||
def test_multi_page_roundtrip():
|
||
"""Round-trip a programmatic multi-page document through DocLang.
|
||
|
||
Page 1: title; page 2: document index; page 3: three text paragraphs.
|
||
Materializes serialized/reserialized XML and input/deserialized JSON goldens.
|
||
"""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / "multi_page_roundtrip"
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = _create_multi_page_roundtrip_doc()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
|
||
dt = _serialize(doc)
|
||
verify(serialized_dclg, dt)
|
||
assert_valid_dclg_xml(dt)
|
||
assert dt.count("<page_break") == 2
|
||
assert "<index>" in dt
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize(doc2)
|
||
verify(reserialized_dclg, dt2)
|
||
assert_valid_dclg_xml(dt2)
|
||
|
||
|
||
def _create_multi_page_roundtrip_doc() -> DoclingDocument:
|
||
"""Build a three-page document with title, index, and body paragraphs."""
|
||
page_size = Size(width=512, height=512)
|
||
doc = DoclingDocument(name="multi_page_roundtrip")
|
||
for page_no in (1, 2, 3):
|
||
doc.add_page(page_no=page_no, size=page_size, image=None)
|
||
|
||
doc.add_title(
|
||
text="Document Title",
|
||
prov=_page_prov(page_no=1, bbox=(10, 10, 200, 40)),
|
||
)
|
||
|
||
index_data = TableData(num_cols=2)
|
||
index_data.add_row(["Chapter", "Page"])
|
||
index_data.add_row(["Intro", "1"])
|
||
index_data.add_row(["Body", "3"])
|
||
doc.add_table(
|
||
data=index_data,
|
||
label=DocItemLabel.DOCUMENT_INDEX,
|
||
prov=_page_prov(page_no=2, bbox=(10, 10, 400, 120)),
|
||
)
|
||
|
||
for i, text in enumerate(["First paragraph.", "Second paragraph.", "Third paragraph."], start=1):
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text=text,
|
||
prov=_page_prov(page_no=3, bbox=(10, 40 + i * 30, 400, 60 + i * 30)),
|
||
)
|
||
return doc
|
||
|
||
|
||
def _page_prov(*, page_no: int, bbox: tuple[float, float, float, float]) -> ProvenanceItem:
|
||
return ProvenanceItem(
|
||
page_no=page_no,
|
||
bbox=BoundingBox.from_tuple(bbox, origin=CoordOrigin.TOPLEFT),
|
||
charspan=(0, 0),
|
||
)
|
||
|
||
|
||
def _create_virtual_text_table_doc() -> DoclingDocument:
|
||
"""Document exercising virtual vs explicit ``<text>`` in table cells."""
|
||
doc = DoclingDocument(name="virtual_text_table")
|
||
_add_default_page(doc)
|
||
bbox = _VirtualTextMixedBboxFactory()
|
||
|
||
def _add_row_table(*, label: DocItemLabel) -> None:
|
||
table = doc.add_table(data=TableData(num_rows=1, num_cols=3), label=label, prov=bbox.next_prov())
|
||
|
||
cell_group = doc.add_group(parent=table, label=GroupLabel.UNSPECIFIED)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="cell with list:",
|
||
parent=cell_group,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
cell_list = doc.add_list_group(parent=cell_group)
|
||
doc.add_list_item(text="nested in cell", parent=cell_list, prov=bbox.next_prov())
|
||
|
||
rich_group = doc.add_group(parent=table, label=GroupLabel.UNSPECIFIED)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="group text",
|
||
parent=rich_group,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
doc.add_picture(parent=rich_group, prov=bbox.next_prov())
|
||
|
||
doc.add_table_cell(
|
||
table_item=table,
|
||
cell=TableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
text="plain cell",
|
||
bbox=bbox.next_bbox(),
|
||
),
|
||
)
|
||
doc.add_table_cell(
|
||
table_item=table,
|
||
cell=RichTableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=2,
|
||
ref=cell_group.get_ref(),
|
||
text="cell with list:",
|
||
bbox=bbox.next_bbox(),
|
||
),
|
||
)
|
||
doc.add_table_cell(
|
||
table_item=table,
|
||
cell=RichTableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=2,
|
||
end_col_offset_idx=3,
|
||
ref=rich_group.get_ref(),
|
||
text="group cell",
|
||
bbox=bbox.next_bbox(),
|
||
),
|
||
)
|
||
|
||
_add_row_table(label=DocItemLabel.TABLE)
|
||
return doc
|
||
|
||
|
||
def _create_virtual_text_index_doc() -> DoclingDocument:
|
||
"""Document exercising virtual vs explicit ``<text>`` in index cells."""
|
||
doc = DoclingDocument(name="virtual_text_index")
|
||
_add_default_page(doc)
|
||
bbox = _VirtualTextMixedBboxFactory()
|
||
|
||
table = doc.add_table(
|
||
data=TableData(num_rows=1, num_cols=3),
|
||
label=DocItemLabel.DOCUMENT_INDEX,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
|
||
cell_group = doc.add_group(parent=table, label=GroupLabel.UNSPECIFIED)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="cell with list:",
|
||
parent=cell_group,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
cell_list = doc.add_list_group(parent=cell_group)
|
||
doc.add_list_item(text="nested in cell", parent=cell_list, prov=bbox.next_prov())
|
||
|
||
rich_group = doc.add_group(parent=table, label=GroupLabel.UNSPECIFIED)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="group text",
|
||
parent=rich_group,
|
||
prov=bbox.next_prov(),
|
||
)
|
||
doc.add_picture(parent=rich_group, prov=bbox.next_prov())
|
||
|
||
doc.add_table_cell(
|
||
table_item=table,
|
||
cell=TableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
text="plain cell",
|
||
bbox=bbox.next_bbox(),
|
||
),
|
||
)
|
||
doc.add_table_cell(
|
||
table_item=table,
|
||
cell=RichTableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=1,
|
||
end_col_offset_idx=2,
|
||
ref=cell_group.get_ref(),
|
||
text="cell with list:",
|
||
bbox=bbox.next_bbox(),
|
||
),
|
||
)
|
||
doc.add_table_cell(
|
||
table_item=table,
|
||
cell=RichTableCell(
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=2,
|
||
end_col_offset_idx=3,
|
||
ref=rich_group.get_ref(),
|
||
text="group cell",
|
||
bbox=bbox.next_bbox(),
|
||
),
|
||
)
|
||
|
||
return doc
|
||
|
||
|
||
def _assert_virtual_text_table_dclg(dclg: str) -> None:
|
||
host_block = dclg.split("<table>", 1)[1].split("</table>", 1)[0]
|
||
assert "plain cell" in host_block
|
||
assert "<text>plain cell</text>" not in host_block
|
||
assert "cell with list:" in host_block
|
||
assert "<text>" in host_block
|
||
assert "group text" in host_block
|
||
assert "<picture" in host_block
|
||
assert "<location value=" in host_block
|
||
|
||
|
||
def _assert_virtual_text_index_dclg(dclg: str) -> None:
|
||
host_block = dclg.split("<index>", 1)[1].split("</index>", 1)[0]
|
||
assert "plain cell" in host_block
|
||
assert "<text>plain cell</text>" not in host_block
|
||
assert "cell with list:" in host_block
|
||
assert "<text>" in host_block
|
||
assert "group text" in host_block
|
||
assert "<picture" in host_block
|
||
assert "<location value=" in host_block
|
||
|
||
|
||
def _create_referenced_caption_doc() -> DoclingDocument:
|
||
"""Document with picture and table, each with an associated caption."""
|
||
doc = DoclingDocument(name="referenced_caption")
|
||
_add_default_page(doc)
|
||
bbox = _VirtualTextMixedBboxFactory()
|
||
|
||
cap_pic = doc.add_text(
|
||
label=DocItemLabel.CAPTION,
|
||
text="Figure 1",
|
||
prov=bbox.next_prov(),
|
||
)
|
||
doc.add_picture(caption=cap_pic, prov=bbox.next_prov())
|
||
|
||
cap_tbl = doc.add_text(
|
||
label=DocItemLabel.CAPTION,
|
||
text="Table 1",
|
||
prov=bbox.next_prov(),
|
||
)
|
||
td = TableData(num_rows=0, num_cols=2)
|
||
td.add_row(["H1", "H2"])
|
||
td.add_row(["C1", "C2"])
|
||
doc.add_table(data=td, caption=cap_tbl, prov=bbox.next_prov())
|
||
|
||
return doc
|
||
|
||
|
||
def _assert_referenced_caption_doc(doc: DoclingDocument) -> None:
|
||
"""Verify picture/table caption refs are preserved in the document model."""
|
||
assert len(doc.pictures) == 1
|
||
pic = doc.pictures[0]
|
||
assert len(pic.captions) == 1
|
||
pic_cap = pic.captions[0].resolve(doc)
|
||
assert pic_cap.label == DocItemLabel.CAPTION
|
||
assert pic_cap.text == "Figure 1"
|
||
|
||
assert len(doc.tables) == 1
|
||
tbl = doc.tables[0]
|
||
assert len(tbl.captions) == 1
|
||
tbl_cap = tbl.captions[0].resolve(doc)
|
||
assert tbl_cap.label == DocItemLabel.CAPTION
|
||
assert tbl_cap.text == "Table 1"
|
||
grid_texts = [[cell.text for cell in row] for row in tbl.data.grid]
|
||
assert grid_texts == [["H1", "H2"], ["C1", "C2"]]
|
||
|
||
|
||
def _assert_referenced_caption_dclg(dclg: str, *, with_location: bool) -> None:
|
||
"""Sanity-check referenced captions appear in host element heads."""
|
||
pic_block = dclg.split("<picture>", 1)[1].split("</picture>", 1)[0]
|
||
assert "Figure 1" in pic_block
|
||
assert re.search(r"<caption>\s*(?:<location[^>]*/>\s*)*Figure 1\s*</caption>", pic_block)
|
||
|
||
tbl_block = dclg.split("<table>", 1)[1].split("</table>", 1)[0]
|
||
assert "Table 1" in tbl_block
|
||
assert re.search(r"<caption>\s*(?:<location[^>]*/>\s*)*Table 1\s*</caption>", tbl_block)
|
||
assert "H1" in tbl_block and "C2" in tbl_block
|
||
|
||
if with_location:
|
||
assert "<location value=" in dclg
|
||
|
||
|
||
@doclang_validator
|
||
def test_referenced_caption_roundtrip():
|
||
"""Round-trip picture/table with associated captions through DocLang."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / "referenced_caption"
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = _create_referenced_caption_doc()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
_assert_referenced_caption_doc(doc)
|
||
|
||
dt = _serialize_virtual_text_mixed(doc)
|
||
verify(serialized_dclg, dt)
|
||
_assert_referenced_caption_dclg(dt, with_location=True)
|
||
assert_valid_dclg_xml(dt)
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
_assert_referenced_caption_doc(doc2)
|
||
|
||
dt2 = _serialize_virtual_text_mixed(doc2, add_location=True)
|
||
verify(reserialized_dclg, dt2)
|
||
_assert_referenced_caption_dclg(dt2, with_location=True)
|
||
assert_valid_dclg_xml(dt2)
|
||
|
||
|
||
############################################
|
||
### Feature complete document test-cases ###
|
||
############################################
|
||
|
||
|
||
def test_constructed_doc(sample_doc: DoclingDocument):
|
||
doc = sample_doc
|
||
|
||
dt = _serialize(doc)
|
||
doc2 = _deserialize(dt)
|
||
dt2 = _serialize(doc2)
|
||
|
||
exp_reserialized_dt_file = Path(__file__).parent / "data" / "doc" / "constr_doc_reserialized.dclg.xml"
|
||
verify(exp_reserialized_dt_file, dt2)
|
||
|
||
|
||
def test_constructed_rich_table_doc(rich_table_doc: DoclingDocument):
|
||
doc = rich_table_doc
|
||
|
||
dt = _serialize(doc)
|
||
|
||
doc2 = _deserialize(dt)
|
||
|
||
dt2 = _serialize(doc2)
|
||
|
||
assert dt2 == dt
|
||
|
||
|
||
def test_wrapping():
|
||
dt = """
|
||
<doclang>
|
||
<text>simple</text>
|
||
<text><content> leading</content></text>
|
||
<text><content>trailing </content></text>
|
||
<text><![CDATA[< special]]></text>
|
||
<text><content><![CDATA[ leading and < special]]></content></text>
|
||
<text>
|
||
<location value="5"/>
|
||
<location value="492"/>
|
||
<location value="15"/>
|
||
<location value="502"/>
|
||
w/prov simple
|
||
</text>
|
||
<text>
|
||
<location value="5"/>
|
||
<location value="492"/>
|
||
<location value="15"/>
|
||
<location value="502"/>
|
||
<content> w/prov leading</content>
|
||
</text>
|
||
<text>
|
||
<location value="5"/>
|
||
<location value="492"/>
|
||
<location value="15"/>
|
||
<location value="502"/>
|
||
<content>w/prov trailing </content>
|
||
</text>
|
||
<text>
|
||
<location value="5"/>
|
||
<location value="492"/>
|
||
<location value="15"/>
|
||
<location value="502"/>
|
||
<![CDATA[w/prov < special]]>
|
||
</text>
|
||
<text>
|
||
<location value="5"/>
|
||
<location value="492"/>
|
||
<location value="15"/>
|
||
<location value="502"/>
|
||
<content><![CDATA[ w/prov leading and < special]]></content>
|
||
</text>
|
||
</doclang>
|
||
"""
|
||
doc = _deserialize(dt)
|
||
dt2 = _serialize(doc)
|
||
assert dt2.strip() == dt.strip()
|
||
|
||
|
||
def test_rich_table_cells():
|
||
dt = """
|
||
<doclang>
|
||
<table>
|
||
<fcel/>
|
||
<text>foo</text>
|
||
<fcel/>
|
||
<text><italic>text in italic</italic></text>
|
||
<nl/>
|
||
<fcel/>
|
||
<table>
|
||
<fcel/>
|
||
<text>inner cell 0,0</text>
|
||
<fcel/>
|
||
<text>inner cell 0,1</text>
|
||
<nl/>
|
||
<fcel/>
|
||
<text>inner cell 1,0</text>
|
||
<fcel/>
|
||
<text>inner cell 1,1 <bold>in bold</bold></text>
|
||
<nl/>
|
||
</table>
|
||
<fcel/>
|
||
<text>bar</text>
|
||
<nl/>
|
||
</table>
|
||
</doclang>
|
||
"""
|
||
doc = _deserialize(dt)
|
||
dt2 = _serialize(doc)
|
||
assert dt2.strip() == dt.strip()
|
||
|
||
|
||
def test_picture_tabular_chart_content_cdata_cells():
|
||
"""Deserializer must extract text from <content><![CDATA[...]]></content> in OTSL cells."""
|
||
doclang = """<doclang><group><picture class="chart"><location value="0"/><location value="0"/><location value="511"/><location value="511"/><tabular><fcel/><content><![CDATA[Characteristic]]></content><fcel/><content><![CDATA[Player expenses in million U.S. dollars]]></content><nl/><fcel/><content><![CDATA[19/20]]></content><fcel/><content><![CDATA[111]]></content><nl/></tabular></picture></group></doclang>"""
|
||
doc = _deserialize(doclang)
|
||
first_cell_text = doc.pictures[0].meta.tabular_chart.chart_data.grid[0][0].text
|
||
assert first_cell_text == "Characteristic"
|
||
assert doc.pictures[0].meta.tabular_chart.chart_data.grid[0][1].text == "Player expenses in million U.S. dollars"
|
||
assert doc.pictures[0].meta.tabular_chart.chart_data.grid[1][0].text == "19/20"
|
||
assert doc.pictures[0].meta.tabular_chart.chart_data.grid[1][1].text == "111"
|
||
|
||
|
||
@pytest.mark.parametrize("escape_mode", [EscapeMode.AUTO, EscapeMode.ALWAYS])
|
||
def test_roundtrip_otsl_xml_sensitive_cell_content(escape_mode: EscapeMode):
|
||
"""OTSL cell text remains exact when DOM fragments are reparsed."""
|
||
cell_texts = [
|
||
'Parsing (Structure&Semantic) <test> "quoted"',
|
||
"ampersand & less < greater >",
|
||
"\"quotes\" and 'apostrophes'",
|
||
"literal entity-like text: &",
|
||
"CDATA terminator: ]]>",
|
||
"multiline code:\nif a < b:\n return a & b",
|
||
"Unicode and math: café ∀x ∈ ℝ, x² ≥ 0", # noqa: RUF001
|
||
]
|
||
doc = DoclingDocument(name="otsl-xml-sensitive")
|
||
table_data = TableData(num_rows=0, num_cols=1)
|
||
for text in cell_texts:
|
||
table_data.add_row([text])
|
||
doc.add_table(data=table_data)
|
||
|
||
serialized = (
|
||
DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False, escape_mode=escape_mode),
|
||
)
|
||
.serialize()
|
||
.text
|
||
)
|
||
roundtripped = _deserialize(serialized)
|
||
|
||
assert [row[0].text for row in roundtripped.tables[0].data.grid] == cell_texts
|
||
|
||
|
||
def test_otsl_xml_sensitive_virtual_and_explicit_text_cells():
|
||
"""Both virtual and explicit rich OTSL cell bodies preserve safe text."""
|
||
doclang = """\
|
||
<doclang>
|
||
<table>
|
||
<fcel/><content><![CDATA[virtual & <cell> "quoted" 'apostrophe']]></content>
|
||
<fcel/><text><content><![CDATA[nested & <cell>]]></content><bold> bold & styled</bold></text>
|
||
<nl/>
|
||
</table>
|
||
</doclang>
|
||
"""
|
||
doc = _deserialize(doclang)
|
||
|
||
assert doc.tables[0].data.grid[0][0].text == "virtual & <cell> \"quoted\" 'apostrophe'"
|
||
# The source writes `<bold> bold & styled</bold>`; that leading space is a significant
|
||
# run boundary under the inline whitespace contract, so it survives.
|
||
assert doc.tables[0].data.grid[0][1].text == "nested & <cell> bold & styled"
|
||
assert isinstance(doc.tables[0].data.grid[0][1], RichTableCell)
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"literal",
|
||
[
|
||
"<none>",
|
||
"<none/>",
|
||
"<Month number>",
|
||
"<text>not literal</text>",
|
||
"<fcel/>",
|
||
],
|
||
)
|
||
def test_otsl_angle_delimited_text_remains_text(literal: str) -> None:
|
||
xml = f'<doclang version="0.7"><table><fcel/><content><![CDATA[{literal}]]></content><nl/></table></doclang>'
|
||
assert_valid_dclg_xml(xml)
|
||
|
||
cell = DocLangDocDeserializer().deserialize_str(xml).tables[0].data.grid[0][0]
|
||
|
||
assert type(cell) is TableCell
|
||
assert cell.text == literal
|
||
|
||
|
||
def test_otsl_real_text_element_remains_rich() -> None:
|
||
xml = '<doclang version="0.7"><table><fcel/><text>not literal</text><nl/></table></doclang>'
|
||
|
||
cell = DocLangDocDeserializer().deserialize_str(xml).tables[0].data.grid[0][0]
|
||
|
||
assert isinstance(cell, RichTableCell)
|
||
assert cell.text == "not literal"
|
||
|
||
|
||
def test_picture_body_table_is_semantic_content_not_chart_tabular():
|
||
"""``<table>`` after the preamble is nested picture content, not ``meta.tabular_chart``."""
|
||
doclang = (
|
||
'<doclang><picture class="chart">'
|
||
"<tabular><fcel/>Chart<fcel/>1<nl/></tabular>"
|
||
"<table><fcel/>Nested<fcel/>Cell<nl/></table>"
|
||
"</picture></doclang>"
|
||
)
|
||
doc = _deserialize(doclang)
|
||
pic = doc.pictures[0]
|
||
assert pic.meta is not None
|
||
assert pic.meta.tabular_chart.chart_data.grid[0][0].text == "Chart"
|
||
assert len(pic.children) == 1
|
||
nested = pic.children[0].resolve(doc)
|
||
assert isinstance(nested, TableItem)
|
||
assert nested.data.grid[0][0].text == "Nested"
|
||
|
||
|
||
# SMILES from tests/data/doc/dummy_doc_with_meta.yaml (molecule_data annotation)
|
||
_EXAMPLE_MOLECULE_SMILES = "CC1=NNC(C2=CN3C=CN=C3C(CC3=CC(F)=CC(F)=C3)=N2)=N1"
|
||
_PICTURE_META_SUMMARY = "Picture meta summary"
|
||
_PICTURE_META_DESCRIPTION = "Picture meta description"
|
||
_PICTURE_META_CUSTOM_VALUE = "custom field on picture meta"
|
||
_CHART_TITLE = "Chart Title"
|
||
|
||
|
||
def _create_picture_molecule_doc() -> DoclingDocument:
|
||
doc = DoclingDocument(name="picture_molecule_meta")
|
||
pic = doc.add_picture()
|
||
pic.meta = PictureMeta(
|
||
summary=SummaryMetaField(text=_PICTURE_META_SUMMARY),
|
||
description=DescriptionMetaField(text=_PICTURE_META_DESCRIPTION),
|
||
classification=PictureClassificationMetaField(
|
||
predictions=[
|
||
PictureClassificationPrediction(
|
||
class_name=PictureClassificationLabel.PIE_CHART.value,
|
||
confidence=1.0,
|
||
)
|
||
]
|
||
),
|
||
molecule=MoleculeMetaField(smi=_EXAMPLE_MOLECULE_SMILES),
|
||
)
|
||
pic.meta.set_custom_field(
|
||
namespace="my_corp",
|
||
name="note",
|
||
value=_PICTURE_META_CUSTOM_VALUE,
|
||
)
|
||
# Tabular chart data (pattern from test_serialization_doclang._create_content_filtering_doc)
|
||
chart_data = TableData(num_cols=2)
|
||
chart_data.add_row(["Foo", "Bar"])
|
||
chart_data.add_row(["One", "Two"])
|
||
pic.meta.tabular_chart = TabularChartMetaField(
|
||
title=_CHART_TITLE,
|
||
chart_data=chart_data,
|
||
)
|
||
return doc
|
||
|
||
|
||
_KV_ANNOT_ROOT = Path(__file__).parent / "data" / "doc" / "kv"
|
||
|
||
# KV annot fixtures with lossless DocLang XML round-trip today.
|
||
_KV_ANNOT_XML_LOSSLESS = frozenset(
|
||
{
|
||
"01d07afe1cb54ecd23eedfe4d91b81dd88e61bf4e0dbe2467784db4177a6c691",
|
||
"08212053e2db1a70dd60a4f85650ceb33d7519af34f502e3ac894389d76663d6",
|
||
"1eac20e5ac5fac655a611343f86927d6a76277e170430c1eba741585437a2e90",
|
||
"ba4120cada21304563625490e9ad13911e96114d3f07df056a6bf62397a859e1",
|
||
}
|
||
)
|
||
|
||
|
||
def _kv_annot_fixture_dirs() -> list[Path]:
|
||
"""Return migrated KV annot fixture dirs with serialized DocLang goldens."""
|
||
return sorted(
|
||
p
|
||
for p in _KV_ANNOT_ROOT.iterdir()
|
||
if p.is_dir() and (p / "output.json").is_file() and (p / "output.dclg.xml").is_file()
|
||
)
|
||
|
||
|
||
def _serialize_kv_annot_fixture(doc: DoclingDocument) -> str:
|
||
text = DocLangDocSerializer(doc=doc, params=DocLangParams(include_version=False)).serialize().text
|
||
if not GEN_TEST_DATA:
|
||
assert_valid_dclg_xml(text)
|
||
return text
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"fixture_dir",
|
||
_kv_annot_fixture_dirs(),
|
||
ids=[p.name for p in _kv_annot_fixture_dirs()],
|
||
)
|
||
def test_kv_annot_doclang_roundtrip(fixture_dir: Path):
|
||
"""Round-trip migrated KV annot fixtures through DocLang (see ``tests/data/doc/kv/``)."""
|
||
output_json = fixture_dir / "output.json"
|
||
serialized_dclg = fixture_dir / "output.dclg.xml"
|
||
deserialized_json = fixture_dir / "deserialized.json"
|
||
reserialized_dclg = fixture_dir / "reserialized.dclg.xml"
|
||
|
||
doc = DoclingDocument.load_from_json(output_json)
|
||
|
||
dt = _serialize_kv_annot_fixture(doc)
|
||
verify_doclang(exp_file=serialized_dclg, actual=dt)
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize_kv_annot_fixture(doc2)
|
||
verify_doclang(exp_file=reserialized_dclg, actual=dt2)
|
||
|
||
if fixture_dir.name in _KV_ANNOT_XML_LOSSLESS:
|
||
assert dt.strip() == dt2.strip()
|
||
|
||
|
||
def _serialize_field_region_fixture(doc: DoclingDocument, *, fixture_dir: str) -> str:
|
||
"""Serialize a field-region fixture doc (invoice uses placeholder pictures)."""
|
||
params = (
|
||
DocLangParams(include_version=False, image_mode=ImageRefMode.PLACEHOLDER)
|
||
if fixture_dir == "field_region_kv_invoice"
|
||
else DocLangParams(include_version=False)
|
||
)
|
||
text = DocLangDocSerializer(doc=doc, params=params).serialize().text
|
||
if not GEN_TEST_DATA:
|
||
assert_valid_dclg_xml(text)
|
||
return text
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"fixture_dir",
|
||
[
|
||
"field_region_kv_migration",
|
||
"field_region_kv",
|
||
"field_region_kv_invoice",
|
||
],
|
||
)
|
||
def test_field_region_doclang_roundtrip(fixture_dir: str):
|
||
"""Round-trip field regions/items through DocLang deserialization."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / fixture_dir
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = DoclingDocument.load_from_json(input_json)
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
assert doc.field_regions
|
||
|
||
dt = _serialize_field_region_fixture(doc, fixture_dir=fixture_dir)
|
||
verify_doclang(exp_file=serialized_dclg, actual=dt)
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
assert doc2.field_regions
|
||
|
||
dt2 = _serialize_field_region_fixture(doc2, fixture_dir=fixture_dir)
|
||
verify_doclang(exp_file=reserialized_dclg, actual=dt2)
|
||
assert dt.strip() == dt2.strip()
|
||
|
||
|
||
def test_picture_molecule_meta_roundtrip():
|
||
"""Round-trip picture meta (molecule, chart/tabular, summary, description, custom) through DocLang."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / "picture_molecule_meta"
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = _create_picture_molecule_doc()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
|
||
dt = _serialize(doc)
|
||
verify(serialized_dclg, dt)
|
||
assert f"<summary>{_PICTURE_META_SUMMARY}</summary>" in dt
|
||
assert f"<description>{_PICTURE_META_DESCRIPTION}</description>" in dt
|
||
assert f"<docling__smiles>{_EXAMPLE_MOLECULE_SMILES}</docling__smiles>" in dt
|
||
assert f"<my_corp__note>{_PICTURE_META_CUSTOM_VALUE}</my_corp__note>" in dt
|
||
assert 'class="chart"' in dt
|
||
assert 'value="pie_chart"' in dt
|
||
assert "<tabular>" in dt
|
||
assert "Foo" in dt and "Bar" in dt and "One" in dt and "Two" in dt
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize(doc2)
|
||
verify(reserialized_dclg, dt2)
|
||
|
||
|
||
def test_roundtrip_with_layers():
|
||
"""Test roundtrip with content layers."""
|
||
from docling_core.types.doc import ContentLayer
|
||
|
||
doc = DoclingDocument(name="t")
|
||
# Add items with different layers
|
||
doc.add_text(label=DocItemLabel.PAGE_HEADER, text="Header", content_layer=ContentLayer.FURNITURE)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="Body text", content_layer=ContentLayer.BODY)
|
||
doc.add_text(label=DocItemLabel.PAGE_FOOTER, text="Footer", content_layer=ContentLayer.FURNITURE)
|
||
|
||
# Serialize with ALWAYS mode to ensure layers are included
|
||
from docling_core.transforms.serializer.doclang import LayerMode
|
||
|
||
ser = DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False, layer_mode=LayerMode.ALWAYS),
|
||
)
|
||
dt = ser.serialize().text
|
||
|
||
# Deserialize
|
||
doc2 = _deserialize(dt)
|
||
|
||
# Verify layers are preserved
|
||
assert len(doc2.body.children) == 3
|
||
items = [doc2.body.children[i].resolve(doc2) for i in range(3)]
|
||
assert items[0].content_layer == ContentLayer.FURNITURE
|
||
assert items[1].content_layer == ContentLayer.BODY
|
||
assert items[2].content_layer == ContentLayer.FURNITURE
|
||
|
||
|
||
def test_roundtrip_with_newlines():
|
||
"""Test that newlines in <content> survive deserialization and reserialization."""
|
||
doclang_str = """
|
||
<doclang>
|
||
<text><content>foo
|
||
bar</content></text>
|
||
<text><content>zoo
|
||
</content><bold>zen</bold></text>
|
||
</doclang>"""
|
||
|
||
doc = _deserialize(doclang_str)
|
||
dt2 = _serialize(doc)
|
||
assert dt2.strip() == doclang_str.strip()
|
||
|
||
|
||
def test_roundtrip_document_index_table():
|
||
"""Test that DOCUMENT_INDEX label is preserved through serialization/deserialization."""
|
||
doc = DoclingDocument(name="test")
|
||
_add_default_page(doc)
|
||
|
||
# Add a regular table
|
||
table_data = TableData(num_cols=2)
|
||
table_data.add_row(["Header 1", "Header 2"])
|
||
table_data.grid[0][0].column_header = True
|
||
table_data.grid[0][1].column_header = True
|
||
table_data.add_row(["Data 1", "Data 2"])
|
||
doc.add_table(data=table_data, label=DocItemLabel.TABLE, prov=_default_prov())
|
||
|
||
# Add a DOCUMENT_INDEX table
|
||
index_data = TableData(num_cols=2)
|
||
index_data.add_row(["Index 1", "Page 1"])
|
||
index_data.add_row(["Index 2", "Page 2"])
|
||
doc.add_table(data=index_data, label=DocItemLabel.DOCUMENT_INDEX, prov=_default_prov())
|
||
|
||
# Serialize
|
||
xml_str = _serialize(doc)
|
||
|
||
assert "<index>" in xml_str
|
||
|
||
# Deserialize
|
||
doc2 = _deserialize(xml_str)
|
||
|
||
# Verify we have 2 tables
|
||
assert len(doc2.tables) == 2
|
||
|
||
# Verify labels are preserved
|
||
assert doc2.tables[0].label == DocItemLabel.TABLE
|
||
assert doc2.tables[1].label == DocItemLabel.DOCUMENT_INDEX
|
||
|
||
# Verify table data is preserved
|
||
assert doc2.tables[0].data.num_rows == 2
|
||
assert doc2.tables[0].data.num_cols == 2
|
||
assert doc2.tables[1].data.num_rows == 2
|
||
assert doc2.tables[1].data.num_cols == 2
|
||
|
||
|
||
def _thread_roundtrip(
|
||
*,
|
||
name: str,
|
||
doc_factory: Callable[[], DoclingDocument],
|
||
page_breaks: int = 0,
|
||
) -> None:
|
||
"""Round-trip a threaded document fixture through serialize → deserialize → reserialize."""
|
||
data_dir = Path(__file__).parent / "data" / "doc" / name
|
||
input_json = data_dir / "input.json"
|
||
serialized_dclg = data_dir / "serialized.dclg.xml"
|
||
deserialized_json = data_dir / "deserialized.json"
|
||
reserialized_dclg = data_dir / "reserialized.dclg.xml"
|
||
|
||
doc = doc_factory()
|
||
_verify_doc(doc=doc, exp_json=input_json)
|
||
|
||
dt = _serialize(doc)
|
||
verify(serialized_dclg, dt)
|
||
if page_breaks:
|
||
assert dt.count("<page_break") == page_breaks
|
||
|
||
doc2 = _deserialize(dt)
|
||
_verify_doc(doc=doc2, exp_json=deserialized_json)
|
||
|
||
dt2 = _serialize(doc2)
|
||
verify(reserialized_dclg, dt2)
|
||
|
||
|
||
@doclang_validator
|
||
def test_cross_page_paragraph_roundtrip():
|
||
"""Round-trip a cross-page threaded paragraph."""
|
||
_thread_roundtrip(name="cross_page_paragraph", doc_factory=_doc_cross_page_paragraph, page_breaks=1)
|
||
|
||
|
||
@doclang_validator
|
||
def test_cross_column_paragraph_roundtrip():
|
||
"""Round-trip a cross-column threaded paragraph (same page, two boxes)."""
|
||
_thread_roundtrip(name="multi_prov_thread", doc_factory=_doc_multi_prov_text, page_breaks=0)
|
||
|
||
|
||
@doclang_validator
|
||
def test_cross_page_list_roundtrip():
|
||
"""Round-trip a cross-page threaded list (whole items per page)."""
|
||
_thread_roundtrip(name="cross_page_list", doc_factory=_doc_cross_page_list, page_breaks=1)
|
||
|
||
|
||
@doclang_validator
|
||
def test_cross_page_table_roundtrip():
|
||
"""Round-trip a cross-page threaded table."""
|
||
_thread_roundtrip(name="cross_page_table", doc_factory=_doc_cross_page_table, page_breaks=1)
|
||
|
||
|
||
@doclang_validator
|
||
def test_cross_column_list_roundtrip():
|
||
"""Round-trip a cross-column list (same page, one item per column)."""
|
||
_thread_roundtrip(name="cross_column_list", doc_factory=_doc_cross_column_list, page_breaks=0)
|
||
|
||
|
||
def test_table_with_class_raises_error():
|
||
r"""Test that ``<table class=\"…\">`` is rejected (v0.5: use ``<index>`` for document indexes)."""
|
||
xml_str = """<doclang>
|
||
<table class="index">
|
||
<fcel/>
|
||
<text>Data 1</text>
|
||
<nl/>
|
||
</table>
|
||
</doclang>"""
|
||
|
||
with pytest.raises(ValueError, match="table element must not have a class attribute"):
|
||
_deserialize(xml_str, validate=False)
|
||
|
||
|
||
def test_deserialize_rejects_oversize_xml() -> None:
|
||
"""Huge DocLang markup must fail closed before DOM construction grows unchecked."""
|
||
xml = "<doclang>" + ("x" * 200) + "</doclang>"
|
||
with pytest.raises(ValueError, match="exceeds size limit"):
|
||
DocLangDocDeserializer().deserialize_str(xml, max_xml_bytes=64)
|
||
|
||
|
||
def test_deserialize_rejects_over_deep_xml() -> None:
|
||
"""Extreme element nesting must fail closed (defusedxml does not bound depth)."""
|
||
depth = 20
|
||
xml = "".join(f"<g{i}>" for i in range(depth)) + "x" + "".join(f"</g{i}>" for i in reversed(range(depth)))
|
||
xml = f"<doclang>{xml}</doclang>"
|
||
with pytest.raises(ValueError, match="exceeds nesting depth limit"):
|
||
DocLangDocDeserializer().deserialize_str(xml, max_xml_depth=8)
|
||
|
||
|
||
def test_deserialize_rejects_too_many_elements() -> None:
|
||
"""Element floods must fail closed even when total byte size is modest."""
|
||
# 30 sibling elements under doclang — over a tight element budget.
|
||
children = "".join(f"<text>t{i}</text>" for i in range(30))
|
||
xml = f"<doclang>{children}</doclang>"
|
||
with pytest.raises(ValueError, match="exceeds element count limit"):
|
||
DocLangDocDeserializer().deserialize_str(xml, max_xml_elements=10)
|
||
|
||
|
||
def test_deserialize_accepts_document_within_budgets() -> None:
|
||
xml = "<doclang><text>hello</text></doclang>"
|
||
doc = DocLangDocDeserializer().deserialize_str(
|
||
xml,
|
||
max_xml_bytes=1024,
|
||
max_xml_depth=16,
|
||
max_xml_elements=100,
|
||
)
|
||
assert len(doc.texts) == 1
|
||
assert doc.texts[0].text == "hello"
|
||
|
||
|
||
def _doc_with_named_groups() -> DoclingDocument:
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
doc.add_text(label=DocItemLabel.TITLE, text="Masthead")
|
||
for index in (1, 2):
|
||
group = doc.add_group(label=GroupLabel.SECTION, name="article")
|
||
doc.add_heading(text=f"Headline {index}", level=2, parent=group)
|
||
doc.add_text(label=DocItemLabel.TEXT, text=f"Body {index}", parent=group)
|
||
return doc
|
||
|
||
|
||
def _serialize_named_groups(doc: DoclingDocument, *, add_named_groups: bool) -> str:
|
||
ser = DocLangDocSerializer(
|
||
doc=doc,
|
||
params=DocLangParams(include_version=False, add_named_groups=add_named_groups),
|
||
)
|
||
return ser.serialize().text
|
||
|
||
|
||
def test_named_groups_are_off_by_default() -> None:
|
||
"""A plain group stays transparent unless ``add_named_groups`` is set."""
|
||
text = _serialize_named_groups(_doc_with_named_groups(), add_named_groups=False)
|
||
assert "<group" not in text
|
||
assert_valid_dclg_xml(text)
|
||
|
||
doc = _deserialize(text)
|
||
assert doc.groups == []
|
||
assert [item.text for item in doc.texts] == [
|
||
"Masthead",
|
||
"Headline 1",
|
||
"Body 1",
|
||
"Headline 2",
|
||
"Body 2",
|
||
]
|
||
|
||
|
||
def test_named_group_roundtrip() -> None:
|
||
"""``add_named_groups`` keeps the group, its name and its label across a roundtrip."""
|
||
text = _serialize_named_groups(_doc_with_named_groups(), add_named_groups=True)
|
||
assert text.count('<group name="article">') == 2
|
||
assert text.count('<label value="section"/>') == 2
|
||
|
||
# Not validated against the DocLang XSD: the schema has no `name` attribute
|
||
# on <group> yet.
|
||
doc = _deserialize(text, validate=False)
|
||
assert [(group.label, group.name) for group in doc.groups] == [
|
||
(GroupLabel.SECTION, "article"),
|
||
(GroupLabel.SECTION, "article"),
|
||
]
|
||
assert [child.cref for child in doc.body.children] == [
|
||
"#/texts/0",
|
||
"#/groups/0",
|
||
"#/groups/1",
|
||
]
|
||
for group in doc.groups:
|
||
children = [child.resolve(doc) for child in group.children]
|
||
assert [item.label for item in children] == [
|
||
DocItemLabel.SECTION_HEADER,
|
||
DocItemLabel.TEXT,
|
||
]
|
||
assert _serialize_named_groups(doc, add_named_groups=True) == text
|
||
|
||
|
||
def test_named_group_escapes_its_name() -> None:
|
||
doc = DoclingDocument(name="t")
|
||
group = doc.add_group(label=GroupLabel.UNSPECIFIED, name='a & "b"')
|
||
doc.add_text(label=DocItemLabel.TEXT, text="child", parent=group)
|
||
|
||
text = _serialize_named_groups(doc, add_named_groups=True)
|
||
assert '<group name="a & "b"">' in text
|
||
assert "<label" not in text
|
||
|
||
parsed = _deserialize(text, validate=False)
|
||
assert [(g.label, g.name) for g in parsed.groups] == [(GroupLabel.UNSPECIFIED, 'a & "b"')]
|
||
|
||
|
||
def test_named_group_keeps_a_picture_and_a_table_inside() -> None:
|
||
"""A named group holding a float is a group, not the float+footnote wrapper."""
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
group = doc.add_group(label=GroupLabel.SECTION, name="article")
|
||
doc.add_heading(text="Headline", level=2, parent=group)
|
||
doc.add_picture(parent=group)
|
||
doc.add_table(
|
||
data=TableData(
|
||
num_rows=1,
|
||
num_cols=1,
|
||
table_cells=[
|
||
TableCell(
|
||
text="cell",
|
||
start_row_offset_idx=0,
|
||
end_row_offset_idx=1,
|
||
start_col_offset_idx=0,
|
||
end_col_offset_idx=1,
|
||
)
|
||
],
|
||
),
|
||
parent=group,
|
||
)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="Body", parent=group)
|
||
|
||
text = _serialize_named_groups(doc, add_named_groups=True)
|
||
parsed = _deserialize(text, validate=False)
|
||
|
||
assert len(parsed.groups) == 1
|
||
assert len(parsed.pictures) == 1
|
||
assert len(parsed.tables) == 1
|
||
children = [child.resolve(parsed) for child in parsed.groups[0].children]
|
||
assert [item.label for item in children] == [
|
||
DocItemLabel.SECTION_HEADER,
|
||
DocItemLabel.PICTURE,
|
||
DocItemLabel.TABLE,
|
||
DocItemLabel.TEXT,
|
||
]
|
||
|
||
|
||
def test_unnamed_group_markup_still_wraps_floats() -> None:
|
||
"""The float+footnote ``<group>`` wrapper keeps parsing as one unit."""
|
||
xml = "<doclang><group><picture><caption>cap</caption></picture><footnote>note</footnote></group></doclang>"
|
||
doc = _deserialize(xml, validate=False)
|
||
assert doc.groups == []
|
||
assert len(doc.pictures) == 1
|
||
|
||
|
||
def test_default_resolution_sets_page_coordinate_space():
|
||
"""<default_resolution> establishes the page size so locations are not clamped."""
|
||
import warnings
|
||
|
||
xml = (
|
||
'<doclang version="0.7">'
|
||
'<head><default_resolution width="2048" height="2048"/></head>'
|
||
"<text>"
|
||
'<location value="27"/><location value="14"/>'
|
||
'<location value="1316"/><location value="1900"/>'
|
||
"<content>Hello</content></text>"
|
||
"</doclang>"
|
||
)
|
||
doc = _deserialize(xml, validate=False)
|
||
|
||
assert doc.pages[1].size.width == 2048
|
||
assert doc.pages[1].size.height == 2048
|
||
bbox = doc.texts[0].prov[0].bbox
|
||
assert (bbox.l, bbox.t, bbox.r, bbox.b) == (27, 14, 1316, 1900)
|
||
|
||
with warnings.catch_warnings(record=True) as caught:
|
||
warnings.simplefilter("always")
|
||
DoclingDocument.validate_document(doc)
|
||
assert [w for w in caught if "clamping" in str(w.message)] == []
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("frag", "expected"),
|
||
[
|
||
("<text>2<superscript>nd</superscript></text>", ["2", "nd"]),
|
||
("<text>plain <bold>b</bold></text>", ["plain", "b"]),
|
||
("<footnote>see <italic>ibid</italic></footnote>", ["see", "ibid"]),
|
||
("<formula>E <bold>=</bold></formula>", ["E", "="]),
|
||
("<text>x <bold>a <italic>b</italic></bold></text>", ["x", "a", "b"]),
|
||
# shapes that already worked
|
||
("<text><superscript>nd</superscript> place</text>", ["nd", "place"]),
|
||
("<text>H<subscript>2</subscript>O</text>", ["H", "2", "O"]),
|
||
],
|
||
)
|
||
def test_leading_text_before_lone_formatting_tag_is_preserved(frag: str, expected: list[str]):
|
||
doc = _deserialize(f'<doclang version="0.7">{frag}</doclang>')
|
||
assert [t.text.strip() for t in doc.texts if t.text] == expected
|
||
|
||
|
||
def test_mixed_content_footnote_keeps_label_and_provenance():
|
||
doc = _deserialize(
|
||
'<doclang version="0.7"><footnote>'
|
||
'<location value="87"/><location value="417"/><location value="387"/><location value="426"/>'
|
||
"<superscript>①</superscript>盐池县县志编纂委员会编《盐池县志》</footnote></doclang>"
|
||
)
|
||
host = doc.texts[0]
|
||
assert host.label == DocItemLabel.FOOTNOTE
|
||
assert host.text == ""
|
||
assert host.prov and host.prov[0].bbox.as_tuple() == (87, 417, 387, 426)
|
||
(inline,) = [ref.resolve(doc) for ref in host.children]
|
||
assert inline.label == GroupLabel.INLINE
|
||
runs = [ref.resolve(doc) for ref in inline.children]
|
||
assert [r.text for r in runs] == ["①", "盐池县县志编纂委员会编《盐池县志》"]
|
||
assert runs[0].formatting is not None and runs[0].formatting.script == Script.SUPER
|
||
|
||
|
||
def test_nested_formatting_tag_accumulates_style():
|
||
"""<bold>a <italic>b</italic></bold>: "a" keeps bold, "b" keeps bold *and* italic."""
|
||
doc = _deserialize('<doclang version="0.7"><text>x <bold>a <italic>b</italic></bold></text></doclang>')
|
||
runs = {t.text.strip(): t.formatting for t in doc.texts if t.text}
|
||
assert runs["x"] is None
|
||
assert runs["a"] is not None and runs["a"].bold and not runs["a"].italic
|
||
assert runs["b"] is not None and runs["b"].bold and runs["b"].italic
|
||
|
||
# Round trip must stay valid DocLang (no <text> nested inside <text>).
|
||
xml = _serialize(doc)
|
||
doc2 = _deserialize(xml)
|
||
runs2 = {t.text.strip(): t.formatting for t in doc2.texts if t.text}
|
||
assert runs2["a"].bold and not runs2["a"].italic
|
||
assert runs2["b"].bold and runs2["b"].italic
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("label", "layer"),
|
||
[
|
||
(DocItemLabel.TEXT, ContentLayer.BODY),
|
||
(DocItemLabel.TEXT, ContentLayer.FURNITURE),
|
||
(DocItemLabel.FOOTNOTE, ContentLayer.BODY),
|
||
(DocItemLabel.PAGE_HEADER, ContentLayer.FURNITURE),
|
||
(DocItemLabel.PAGE_FOOTER, ContentLayer.FURNITURE),
|
||
],
|
||
)
|
||
def test_roundtrip_inline_host(label: DocItemLabel, layer: ContentLayer):
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
host = doc.add_text(label=label, text="", prov=_default_prov(), content_layer=layer)
|
||
inline = doc.add_inline_group(parent=host, content_layer=layer)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="1",
|
||
parent=inline,
|
||
formatting=Formatting(script=Script.SUPER),
|
||
content_layer=layer,
|
||
)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="body", parent=inline, content_layer=layer)
|
||
|
||
doc2 = _deserialize(_serialize(doc))
|
||
|
||
host2 = doc2.texts[0]
|
||
assert host2.label == label
|
||
assert host2.text == ""
|
||
assert host2.content_layer == layer
|
||
assert host2.prov
|
||
runs = [
|
||
it
|
||
for it, _ in doc2.iterate_items(root=host2, included_content_layers=set(ContentLayer))
|
||
if isinstance(it, TextItem) and it is not host2
|
||
]
|
||
assert [(r.text, r.content_layer) for r in runs] == [("1", layer), ("body", layer)]
|
||
assert runs[0].formatting is not None and runs[0].formatting.script == Script.SUPER
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("tag", "caption_host"),
|
||
[
|
||
("text", None),
|
||
("caption", "picture"),
|
||
("caption", "table"),
|
||
("footnote", None),
|
||
("page_header", None),
|
||
("page_footer", None),
|
||
],
|
||
)
|
||
@pytest.mark.parametrize("body", ["Before <formula>x</formula> after", "Before <bold>bold</bold> after"])
|
||
@pytest.mark.parametrize("located", [False, True])
|
||
@pytest.mark.parametrize("layer", [ContentLayer.BODY, ContentLayer.FURNITURE])
|
||
def test_mixed_text_like_archive_roundtrip(
|
||
tag: str, caption_host: str | None, body: str, located: bool, layer: ContentLayer, tmp_path: Path
|
||
):
|
||
locations = (
|
||
'<location value="87"/><location value="417"/><location value="387"/><location value="426"/>' if located else ""
|
||
)
|
||
fragment = f'<{tag}><layer value="{layer.value}"/>{locations}{body}</{tag}>'
|
||
if caption_host is not None:
|
||
table_body = "<fcel/>Cell<nl/>" if caption_host == "table" else ""
|
||
fragment = f"<{caption_host}>{fragment}{table_body}</{caption_host}>"
|
||
doc = _deserialize(f'<doclang version="0.7">{fragment}</doclang>')
|
||
for cycle in range(3):
|
||
archive = tmp_path / f"mixed-{tag}-{cycle}.dclx"
|
||
doc.save_as_doclang_archive(archive)
|
||
doc = DoclingDocument.load_from_doclang_archive(archive)
|
||
if tag == "text" and not located:
|
||
inline = doc.body.children[0].resolve(doc)
|
||
assert inline.label == GroupLabel.INLINE
|
||
else:
|
||
host = doc.texts[0]
|
||
assert host.label == DocItemLabel(tag) and host.text == ""
|
||
assert len(host.prov) == int(located)
|
||
if located:
|
||
assert host.prov[0].bbox.as_tuple() == (87, 417, 387, 426)
|
||
if caption_host is not None:
|
||
container = doc.pictures[0] if caption_host == "picture" else doc.tables[0]
|
||
assert container.captions == [host.get_ref()]
|
||
inline = host.children[0].resolve(doc)
|
||
assert inline.label == GroupLabel.INLINE
|
||
root = inline if tag == "text" and not located else host
|
||
assert all(
|
||
node.content_layer == layer
|
||
for node, _ in doc.iterate_items(root=root, with_groups=True, included_content_layers=set(ContentLayer))
|
||
)
|
||
runs = [ref.resolve(doc) for ref in inline.children]
|
||
assert [run.text for run in runs] == ["Before ", "x" if "formula" in body else "bold", " after"]
|
||
if "formula" in body:
|
||
assert runs[1].label == DocItemLabel.FORMULA
|
||
else:
|
||
assert runs[1].formatting is not None and runs[1].formatting.bold
|
||
assert all(not run.prov for run in runs)
|
||
assert _serialize(doc).count("<location") == (4 if located else 0)
|
||
|
||
|
||
@pytest.mark.parametrize("kind", ["table", "picture"])
|
||
@pytest.mark.parametrize("layer", [ContentLayer.BODY, ContentLayer.FURNITURE])
|
||
def test_plain_caption_layer_roundtrip(kind: str, layer: ContentLayer, tmp_path: Path):
|
||
table_body = "<fcel/>A<nl/>" if kind == "table" else ""
|
||
doc = _deserialize(
|
||
f'<doclang version="0.7"><text>Body paragraph</text><{kind}>'
|
||
f'<caption><layer value="{layer.value}"/>Caption</caption>{table_body}</{kind}></doclang>'
|
||
)
|
||
for cycle in range(2):
|
||
owner = doc.tables[0] if kind == "table" else doc.pictures[0]
|
||
caption = owner.captions[0].resolve(doc)
|
||
assert caption.content_layer == layer
|
||
assert doc.texts[0].content_layer == ContentLayer.BODY
|
||
archive = tmp_path / f"plain-caption-layer-{cycle}.dclx"
|
||
doc.save_as_doclang_archive(archive)
|
||
doc = DoclingDocument.load_from_doclang_archive(archive)
|
||
|
||
|
||
@pytest.mark.parametrize("kind", ["table", "picture"])
|
||
@pytest.mark.parametrize(
|
||
("caption", "expected", "markdown", "doctags"),
|
||
[
|
||
("Table 1: results", "Table 1: results", "Table 1: results", "Table 1: results"),
|
||
# DocTags predates the whitespace contract: it strips each run and puts them on separate lines.
|
||
("Table <bold>1</bold>: results", "Table 1: results", "Table **1**: results", "Table 1 : results"),
|
||
(
|
||
"<bold>Table <italic>1</italic></bold>: results",
|
||
"Table 1: results",
|
||
"**Table** ***1***: results",
|
||
"Table 1 : results",
|
||
),
|
||
],
|
||
)
|
||
@pytest.mark.parametrize("layer", [ContentLayer.BODY, ContentLayer.FURNITURE])
|
||
def test_caption_readers_after_archive_roundtrip(
|
||
kind: str, caption: str, expected: str, markdown: str, doctags: str, layer: ContentLayer, tmp_path: Path
|
||
):
|
||
table_body = "<fcel/>A<fcel/>B<nl/><fcel/>1<fcel/>2<nl/>" if kind == "table" else ""
|
||
doc = _deserialize(
|
||
f'<doclang version="0.7"><{kind}><caption><layer value="{layer.value}"/>'
|
||
f"{caption}</caption>{table_body}</{kind}></doclang>"
|
||
)
|
||
for cycle in range(2):
|
||
owner = doc.tables[0] if kind == "table" else doc.pictures[0]
|
||
assert owner.caption_text(doc) == expected
|
||
for serializer in [
|
||
PlainTextDocSerializer(doc=doc),
|
||
MarkdownDocSerializer(doc=doc),
|
||
HTMLDocSerializer(doc=doc),
|
||
DocTagsDocSerializer(doc=doc),
|
||
]:
|
||
result = serializer.serialize_captions(item=owner, layers=set(ContentLayer)).text
|
||
if isinstance(serializer, DocTagsDocSerializer):
|
||
content = " ".join(text.strip() for text in ET.fromstring(result).itertext() if text.strip())
|
||
assert content == doctags
|
||
elif isinstance(serializer, HTMLDocSerializer):
|
||
assert "".join(ET.fromstring(result).itertext()) == expected
|
||
if "<bold>" in caption:
|
||
assert "<strong>1</strong>" in result
|
||
if "<italic>" in caption:
|
||
assert "<em>" in result
|
||
elif isinstance(serializer, PlainTextDocSerializer):
|
||
assert expected in result
|
||
else:
|
||
assert markdown in result
|
||
assert serializer.serialize(layers=set(ContentLayer)).text.count("results") == 1
|
||
if layer == ContentLayer.FURNITURE:
|
||
assert not serializer.serialize_captions(item=owner, layers={ContentLayer.BODY}).text
|
||
assert not serializer.serialize_captions(item=owner, labels={DocItemLabel.TABLE, DocItemLabel.PICTURE}).text
|
||
archive = tmp_path / f"caption-reader-{cycle}.dclx"
|
||
doc.save_as_doclang_archive(archive)
|
||
doc = DoclingDocument.load_from_doclang_archive(archive)
|
||
|
||
|
||
@pytest.mark.parametrize("kind", ["table", "picture"])
|
||
@pytest.mark.parametrize("host_text", ["", "HOST &"])
|
||
def test_native_caption_children_and_filters(kind: str, host_text: str):
|
||
doc = DoclingDocument(name="caption-boundary")
|
||
owner = doc.add_table(data=TableData()) if kind == "table" else doc.add_picture()
|
||
caption = doc.add_text(label=DocItemLabel.CAPTION, text=host_text)
|
||
owner.captions.append(caption.get_ref())
|
||
inline = doc.add_inline_group(parent=caption)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="CHILD ", formatting=Formatting(bold=True), parent=inline)
|
||
doc.add_text(label=DocItemLabel.FORMULA, text="FORMULA", parent=inline)
|
||
DoclingDocument.model_validate(doc.model_dump())
|
||
assert owner.caption_text(doc) == " ".join(text for text in [host_text, "CHILD FORMULA"] if text)
|
||
labels = {DocItemLabel.CAPTION, DocItemLabel.TEXT, DocItemLabel.TABLE, DocItemLabel.PICTURE}
|
||
for serializer in [
|
||
MarkdownDocSerializer(doc=doc),
|
||
PlainTextDocSerializer(doc=doc),
|
||
HTMLDocSerializer(doc=doc),
|
||
LaTeXDocSerializer(doc=doc),
|
||
DocTagsDocSerializer(doc=doc),
|
||
DocLangDocSerializer(doc=doc),
|
||
]:
|
||
result = serializer.serialize_captions(item=owner, labels=labels).text
|
||
assert "CHILD" in result
|
||
assert "FORMULA" not in result
|
||
whole = serializer.serialize(labels=labels).text
|
||
assert whole.count("CHILD") == 1
|
||
assert "FORMULA" not in whole
|
||
if host_text:
|
||
assert "HOST" in result
|
||
assert whole.count("HOST") == 1
|
||
if isinstance(serializer, HTMLDocSerializer):
|
||
assert "<strong>CHILD</strong>" in result
|
||
if host_text:
|
||
# Host text and its inline group are joined with a space; the runs inside are not.
|
||
assert "HOST & <span class='inline-group'><strong>CHILD</strong>" in result
|
||
assert "&amp;" not in result
|
||
elif isinstance(serializer, LaTeXDocSerializer):
|
||
assert r"\textbf{CHILD}" in result
|
||
elif isinstance(serializer, MarkdownDocSerializer) and not isinstance(serializer, PlainTextDocSerializer):
|
||
assert "**CHILD**" in result
|
||
if host_text:
|
||
assert "HOST & **CHILD**" in result
|
||
|
||
|
||
def test_rich_caption_html_direction():
|
||
doc = _deserialize('<doclang version="0.7"><picture><caption><bold>אב</bold> גד</caption></picture></doclang>')
|
||
result = HTMLDocSerializer(doc=doc).serialize_captions(item=doc.pictures[0]).text
|
||
assert '<div class="caption" dir="rtl">' in result
|
||
assert "<strong>אב</strong>" in result
|
||
|
||
|
||
def test_roundtrip_furniture_inline_group():
|
||
doc = DoclingDocument(name="t")
|
||
_add_default_page(doc)
|
||
inline = doc.add_inline_group(content_layer=ContentLayer.FURNITURE)
|
||
doc.add_text(label=DocItemLabel.TEXT, text="a", parent=inline, content_layer=ContentLayer.FURNITURE)
|
||
doc.add_text(
|
||
label=DocItemLabel.TEXT,
|
||
text="b",
|
||
parent=inline,
|
||
formatting=Formatting(bold=True),
|
||
content_layer=ContentLayer.FURNITURE,
|
||
)
|
||
|
||
doc2 = _deserialize(_serialize(doc))
|
||
|
||
items = [it for it, _ in doc2.iterate_items(with_groups=True, included_content_layers=set(ContentLayer))][1:]
|
||
assert [it.label for it in items] == [GroupLabel.INLINE, DocItemLabel.TEXT, DocItemLabel.TEXT]
|
||
assert {it.content_layer for it in items} == {ContentLayer.FURNITURE}
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("body", "expected"),
|
||
[
|
||
("Wiley & Sons", ["Wiley & Sons"]),
|
||
("$q < 0$ and p <0.05 and $a<\\ln b$", ["$q < 0$ and p <0.05 and $a<\\ln b$"]),
|
||
("x]]> y", ["x]]> y"]),
|
||
("a\x18b", ["ab"]),
|
||
# The run after the CDATA section owns its leading space under the whitespace contract.
|
||
("<![CDATA[a & b < c]]> & d", ["a & b < c", " & d"]),
|
||
],
|
||
)
|
||
def test_repair_unescaped_text(body: str, expected: list[str]):
|
||
xml = f"<doclang><text>{body}</text></doclang>"
|
||
with pytest.raises(ValueError, match="Invalid DocLang XML"):
|
||
DocLangDocDeserializer().deserialize_str(xml)
|
||
doc = DocLangDocDeserializer().deserialize_str(xml, repair_unescaped_text=True)
|
||
assert [t.text for t in doc.texts] == expected
|
||
|
||
|
||
@pytest.mark.parametrize("kwargs", [{}, {"repair_unescaped_text": False}])
|
||
def test_repair_unescaped_text_is_opt_in(kwargs):
|
||
with pytest.raises(ValueError, match="Invalid DocLang XML"):
|
||
DocLangDocDeserializer().deserialize_str("<doclang><text>a & b</text></doclang>", **kwargs)
|
||
|
||
|
||
def test_repair_keeps_well_formed_markup_and_entities():
|
||
xml = '<doclang><text>a & < ><![CDATA[ & < ]]></text><heading level="1">H</heading></doclang>'
|
||
plain = DocLangDocDeserializer().deserialize_str(xml)
|
||
repaired = DocLangDocDeserializer().deserialize_str(xml, repair_unescaped_text=True)
|
||
assert repaired.export_to_dict() == plain.export_to_dict()
|
||
|
||
|
||
def test_empty_list_item_does_not_replay_siblings():
|
||
xml = """<doclang version="0.7"><list>
|
||
<ldiv/><content>Alpha</content>
|
||
<ldiv/><location value="1"/><location value="2"/>
|
||
<location value="3"/><location value="4"/>
|
||
<ldiv/><content>Beta</content>
|
||
</list></doclang>"""
|
||
doc = DocLangDocDeserializer().deserialize_str(xml)
|
||
assert sorted(it.text for it in doc.texts if it.text.strip()) == ["Alpha", "Beta"]
|
||
assert [g for g in doc.groups if not g.children] == []
|
||
|
||
|
||
@pytest.mark.parametrize("markup", ["<text></text>", "<text>\n </text>"])
|
||
def test_empty_text_is_an_empty_text_item(markup: str) -> None:
|
||
doc = DocLangDocDeserializer().deserialize_str(f'<doclang version="0.7">\n<text>a</text>\n{markup}\n</doclang>')
|
||
|
||
assert [t.text for t in doc.texts] == ["a", ""]
|
||
assert not doc.groups
|
||
|
||
|
||
def test_empty_text_keeps_its_locations_and_layer() -> None:
|
||
locs = "".join(f'<location value="{v}"/>' for v in (10, 20, 30, 40))
|
||
doc = DocLangDocDeserializer().deserialize_str(
|
||
f'<doclang version="0.7"><text><layer value="furniture"/>{locs}</text></doclang>'
|
||
)
|
||
|
||
(item,) = doc.texts
|
||
assert item.text == ""
|
||
assert item.content_layer.value == "furniture"
|
||
assert len(item.prov) == 1 and item.prov[0].bbox.l < item.prov[0].bbox.r
|
||
assert not doc.groups
|
||
|
||
|
||
def test_track_is_skipped_with_a_warning() -> None:
|
||
xml = (
|
||
'<doclang version="0.8"><text>Before</text>'
|
||
"<track><caption>Talk</caption>"
|
||
'<bdiv/><seconds value="0"/><seconds value="4"/><chapter><text>Intro</text></chapter>Hello world'
|
||
'<bdiv/><seconds value="4"/>Second cue</track>'
|
||
"<text>After</text></doclang>"
|
||
)
|
||
with pytest.warns(UserWarning, match="<track> is not supported yet"):
|
||
doc = _deserialize(xml)
|
||
|
||
assert [t.text for t in doc.texts] == ["Before", "After"]
|
||
|
||
|
||
def test_empty_formula_is_kept() -> None:
|
||
doc = DocLangDocDeserializer().deserialize_str('<doclang version="0.7"><formula></formula><formula/></doclang>')
|
||
|
||
assert [type(t).__name__ for t in doc.texts] == ["FormulaItem", "FormulaItem"]
|
||
assert all(t.text == "" for t in doc.texts)
|
||
assert not doc.groups
|
||
|
||
|
||
def test_empty_code_is_kept() -> None:
|
||
doc = DocLangDocDeserializer().deserialize_str('<doclang version="0.7"><code></code><code/></doclang>')
|
||
|
||
assert [type(t).__name__ for t in doc.texts] == ["CodeItem", "CodeItem"]
|
||
assert all(t.text == "" for t in doc.texts)
|
||
assert not doc.groups
|
||
|
||
|
||
def _cdata(text: str) -> str:
|
||
return f"<![CDATA[{text.replace(']]>', ']]]]><![CDATA[>')}]]>"
|
||
|
||
|
||
def test_custom_field_text_roundtrip() -> None:
|
||
# custom content (here with markup and CDATA) must round-trip as text
|
||
# and must not leak into the text of the host element
|
||
value = "<opt><formula>a</formula><formula><![CDATA[b < c]]></formula></opt>"
|
||
xml = (
|
||
'<doclang version="0.7"><formula><custom>'
|
||
f"<acme__alt>{_cdata(value)}</acme__alt></custom>a b c</formula></doclang>"
|
||
)
|
||
|
||
doc = DocLangDocDeserializer().deserialize_str(xml)
|
||
|
||
(item,) = doc.texts
|
||
assert item.text == "a b c"
|
||
assert item.meta is not None
|
||
assert item.meta.get_custom_part() == {"acme__alt": value}
|
||
|
||
doc2 = DocLangDocDeserializer().deserialize_str(DocLangDocSerializer(doc=doc).serialize().text)
|
||
assert doc2.texts[0].text == "a b c"
|
||
assert doc2.texts[0].meta is not None
|
||
assert doc2.texts[0].meta.get_custom_part() == {"acme__alt": value}
|
||
|
||
|
||
def test_custom_field_structured_roundtrip() -> None:
|
||
opt1 = "<formula>a</formula>\n<formula><![CDATA[b < c]]></formula>"
|
||
opt2 = "another option"
|
||
xml = (
|
||
'<doclang version="0.7"><formula><custom><docling__alt_repr>'
|
||
f"<docling__alt_opt>{_cdata(opt1)}</docling__alt_opt>"
|
||
f"<docling__alt_opt>{_cdata(opt2)}</docling__alt_opt>"
|
||
"</docling__alt_repr><acme__tag>x</acme__tag><acme__tag>y</acme__tag></custom>a b c</formula></doclang>"
|
||
)
|
||
expected = {"docling__alt_repr": {"docling__alt_opt": [opt1, opt2]}, "acme__tag": ["x", "y"]}
|
||
|
||
doc = DocLangDocDeserializer().deserialize_str(xml)
|
||
|
||
(item,) = doc.texts
|
||
assert item.text == "a b c"
|
||
assert item.meta is not None
|
||
assert item.meta.get_custom_part() == expected
|
||
|
||
doc2 = DocLangDocDeserializer().deserialize_str(DocLangDocSerializer(doc=doc).serialize().text)
|
||
assert doc2.texts[0].meta is not None
|
||
assert doc2.texts[0].meta.get_custom_part() == expected
|
||
out = DocLangDocSerializer(doc=doc2).serialize().text
|
||
assert out == DocLangDocSerializer(doc=doc).serialize().text
|
||
assert "<content>" not in out # nested values are not wrapped just for containing newlines
|