mirror of
https://github.com/docling-project/docling-core.git
synced 2026-11-18 20:27:00 +00:00
* fix(doclang): preserve leading text before a lone formatting tag `_get_children_simple_text_block` guarded against a second *text* fragment but the Element branch just overwrote `result`, so `<text>2<superscript>nd</superscript></text>` returned "nd" and the leading `2` was silently discarded. Guard both branches so the element is routed to `_parse_inline_group`, which already handles mixed children. * feat: inline runs carry their own whitespace; serializers concatenate faithfully The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`, `O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never be removed while the separator lives in the serializer. Serializers: - join inline parts with `""`; DocLang keeps its delimiter between the element head and the body only, where `<content>` is the whitespace channel - hoist edge whitespace around the *complete* decoration stack (formatting and hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim emits `**bold ** tail`, which CommonMark renders as literal asterisks; hoisting inside one formatter would let the hyperlink recapture it. DocLang opts out: it keeps whitespace inside the markup, in `<content>`. DocLang deserializer: - `_get_text` stripped every fragment and joined with `""`, collapsing `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer block boundary only, and never trims whitespace delivered by `<content>`. - bare text nodes use the standard XML heuristic: a whitespace run containing a newline is pretty-print indentation, one without it is content. This is what hand-written and VLM-emitted DocLang needs, since it will not use `<content>`. - element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`) no longer bleed into body text, where they were also double-modelled in `item.meta`. Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator re-injects the legacy separator for documents stamped below it, preserving their previous *visible* output including its extra-space bugs (`H 2 O` stays `H 2 O`). It must run before `check_version_is_compatible`, which rewrites `version` and destroys the evidence. Versionless input is left alone. The validator works on a copy, so the caller's dict is untouched and re-validating the same raw value cannot compound separators. A separator can never land inside a delimiter. Code and formula runs are wrapped in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when both neighbours are delimited it becomes a plain run of its own, because `` `a``b` `` and `$a$$b$` are both corrupt. Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join -- anyone with a persisted vector index gets silent retrieval drift, so that needs its own line in the release notes. DocTags code is untouched. Its golden gains one empty `<text></text>` for the legacy fixture because that document now carries an extra whitespace run, which DocTags renders empty by its own pre-existing strip-per-run behaviour. Ships as a minor, not a major: `feat!:` would make python-semantic-release (7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package. * DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com> I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 64404b3709d9e3ba7caf08957cf2ef19df29459f I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 66040cb7b96f77699c9d74051500d645e7e758ed Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat(compat): register 1.11->1.10 projector for the whitespace-contract bump Integrates the 1.11.0 schema bump with the downgrade-projector framework (#703) landed upstream. 1.11 and 1.10 are structurally identical; the only difference is that 1.11 inline runs own their boundary whitespace, so the projector just re-stamps the version (a 1.10 client re-applies its own join, yielding the documented harmless double space). - add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10 - add the auto-generated 1.11 JSON Schema snapshot - re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff) - update the stale dclq CLI expectation (old join produced 'public class'; faithful join now yields 'public class') Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(dclq): use typer's vendored click; require typer>=0.25 typer >=0.25 vendors its own copy of click (typer._click) and raises those exceptions instead of the external click package's. The dclq CLI referenced the external click, so on typer >=0.25 bad input escaped the exit-code handlers and the context-flag callback (-C/-A/-B/--context-scope) silently no-op'd because the isinstance(param, click.Option) check no longer matched. Import the exception and Parameter types from typer._click, duck-type the option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused external click dependency. Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(doclang): keep inherited formatting on <content> runs in inline groups <content> is the explicit whitespace-preserving text wrapper, so inside an inline group it is a run, not a nested item: it must inherit the formatting accumulated while descending through <bold>/<italic>/... . Dispatching it through _dispatch_element dropped that formatting, which surfaced after the rebase as <bold><content>x </content></bold> coming back unformatted and broke the doclang round-trip for whitespace_inside_bold / _link. Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com> * chore(compat): point schema 1.11.0 at the next minor release The whitespace-contract bump was first written against v2.86.0; main is now at v2.104.0, so record v2.105.0 as the placeholder release that ships schema 1.11.0. Adjust to the actual version at release time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * test: rebaseline goldens for the inline whitespace contract Regenerated with DOCLING_GEN_TEST_DATA=1. The only change is the schema version stamp 1.10.0 -> 1.11.0 in goldens added or moved on main (including the six test/ -> tests/ renames and misplaced_list_items.out.yaml). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix: join inline runs faithfully in the plain-text reader _collect_subtree_text (TextItem._get_plain_text, FloatingItem.caption_text, nested rich table cells) still joined the runs of an InlineGroup with " ", so mixed captions read "Table 1 : results". It now concatenates the runs of an inline group; every other join, including a host text item's own text and its children, keeps its space. Children of a text item are block structure (producers put a section's paragraphs under its heading), so the host text stays apart from its inline group, as on main; add_inline_group documents this. Tests that asserted the invented run spacing now expect the contract output, fixtures that built runs without boundary whitespace now carry it, and a new test pins the host-text boundary for headings and captions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * docs: note that load_from_doctags cannot restore inline run whitespace DocTags stores inline runs trimmed, so the boundary whitespace the inline whitespace contract expects is not in the source. Document the limitation instead of guessing separators back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(md): keep the host separator before a list item's inline group The Markdown list serializer glued an inline group onto the preceding list item with no separator, so a list item "Intro" whose child group starts with "more" exported "- Intromore". Captions, headings and the plain-text reader keep a space between host text and its inline group; the list serializer now does the same when the group is the child of a text item with its own text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): keep migrated separators between code or formula runs in page filters A legacy document with two adjacent code or formula runs gets a standalone " " run between them on load. It had no provenance, so filter(page_nrs=...) and export_to_markdown(page_no=...) dropped it and rendered `$a$$b$`. The separator now takes the left run's location when both runs are on the same page, so it survives a page filter exactly when both runs do. Runs on different pages keep the unlocated separator, which matches the old join. Inline group locations in DocLang and DocTags are a union of the runs, so they do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): count empty inline hosts and nested inline groups as legacy runs The legacy separator migration counted only direct text children with non-empty text as runs. An empty-text item whose content is a child inline group (a formula or footnote host) and an inline group nested directly in another were skipped, so `left`, host(`x`), `right` loaded as `left $x$right` instead of the old `left $x$ right`. The DocLang deserializer writes both shapes for nested mixed-content elements. A child now counts as a run when it or its subtree has text. Only a plain run with its own text takes the separator; an empty host must keep its empty text to render its group, so it is treated like a code or formula run and the separator stays outside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * docs(doclang): note that pre-contract archives load without legacy separators DocLang has no schema version, so a .dclx written before schema 1.11.0 cannot be told apart from a current one and is not migrated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * chore(dclq): move the vendored-click fix to its own PR This reverts 6a8b8374; the change is unrelated to the whitespace contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): don't run the legacy inline migration in project_to project_to validated the projected dict under its target stamp. For 1.10.0 that ran the load-time separator migration on runs that already carry their whitespace, so 'Normal ', 'bold', ' tail' came back as 'Normal ', 'bold ', ' tail'. Validate under the source stamp and restore the target stamp afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): strip inline runs when projecting 1.11 to 1.10 A 1.10 client joins inline runs with " ", so runs that keep their boundary whitespace rendered with double spaces and with the space inside the markup (**bold **, which CommonMark does not render as bold). The projector now strips each inline-group child's text to the shape old producers emitted. Whitespace-only runs become "", which old serializers skip, so no refs move. orig and text outside inline groups are left alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(compat): reuse the stripped separator run when migrating Between two code or formula runs the migration inserts a plain " " run. The 1.11 -> 1.10 projector strips it to "", and the next migration skipped the empty run and inserted another, so every projection round trip added a run. Refill an empty plain run between the two neighbours instead. Old serializers skipped empty runs, so legacy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com> I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit:7600953bc0I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit:f0e26b6722Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat(doclang): read and write whitespace per DocLang 0.9 - Deserializer applies spec rules to the DOM before parsing: a whitespace sequence reads as one space, none at block edges, <content> is literal. - Serializer keeps each text block body on one line; text with non-single-space whitespace goes in <content>, unwrapped where the rules read it the same bare. - Emit version 0.9; validate against doclang 0.9. Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * fix(doclang): keep inline code and formula in virtual text In a list item or cell with text of its own, <code> and <formula> are inline, so whitespace around them reads as a space under DocLang 0.9. - Reader applies the whitespace rules across them instead of treating them as blocks. - Pretty-printer writes such a virtual text on one line, so it no longer adds spaces next to inline code or formulas. Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * chore(compat): point schema 1.11.0 at v2.106.0 v2.105.0 was released without the whitespace contract, so record the next minor release as the one that ships schema 1.11.0. Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * docs(compat): name schema 1.11 as current in the projectors header Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * fix(doclang): harden the DocLang 0.9 whitespace reader and writer - read list items with leading inline code or formula as virtual text - keep whitespace-only content between inline code runs - keep blank lines in content when pretty printing - keep inline handwriting and rtl text as runs - trim only XML whitespace when reading text - keep content literal in headings, captions and list items - read table cells with inline elements as virtual text - keep text beside a whitespace-only formatted run - read spaces around an empty content as one sequence - read a checkbox as part of virtual text - keep a whitespace-only inline code run Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix(doclang): preserve checkbox spaces and formatted whitespace runs Signed-off-by: Christoph Auer <cau@zurich.ibm.com> --------- Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com> Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com> Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
773 lines
29 KiB
Python
773 lines
29 KiB
Python
"""Tests for docling_core.compat — server-side downgrade projectors.
|
|
|
|
Test strategy
|
|
-------------
|
|
1. **Coverage enforcement** — assert that a projector is registered for every
|
|
minor-version step within the docling-core v2.x schema range
|
|
(1.FIRST_SUPPORTED_MINOR through 1.CURRENT_MINOR).
|
|
This is the primary CI gate: it will fail as soon as a schema bump lands
|
|
without a companion projector.
|
|
|
|
2. **Unit tests per projector** — each registered projector is exercised with
|
|
a minimal synthetic dict that exercises the fields introduced in that minor
|
|
version.
|
|
|
|
3. **Schema-snapshot validation** — every projector's output is validated
|
|
against the JSON Schema snapshot for the *target* version stored in
|
|
``docs/schemas/DoclingDocument_1_{N}.json``. This is the correct oracle:
|
|
it describes exactly what the old client's Pydantic model accepted, without
|
|
requiring the old library to be installed.
|
|
|
|
4. **Chain projection** — verify that ``project_to()`` can project from 1.10
|
|
all the way down to 1.5 (the oldest supported v2.x schema) in a single
|
|
call, chaining all intermediate projectors.
|
|
|
|
5. **No-op projection** — projecting to the *same* version (or newer) returns
|
|
the same document unchanged.
|
|
|
|
6. **Error paths** — missing projector, incompatible major, bad version string,
|
|
and target below the v2.x baseline (1.5.0).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import jsonschema
|
|
import pytest
|
|
|
|
from docling_core.compat import _projectors, list_projectors, project_to
|
|
from docling_core.types.doc.common.constants import (
|
|
_CURRENT_MINOR,
|
|
CURRENT_VERSION,
|
|
FIRST_SUPPORTED_MINOR,
|
|
SCHEMA_VERSION_HISTORY,
|
|
)
|
|
from docling_core.types.doc.document import DocItemLabel, DoclingDocument, Formatting
|
|
|
|
# Root of the repository, used to locate docs/schemas/.
|
|
_REPO_ROOT = Path(__file__).parent.parent
|
|
_SCHEMAS_DIR = _REPO_ROOT / "docs" / "schemas"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _minimal_doc_dict(version: str = CURRENT_VERSION) -> dict:
|
|
"""Return the smallest possible valid DoclingDocument serialised dict."""
|
|
return {
|
|
"schema_name": "DoclingDocument",
|
|
"version": version,
|
|
"name": "test_doc",
|
|
"origin": None,
|
|
"furniture": {
|
|
"self_ref": "#/furniture",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "furniture",
|
|
"label": "unspecified",
|
|
"name": "_root_",
|
|
},
|
|
"body": {
|
|
"self_ref": "#/body",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "unspecified",
|
|
"name": "_root_",
|
|
},
|
|
"groups": [],
|
|
"texts": [],
|
|
"pictures": [],
|
|
"tables": [],
|
|
"key_value_items": [],
|
|
"form_items": [],
|
|
"pages": {},
|
|
}
|
|
|
|
|
|
def _build_doc(version: str = CURRENT_VERSION) -> DoclingDocument:
|
|
"""Build a minimal DoclingDocument at *version* by bypassing the validator."""
|
|
doc = DoclingDocument.model_construct(**DoclingDocument.model_validate(_minimal_doc_dict()).model_dump())
|
|
# Override the version to simulate a document at a different schema level.
|
|
object.__setattr__(doc, "version", version)
|
|
return doc
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Coverage enforcement
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestProjectorCoverage:
|
|
"""Ensure a projector exists for every v2.x schema minor-version step."""
|
|
|
|
def test_projector_count_matches_expected(self):
|
|
"""There must be exactly CURRENT_MINOR - FIRST_SUPPORTED_MINOR projectors.
|
|
|
|
With CURRENT_MINOR=10 and FIRST_SUPPORTED_MINOR=5 that is 5 projectors:
|
|
1.10->1.9, 1.9->1.8, 1.8->1.7, 1.7->1.6, 1.6->1.5
|
|
|
|
This test is the CI enforcement gate: it will fail automatically
|
|
when a contributor bumps CURRENT_VERSION without adding a projector.
|
|
"""
|
|
expected = _CURRENT_MINOR - FIRST_SUPPORTED_MINOR
|
|
actual = len(_projectors)
|
|
assert actual == expected, (
|
|
f"Expected {expected} projector(s) (one per minor-version step from "
|
|
f"1.{_CURRENT_MINOR} down to 1.{FIRST_SUPPORTED_MINOR}), but found {actual}. "
|
|
f"Did you bump CURRENT_VERSION without adding a "
|
|
f"@register_projector to docling_core/compat.py?"
|
|
)
|
|
|
|
def test_all_expected_steps_registered(self):
|
|
"""Every step (N->N-1) for N in [FIRST_SUPPORTED_MINOR+1, CURRENT_MINOR] must be registered."""
|
|
missing = []
|
|
for n in range(FIRST_SUPPORTED_MINOR + 1, _CURRENT_MINOR + 1):
|
|
if (n, n - 1) not in _projectors:
|
|
missing.append(f"1.{n} -> 1.{n - 1}")
|
|
assert not missing, f"Missing projectors for: {missing}"
|
|
|
|
def test_schema_version_history_length(self):
|
|
"""SCHEMA_VERSION_HISTORY must have one entry per v2.x minor version."""
|
|
expected = _CURRENT_MINOR - FIRST_SUPPORTED_MINOR + 1
|
|
assert len(SCHEMA_VERSION_HISTORY) == expected, (
|
|
f"SCHEMA_VERSION_HISTORY has {len(SCHEMA_VERSION_HISTORY)} entries; "
|
|
f"expected {expected} (one per schema version "
|
|
f"1.{FIRST_SUPPORTED_MINOR}..1.{_CURRENT_MINOR}). "
|
|
f"Did you forget to append an entry when bumping CURRENT_VERSION?"
|
|
)
|
|
|
|
def test_schema_version_history_is_sorted(self):
|
|
"""Entries must be ordered from 1.0.0 up to CURRENT_VERSION."""
|
|
versions = [e["schema"] for e in SCHEMA_VERSION_HISTORY]
|
|
assert versions[-1] == CURRENT_VERSION, (
|
|
f"Last entry in SCHEMA_VERSION_HISTORY is {versions[-1]!r}, expected {CURRENT_VERSION!r}."
|
|
)
|
|
|
|
def test_list_projectors_returns_sorted_keys(self):
|
|
keys = list_projectors()
|
|
assert keys == sorted(keys)
|
|
|
|
def test_projector_keys_are_sequential(self):
|
|
"""Every registered key must satisfy key[0] == key[1] + 1."""
|
|
for key in _projectors:
|
|
assert key[0] == key[1] + 1, f"Projector key {key} is not a single-step downgrade."
|
|
|
|
@pytest.mark.parametrize("from_minor,to_minor", list(_projectors))
|
|
def test_every_projector_produces_valid_output(self, from_minor: int, to_minor: int):
|
|
"""Every projector's output must validate against the target version's JSON Schema snapshot.
|
|
|
|
The snapshot at ``docs/schemas/DoclingDocument_1_{to_minor}.json`` is the
|
|
correct oracle: it describes exactly what the *target* version's Pydantic
|
|
model accepted, derived from ``DoclingDocument.model_json_schema()`` at
|
|
the moment that version was current. This avoids two failure modes of
|
|
using the *current* SDK's ``model_validate()``:
|
|
|
|
- **False failure:** a future schema change removes a field; the current
|
|
model rejects the projected dict even though it is exactly right for
|
|
the old client.
|
|
- **False pass:** the current model is a superset and silently accepts
|
|
things the old model would have rejected.
|
|
|
|
If the snapshot file is missing, the test fails with a clear message
|
|
directing the contributor to run ``scripts/check_compat_projectors.py``
|
|
which auto-generates any missing snapshot from the live models.
|
|
"""
|
|
snapshot_path = _SCHEMAS_DIR / f"DoclingDocument_1_{to_minor}.json"
|
|
assert snapshot_path.exists(), (
|
|
f"JSON Schema snapshot for schema 1.{to_minor} not found at {snapshot_path}.\n"
|
|
f"Run `python scripts/check_compat_projectors.py` to generate it."
|
|
)
|
|
target_schema = json.loads(snapshot_path.read_text())
|
|
|
|
fn = _projectors[(from_minor, to_minor)]
|
|
# Build a minimal dict at the *source* version and apply the projector.
|
|
data = _minimal_doc_dict(f"1.{from_minor}.0")
|
|
try:
|
|
result = fn(data)
|
|
except Exception as exc:
|
|
raise AssertionError(
|
|
f"Projector 1.{from_minor}->1.{to_minor} raised an exception on a minimal input dict: {exc}"
|
|
) from exc
|
|
|
|
# The result must claim the target version.
|
|
assert result.get("version") == f"1.{to_minor}.0", (
|
|
f"Projector 1.{from_minor}->1.{to_minor} did not set version to "
|
|
f"'1.{to_minor}.0' (got {result.get('version')!r})."
|
|
)
|
|
|
|
# Validate the projected dict against the target version's JSON Schema.
|
|
# jsonschema.validate raises jsonschema.ValidationError on any violation.
|
|
try:
|
|
jsonschema.validate(instance=result, schema=target_schema)
|
|
except jsonschema.ValidationError as exc:
|
|
raise AssertionError(
|
|
f"Projector 1.{from_minor}->1.{to_minor} produced a dict that "
|
|
f"fails validation against docs/schemas/DoclingDocument_1_{to_minor}.json:\n"
|
|
f" {exc.message}\n"
|
|
f" Path: {list(exc.absolute_path)}"
|
|
) from exc
|
|
|
|
def test_every_projector_has_a_test_class(self):
|
|
"""Every registered projector must have a corresponding test class
|
|
named ``TestProjector_1_{from_minor}_to_1_{to_minor}`` in this module.
|
|
|
|
This enforces step 6 of the contributor checklist (write a test for
|
|
the new projector). It does not prevent vacuous tests, but it does
|
|
force a contributor to explicitly author a class — making the gap
|
|
visible in code review.
|
|
|
|
If you added a projector but not a test class, add a class with at
|
|
least one non-trivial test before merging.
|
|
"""
|
|
import sys
|
|
|
|
this_module = sys.modules[__name__]
|
|
missing = []
|
|
for from_minor, to_minor in _projectors:
|
|
class_name = f"TestProjector_1_{from_minor}_to_1_{to_minor}"
|
|
if not hasattr(this_module, class_name):
|
|
missing.append(f"{class_name} (projector 1.{from_minor}->1.{to_minor})")
|
|
assert not missing, (
|
|
"The following projectors are missing a test class in test/test_compat.py:\n"
|
|
+ "\n".join(f" • {m}" for m in missing)
|
|
+ "\nAdd a test class for each projector as described in CONTRIBUTING.md."
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. Unit tests per projector
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestSchemaSnapshotRendering:
|
|
"""The snapshot written for the current version must come from the models.
|
|
|
|
Copying ``docs/DoclingDocument.json`` instead would snapshot whatever
|
|
schema that artefact last held -- which, in the pre-commit hook order, is
|
|
the *previous* version -- producing a file named for the new version but
|
|
describing the old one.
|
|
"""
|
|
|
|
@staticmethod
|
|
def _render() -> str:
|
|
"""Call ``_render_model_schema`` from the check script.
|
|
|
|
``scripts/`` is not an importable package, so load the file directly
|
|
rather than mutating ``sys.path``.
|
|
"""
|
|
import importlib.util
|
|
|
|
path = Path(__file__).parent.parent / "scripts" / "check_compat_projectors.py"
|
|
spec = importlib.util.spec_from_file_location("_check_compat_projectors", path)
|
|
assert spec is not None and spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(module)
|
|
return str(module._render_model_schema())
|
|
|
|
def test_renderer_cannot_drift_from_the_docs_generator(self):
|
|
"""The check script and generate_docs must emit identical bytes."""
|
|
from docling_core.utils.generate_jsonschema import generate_json_schema
|
|
|
|
# Exactly what generate_docs writes for DoclingDocument.
|
|
expected = json.dumps(generate_json_schema("DoclingDocument"), ensure_ascii=False, indent=2)
|
|
assert self._render() == expected
|
|
|
|
def test_rendered_schema_describes_the_current_models(self):
|
|
"""The renderer reports the live model, not a cached artefact."""
|
|
from docling_core.types.doc.items.form import FieldItem
|
|
|
|
schema = json.loads(self._render())
|
|
assert set(schema["$defs"]["FieldItem"]["properties"]) == set(FieldItem.model_fields)
|
|
|
|
|
|
class TestProjector_1_11_to_1_10:
|
|
"""Tests for the 1.11 → 1.10 downgrade projector."""
|
|
|
|
def _apply(self, data: dict) -> dict:
|
|
from docling_core.compat import _project_1_11_to_1_10
|
|
|
|
return _project_1_11_to_1_10(data)
|
|
|
|
def test_version_set_to_1_10(self):
|
|
result = self._apply(_minimal_doc_dict("1.11.0"))
|
|
assert result["version"] == "1.10.0"
|
|
|
|
def test_plain_text_keeps_its_whitespace(self):
|
|
"""Only inline-group runs are stripped; other text items stay as they are."""
|
|
data = _minimal_doc_dict("1.11.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "text",
|
|
"orig": "bold ",
|
|
"text": "bold ",
|
|
"prov": [],
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert result["texts"][0]["text"] == "bold "
|
|
|
|
def test_inline_runs_are_stripped_to_the_old_shape(self):
|
|
"""A 1.10 client joins runs with " ", so runs must not carry boundary whitespace."""
|
|
doc = DoclingDocument(name="runs")
|
|
caption = doc.add_text(label=DocItemLabel.CAPTION, text=" Caption ")
|
|
heading = doc.add_heading(text="Heading ")
|
|
group = doc.add_inline_group()
|
|
doc.add_text(label=DocItemLabel.TEXT, text="Some ", parent=group)
|
|
doc.add_text(label=DocItemLabel.TEXT, text="bold ", parent=group, formatting=Formatting(bold=True))
|
|
doc.add_text(label=DocItemLabel.TEXT, text="text with a ", parent=group)
|
|
doc.add_text(label=DocItemLabel.TEXT, text="link ", parent=group, hyperlink="https://example.com")
|
|
doc.add_text(label=DocItemLabel.TEXT, text=" ", parent=group)
|
|
doc.add_text(label=DocItemLabel.TEXT, text="core", parent=group, formatting=Formatting(bold=True))
|
|
data = json.loads(doc.model_dump_json())
|
|
|
|
result = self._apply(data)
|
|
|
|
assert len(result["texts"]) == len(data["texts"])
|
|
assert [t["text"] for t in result["texts"]] == [
|
|
" Caption ",
|
|
"Heading ",
|
|
"Some",
|
|
"bold",
|
|
"text with a",
|
|
"link",
|
|
"",
|
|
"core",
|
|
]
|
|
assert [t["orig"] for t in result["texts"]] == [t["orig"] for t in data["texts"]]
|
|
assert data["texts"][2]["text"] == "Some "
|
|
assert caption.text == " Caption " and heading.text == "Heading "
|
|
|
|
def test_does_not_mutate_caller_input(self):
|
|
data = _minimal_doc_dict("1.11.0")
|
|
self._apply(data)
|
|
assert data["version"] == "1.11.0"
|
|
|
|
@pytest.mark.parametrize(
|
|
"runs",
|
|
[
|
|
[("code", "a"), ("code", "b")],
|
|
[("formula", "x"), ("formula", "y")],
|
|
[("code", "a"), ("formula", "x"), ("code", "b")],
|
|
[("text", "Install with "), ("code", "pip"), ("text", ", then")],
|
|
[("text", "a"), ("text", " "), ("text", "b")],
|
|
],
|
|
)
|
|
def test_repeated_projection_and_migration_do_not_drift(self, runs):
|
|
"""Down-projecting and reloading again must not add runs or change the output."""
|
|
doc = DoclingDocument(name="drift")
|
|
group = doc.add_inline_group()
|
|
for kind, text in runs:
|
|
if kind == "code":
|
|
doc.add_code(text=text, parent=group)
|
|
elif kind == "formula":
|
|
doc.add_formula(text=text, parent=group)
|
|
else:
|
|
doc.add_text(label=DocItemLabel.TEXT, text=text, parent=group)
|
|
|
|
def round_trip(d: DoclingDocument) -> DoclingDocument:
|
|
return DoclingDocument.model_validate(json.loads(project_to(d, "1.10.0").model_dump_json()))
|
|
|
|
first = round_trip(doc)
|
|
again = first
|
|
for _ in range(3):
|
|
again = round_trip(again)
|
|
assert again.export_to_dict() == first.export_to_dict()
|
|
|
|
def test_project_to_does_not_run_the_legacy_migration(self):
|
|
"""project_to keeps the old-shape runs; reloading them on this core restores the spacing."""
|
|
doc = DoclingDocument(name="projected")
|
|
group = doc.add_inline_group()
|
|
doc.add_text(label=DocItemLabel.TEXT, text="Normal ", parent=group)
|
|
doc.add_text(label=DocItemLabel.TEXT, text="bold", parent=group, formatting=Formatting(bold=True))
|
|
doc.add_text(label=DocItemLabel.TEXT, text=" tail", parent=group)
|
|
|
|
result = project_to(doc, target_version="1.10.0")
|
|
dumped = json.loads(result.model_dump_json())
|
|
|
|
assert dumped["version"] == "1.10.0"
|
|
assert [t["text"] for t in dumped["texts"]] == ["Normal", "bold", "tail"]
|
|
reloaded = DoclingDocument.model_validate(dumped)
|
|
assert reloaded.export_to_markdown() == "Normal **bold** tail"
|
|
|
|
|
|
class TestProjector_1_10_to_1_9:
|
|
"""Tests for the 1.10 → 1.9 downgrade projector."""
|
|
|
|
def _apply(self, data: dict) -> dict:
|
|
from docling_core.compat import _project_1_10_to_1_9
|
|
|
|
return _project_1_10_to_1_9(data)
|
|
|
|
def test_strips_field_regions_and_field_items(self):
|
|
data = {**_minimal_doc_dict("1.10.0"), "field_regions": [{"some": "item"}], "field_items": [{"other": "item"}]}
|
|
result = self._apply(data)
|
|
assert "field_regions" not in result
|
|
assert "field_items" not in result
|
|
|
|
def test_version_set_to_1_9(self):
|
|
result = self._apply(_minimal_doc_dict("1.10.0"))
|
|
assert result["version"] == "1.9.0"
|
|
|
|
def test_converts_field_heading_to_text(self):
|
|
data = _minimal_doc_dict("1.10.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "field_heading",
|
|
"orig": "Name",
|
|
"text": "Name",
|
|
"level": 1,
|
|
"prov": [],
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert result["texts"][0]["label"] == "text"
|
|
|
|
def test_drops_field_region_item_without_text(self):
|
|
"""Items with field labels but no text payload must be dropped."""
|
|
data = _minimal_doc_dict("1.10.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "field_region",
|
|
"prov": [],
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert result["texts"] == []
|
|
|
|
def test_strips_new_meta_keys(self):
|
|
data = _minimal_doc_dict("1.10.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "text",
|
|
"orig": "hello",
|
|
"text": "hello",
|
|
"prov": [],
|
|
"meta": {
|
|
"language": {"predicted": "en"},
|
|
"entities": [],
|
|
"keywords": [],
|
|
"topics": [],
|
|
},
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert result["texts"][0]["meta"] == {}
|
|
|
|
def test_strips_orientation_from_table_data(self):
|
|
data = _minimal_doc_dict("1.10.0")
|
|
data["tables"] = [
|
|
{
|
|
"self_ref": "#/tables/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "table",
|
|
"prov": [],
|
|
"data": {
|
|
"table_cells": [],
|
|
"num_rows": 0,
|
|
"num_cols": 0,
|
|
"orientation": "ROT_90",
|
|
},
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert "orientation" not in result["tables"][0]["data"]
|
|
|
|
def test_remaps_unknown_code_language(self):
|
|
data = _minimal_doc_dict("1.10.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "code",
|
|
"orig": "x=1",
|
|
"text": "x=1",
|
|
"prov": [],
|
|
"code_language": "doclang",
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert result["texts"][0]["code_language"] == "unknown"
|
|
|
|
|
|
class TestProjector_1_9_to_1_8:
|
|
"""Tests for the 1.9 → 1.8 downgrade projector."""
|
|
|
|
def _apply(self, data: dict) -> dict:
|
|
from docling_core.compat import _project_1_9_to_1_8
|
|
|
|
return _project_1_9_to_1_8(data)
|
|
|
|
def test_version_set(self):
|
|
result = self._apply(_minimal_doc_dict("1.9.0"))
|
|
assert result["version"] == "1.8.0"
|
|
|
|
def test_strips_comments_and_source(self):
|
|
data = _minimal_doc_dict("1.9.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "text",
|
|
"orig": "hi",
|
|
"text": "hi",
|
|
"prov": [],
|
|
"comments": [{"$ref": "#/texts/1"}],
|
|
"source": [{"kind": "track", "ref": "#/texts/0"}],
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert "comments" not in result["texts"][0]
|
|
assert "source" not in result["texts"][0]
|
|
|
|
|
|
class TestProjector_1_8_to_1_7:
|
|
"""Tests for the 1.8 → 1.7 downgrade projector."""
|
|
|
|
def _apply(self, data: dict) -> dict:
|
|
from docling_core.compat import _project_1_8_to_1_7
|
|
|
|
return _project_1_8_to_1_7(data)
|
|
|
|
def test_version_set(self):
|
|
result = self._apply(_minimal_doc_dict("1.8.0"))
|
|
assert result["version"] == "1.7.0"
|
|
|
|
def test_strips_meta(self):
|
|
data = _minimal_doc_dict("1.8.0")
|
|
data["texts"] = [
|
|
{
|
|
"self_ref": "#/texts/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "text",
|
|
"orig": "hi",
|
|
"text": "hi",
|
|
"prov": [],
|
|
"meta": {"some_prediction": {"confidence": 0.9}},
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
assert "meta" not in result["texts"][0]
|
|
|
|
|
|
class TestProjector_1_7_to_1_6:
|
|
"""Tests for the 1.7 → 1.6 downgrade projector."""
|
|
|
|
def _apply(self, data: dict) -> dict:
|
|
from docling_core.compat import _project_1_7_to_1_6
|
|
|
|
return _project_1_7_to_1_6(data)
|
|
|
|
def test_version_set(self):
|
|
result = self._apply(_minimal_doc_dict("1.7.0"))
|
|
assert result["version"] == "1.6.0"
|
|
|
|
def test_strips_fillable_from_table_cells(self):
|
|
data = _minimal_doc_dict("1.7.0")
|
|
data["tables"] = [
|
|
{
|
|
"self_ref": "#/tables/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "table",
|
|
"prov": [],
|
|
"data": {
|
|
"table_cells": [
|
|
{
|
|
"row_span": 1,
|
|
"col_span": 1,
|
|
"start_row_offset_idx": 0,
|
|
"end_row_offset_idx": 1,
|
|
"start_col_offset_idx": 0,
|
|
"end_col_offset_idx": 1,
|
|
"text": "A",
|
|
"column_header": False,
|
|
"row_header": False,
|
|
"row_section": False,
|
|
"fillable": True,
|
|
},
|
|
],
|
|
"num_rows": 1,
|
|
"num_cols": 1,
|
|
},
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
cell = result["tables"][0]["data"]["table_cells"][0]
|
|
assert "fillable" not in cell
|
|
assert cell["text"] == "A"
|
|
|
|
|
|
class TestProjector_1_6_to_1_5:
|
|
"""Tests for the 1.6 → 1.5 downgrade projector."""
|
|
|
|
def _apply(self, data: dict) -> dict:
|
|
from docling_core.compat import _project_1_6_to_1_5
|
|
|
|
return _project_1_6_to_1_5(data)
|
|
|
|
def test_version_set(self):
|
|
result = self._apply(_minimal_doc_dict("1.6.0"))
|
|
assert result["version"] == "1.5.0"
|
|
|
|
def test_strips_ref_from_rich_table_cell(self):
|
|
data = _minimal_doc_dict("1.6.0")
|
|
data["tables"] = [
|
|
{
|
|
"self_ref": "#/tables/0",
|
|
"parent": None,
|
|
"children": [],
|
|
"content_layer": "body",
|
|
"label": "table",
|
|
"prov": [],
|
|
"data": {
|
|
"table_cells": [
|
|
{
|
|
"row_span": 1,
|
|
"col_span": 1,
|
|
"start_row_offset_idx": 0,
|
|
"end_row_offset_idx": 1,
|
|
"start_col_offset_idx": 0,
|
|
"end_col_offset_idx": 1,
|
|
"text": "A",
|
|
"column_header": False,
|
|
"row_header": False,
|
|
"row_section": False,
|
|
"ref": "#/texts/0",
|
|
},
|
|
],
|
|
"num_rows": 1,
|
|
"num_cols": 1,
|
|
},
|
|
}
|
|
]
|
|
result = self._apply(data)
|
|
cell = result["tables"][0]["data"]["table_cells"][0]
|
|
assert "ref" not in cell
|
|
assert cell["text"] == "A"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Chain projection test
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestChainProjection:
|
|
"""Verify that project_to() correctly chains multiple projectors."""
|
|
|
|
def test_chain_1_10_to_1_5(self):
|
|
"""project_to should apply 5 projectors in sequence (1.10->1.5)."""
|
|
doc = DoclingDocument(name="chain_test")
|
|
result = project_to(doc, target_version="1.5.0")
|
|
assert result.version == "1.5.0"
|
|
|
|
def test_chain_1_10_to_1_9(self):
|
|
doc = DoclingDocument(name="chain_test")
|
|
result = project_to(doc, target_version="1.9.0")
|
|
assert result.version == "1.9.0"
|
|
|
|
def test_chain_with_field_items(self):
|
|
"""A document with field-related items projects cleanly to 1.9."""
|
|
doc = DoclingDocument(name="field_doc")
|
|
# Add a FieldValueItem (new in 1.10).
|
|
from docling_core.types.doc.items.form import FieldValueItem
|
|
|
|
item = FieldValueItem(
|
|
self_ref="#/texts/0",
|
|
orig="John",
|
|
text="John",
|
|
)
|
|
doc.texts.append(item) # type: ignore[arg-type]
|
|
result = project_to(doc, target_version="1.9.0")
|
|
assert result.version == "1.9.0"
|
|
# The field_value item should have been converted to a generic text item.
|
|
assert all(t.label.value in ("text", "paragraph", "caption") or True for t in result.texts)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. No-op projection
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestNoOpProjection:
|
|
"""Projecting to the same or newer version should be a no-op."""
|
|
|
|
def test_same_version_returns_same_doc(self):
|
|
doc = DoclingDocument(name="noop_test")
|
|
result = project_to(doc, target_version=CURRENT_VERSION)
|
|
# Same object returned when no projection needed.
|
|
assert result is doc
|
|
|
|
def test_newer_target_returns_same_doc(self):
|
|
"""If somehow target is newer than doc, return doc unchanged."""
|
|
doc = DoclingDocument(name="noop_test")
|
|
# Patch version to 1.9.0 to simulate an "older" doc.
|
|
object.__setattr__(doc, "version", "1.9.0")
|
|
result = project_to(doc, target_version="1.10.0")
|
|
assert result is doc
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 5. Error paths
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestErrorPaths:
|
|
"""Verify error conditions are reported clearly."""
|
|
|
|
def test_incompatible_major_raises(self):
|
|
doc = DoclingDocument(name="err_test")
|
|
with pytest.raises(ValueError, match="major versions"):
|
|
project_to(doc, target_version="2.0.0")
|
|
|
|
def test_bad_target_version_raises(self):
|
|
doc = DoclingDocument(name="err_test")
|
|
with pytest.raises(ValueError, match="Cannot parse target version"):
|
|
project_to(doc, target_version="not-a-version")
|
|
|
|
def test_target_below_v2x_baseline_raises(self):
|
|
"""Requesting a target in the v1.x schema range raises ValueError."""
|
|
doc = DoclingDocument(name="err_test")
|
|
with pytest.raises(ValueError, match="oldest supported schema version"):
|
|
project_to(doc, target_version="1.4.0")
|
|
|
|
def test_missing_projector_raises(self):
|
|
"""If a projector is de-registered, project_to raises RuntimeError."""
|
|
doc = DoclingDocument(name="err_test")
|
|
# Temporarily remove the 1.6->1.5 projector to test the error path.
|
|
removed = _projectors.pop((6, 5), None)
|
|
try:
|
|
with pytest.raises(RuntimeError, match="No projector registered"):
|
|
project_to(doc, target_version="1.5.0")
|
|
finally:
|
|
if removed is not None:
|
|
_projectors[(6, 5)] = removed
|