* feat(doclang): add .dclx archive save and load support
- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add support for `<description>` and `<summary>`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* What was removed
The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.
Deleted packages/modules
- docling_core/types/legacy_doc/ (whole folder)
- docling_core/types/rec/ (whole folder)
- docling_core/utils/legacy.py (legacy conversion functions)
- docling_core/utils/validate.py (RAW/ANN/OCR CLI)
Edited production code
- types/__init__.py — dropped the Record export
- utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
validate_unique_list/validate_datetime
- utils/generate_docs.py — dropped Record from MODELS
Removed resources/artifacts
- resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json
Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.
Verification
- No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
- from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
- Full project suite: 496 passed.
- generate_docs.py now emits only DoclingDocument.json and Generic.json.
The full plan is saved at _plans/remove-legacy-doc-and-rec.md.
Two notes worth flagging:
- Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
bump downstream.
- I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
approval flow.
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the checks.yaml
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the minimal_document_schema_flat.json
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/resources
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the files and updated the imports according to Cesar's suggestion
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the ruff linter
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): align with DocLang v0.6 picture model
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): enhance picture metadata handling with custom fields
- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): deserialize field regions and add round-trip fixtures
- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* test(doclang): add simple multi-page roundtrip test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add container-level threading
- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with v0.5 threading and heads
- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): switch to suppressing namespace and version by default
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with DocLang 0.4
- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add basic virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): extend virtual `<text>` support, unify el. head parsing
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align code/picture labels to DocLang, add LabelMode
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): emit bare `<value>` for read-only field values
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] align with DocLang v0.5
- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage
- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] drop legacy vocabulary
- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* fix: vertical-rows-and-columns
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ran pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added Orientation to TableData to deal with vertical tables
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated docs
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* moved to ROT_0, etc
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the convention of the rotation
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix(html-serializer): emit XHTML-compatible metadata attributes
HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.
Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.
Fixes#598.
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
* fix(html-serializer): rebase changes to 'main' and add tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(html-serializer): remove redundant CSS rule
Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(doclang): default DoclangDeserializer to page 1 and allow start offset
Doclang VLM output was assigning provenance to page 0 while DoclingDocument and ground truth use 1-based page numbers, which broke page-aligned matching.
- Default internal _page_no to 1 and add deserialize(..., page_no=1).
- Replace hardcoded self._page_no = 0 with the page_no argument.
- Update roundtrip_list_item_with_inline deserialized fixture for page 1.
* DCO Remediation Commit for Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
I, Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 0c34abd5c3
Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
---------
Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
Co-authored-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
* chore(serializer): add item label to outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): leverage pydantic for validating and dumping in outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(serializer): test '_format_indented_text_line' from outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): pass kwargs to all outline serializers for format consistency
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat: add 'meta' field to DoclingDocument at root level
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(metadata): reuse DoclingDocument objects with fixtures
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): add document-level meta to the outline serialization
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test: add helper function to compare or regenerate ground truth files
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test: regenerate (upgrade) ground truth files
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(serializer): set default indent to 2 spaces in outline serializer
To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(serializer): add line break after ref in outline md serialization
To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor: rollback adding 'meta' field to DoclingDocument
Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat: make an experimental outline serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* work ongoing
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* working outline serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* style(outline): align style of outline serializer to docling-core's
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): ensure 'include_non_meta' works as designed by CommonParams
Ensure the optional parameter 'include_non_meta' works as designed by CommonParams when used by OutlineDocSerializer.
Add regression tests for OutlineDocSerializer.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): outline serializer to optionally return in JSON format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): enable a custom set of labels for TOC serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(serializer): add summary to title in test data
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): ensure outline serializer keeps structured fields in JSON format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): refactor outline serializer to create proper markdown
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(serializer): use pydantic models for the outine serializer JSON representation
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test: rename outline serialization module to align it with others
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): add indented text format for outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: prevent hang in export_to_markdown() on nested RichTableCells
When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got |-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.
Fix: add two module-level helpers:
_cell_content_has_table(item, doc) — walks the item subtree and
returns True if any node is a TableItem.
_mark_subtree_visited(item, doc, visited) — adds all nodes in the
subtree to the shared 'visited' set that the document serializer uses
to prevent double-emitting items.
In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.
For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.
Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* style: apply ruff formatting to markdown serializer
Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* test(markdown): cover descendant table detection in rich-cell helper
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): preserve nested table content as flattened text
Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items. This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): use TextItem instead of DocItem in subtree text collector
Signed-off-by: Ivan Traus <ivan@liminary.io>
* refactor(markdown): flatten only nested tables, preserve sibling rich formatting
Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.
Signed-off-by: Ivan Traus <ivan@liminary.io>
---------
Signed-off-by: Ivan Traus <ivan@liminary.io>
* feat(vtt): export and save to WebVTT format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(vtt): omit empty blocks in parsing
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(vtt): add tests for exporting and saving to WebVTT
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>