DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.
The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.
Fixes#685
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* feat(doclang): add .dclx archive save and load support
- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add support for `<description>` and `<summary>`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): align with DocLang v0.6 picture model
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): enhance picture metadata handling with custom fields
- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): deserialize field regions and add round-trip fixtures
- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* test(doclang): add simple multi-page roundtrip test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add container-level threading
- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with v0.5 threading and heads
- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): switch to suppressing namespace and version by default
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with DocLang 0.4
- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add basic virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): extend virtual `<text>` support, unify el. head parsing
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align code/picture labels to DocLang, add LabelMode
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): emit bare `<value>` for read-only field values
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] align with DocLang v0.5
- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage
- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] drop legacy vocabulary
- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* fix(Doclang): suppress empty elements in Doclang serialization
Add `suppress_empty_elements` parameter to `DoclangParams` that, when
enabled, omits text items, picture groups, tables, and inline elements
that produce no content instead of emitting empty open/close tag pairs.
This avoids cluttering OCR-oriented output with vacant tags.
Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
* added tests
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
---------
Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
Co-authored-by: MatteoOmenetti <omenetti.matteo@gmail.com>