Commit Graph
23 Commits
Author SHA1 Message Date
Max Hsu b869c67cf8 fix(doclang): clamp deep section headers instead of raising (#688)
DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.

The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.

Fixes #685

Signed-off-by: Max Hsu <maxmilian@gmail.com>
2026-07-27 13:48:31 +02:00
Akhil Trivedi 26a50ab140 fix(doclang): sanitize XML-illegal text characters (#687)
Signed-off-by: AkhilTrivediX <118957648+AkhilTrivediX@users.noreply.github.com>
2026-07-27 13:48:15 +02:00
Panos Vagenas 3704874312 feat(doclang): add support for .dclx archive, <description>, <summary> (#658)
* feat(doclang): add .dclx archive save and load support

- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): add support for `<description>` and `<summary>`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-25 18:06:10 +02:00
Panos Vagenas 9dddcf7137 fix: clamp provenance bboxes and DocLang locations to valid page bounds (#643)
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:44:20 +02:00
Panos Vagenas 34bd98eb3d feat(doclang): default include_version to True (#642)
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:20:20 +02:00
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
Panos Vagenas 0b7690509e feat(doclang): align with DocLang v0.6, add field region deser. (#636)
* feat(doclang): align with DocLang v0.6 picture model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): enhance picture metadata handling with custom fields

- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): deserialize field regions and add round-trip fixtures

- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-11 12:30:09 +02:00
Panos Vagenas 5d740c260a feat(doclang): promote serializer and deserializer (#633)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 15:23:21 +02:00
Panos Vagenas ba110e8671 chore(doclang): add v0.5 container-level threading (#631)
* test(doclang): add simple multi-page roundtrip test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add container-level threading

- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with v0.5 threading and heads

- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 11:17:14 +02:00
Panos Vagenas 28342dcc6c chore(doclang): align with DocLang v0.5 foundation (#630)
* chore(doclang): switch to suppressing namespace and version by default

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with DocLang 0.4

- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add basic virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): extend virtual `<text>` support, unify el. head parsing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align code/picture labels to DocLang, add LabelMode

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): emit bare `<value>` for read-only field values

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] align with DocLang v0.5

- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage

- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] drop legacy vocabulary

- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-08 16:42:35 +02:00
Panos Vagenas 430473ae54 chore(doclang): add document index support (#616)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-05-18 16:51:58 +02:00
Panos Vagenas 9d4b288a33 chore(doclang): align lists, tables, groups, headings (#610)
* chore(doclang): align lists, tables, groups, headings

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add aligned virtual text support to lists & tables

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-05-18 12:36:02 +02:00
Panos Vagenas 00c3bb223d feat(Doclang): add newline handling (#575)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-04-01 17:13:37 +02:00
Panos Vagenas fe9bbfbb0f feat(Doclang): add content layer support (#568)
* feat(Doclang): add content layer support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename layer attribute

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:40:49 +02:00
fb3b603bc4 feat: add handwriting support (#561)
* feat: Add HANDWRITTEN_TEXT label support

Add full integration for the HANDWRITTEN_TEXT document item label.

Changes:
- tokens.py: Add HANDWRITTEN_TEXT to DocumentToken enum and mapping
- document.py: Add HANDWRITTEN_TEXT to DEFAULT_EXPORT_LABELS

Also adds test/test_handwritten_text_label.py for integration tests.

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: acbdfa3902e74da44a5844ad0cecab8657da4904

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

* add handwriting support to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: 000ccc55c5

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

---------

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-26 08:43:29 +01:00
Panos Vagenas 159eb8f021 fix(Doclang): fix group serialization (#566)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-25 14:31:32 +01:00
Ahmed NassarandMatteoOmenetti 91ee7e2302 fix(Doclang): suppress empty elements in Doclang serialization (#554)
* fix(Doclang): suppress empty elements in Doclang serialization

Add `suppress_empty_elements` parameter to `DoclangParams` that, when
enabled, omits text items, picture groups, tables, and inline elements
that produce no content instead of emitting empty open/close tag pairs.
This avoids cluttering OCR-oriented output with vacant tags.

Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>

* added tests

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
Co-authored-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-03-20 14:49:59 +01:00
Panos Vagenas b56f75190f chore(Doclang): remove inline element (#517)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:58:42 +01:00
Panos Vagenas b93d5a3920 feat: introduce field data model incl. Doclang serialization (#519)
* feat: introduce new kv data model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add form-table test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update invoice test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include pre-migration YAML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add nesting test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to field naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve prov migration & location serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable proper nested serialization without inline

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add field marker and test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize marker naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* updated API

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align with TextItem conventions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make FieldItem a DocItem

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend KV migration scope, extend tree manipulation operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend kv migration tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test with KV nesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve key splitting and location assignment

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unnecessary files

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_content parameter, expose export_to_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts, add save_as_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:37:42 +01:00
Panos Vagenas aec74d4eb0 fix(Doclang): align image mode, defaulting to placeholder (#506)
* fix(Doclang): align image mode, defaulting to placeholder

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update crop test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:38:27 +01:00
Panos Vagenas 134cf75b69 test(Doclang): add chart serialization test (#498)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:31:20 +01:00
Panos Vagenas 1d8b78cbad fix(Doclang): fix checkbox serialization (#503)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-03 09:54:26 +01:00
Panos Vagenas ad86b85b92 chore: rename IDocTags to Doclang (#494)
* chore: rename IDocTags to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename remaining file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-30 16:41:09 +01:00