Commit Graph
139 Commits
Author SHA1 Message Date
Panos Vagenas 3704874312 feat(doclang): add support for .dclx archive, <description>, <summary> (#658)
* feat(doclang): add .dclx archive save and load support

- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): add support for `<description>` and `<summary>`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-25 18:06:10 +02:00
Panos Vagenas 9dddcf7137 fix: clamp provenance bboxes and DocLang locations to valid page bounds (#643)
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:44:20 +02:00
Panos Vagenas 34bd98eb3d feat(doclang): default include_version to True (#642)
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:20:20 +02:00
Max SwainandChristoph Auer ab3666c081 perf: Use OpenCV over PIL for PNG encoding in ImageRef.from_pil (#562)
* perf: Use OpenCV over PIL for PNG encoding

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>

* refactor: Make opencv optional and add fallback to PIL

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>

* refactor: Update opencv dependency to use dev group

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>

* Fix RGBA vs RGB handling in openCV

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update test GT

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* empty commit to trigger DCO

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2026-06-17 16:44:14 +02:00
Panos Vagenas 0b7690509e feat(doclang): align with DocLang v0.6, add field region deser. (#636)
* feat(doclang): align with DocLang v0.6 picture model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): enhance picture metadata handling with custom fields

- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): deserialize field regions and add round-trip fixtures

- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-11 12:30:09 +02:00
Panos Vagenas 5d740c260a feat(doclang): promote serializer and deserializer (#633)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 15:23:21 +02:00
Panos Vagenas ba110e8671 chore(doclang): add v0.5 container-level threading (#631)
* test(doclang): add simple multi-page roundtrip test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add container-level threading

- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with v0.5 threading and heads

- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 11:17:14 +02:00
Panos Vagenas 28342dcc6c chore(doclang): align with DocLang v0.5 foundation (#630)
* chore(doclang): switch to suppressing namespace and version by default

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with DocLang 0.4

- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add basic virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): extend virtual `<text>` support, unify el. head parsing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align code/picture labels to DocLang, add LabelMode

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): emit bare `<value>` for read-only field values

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] align with DocLang v0.5

- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage

- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] drop legacy vocabulary

- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-08 16:42:35 +02:00
Peter W. J. Staar c1c9ea80a5 feat: support vertical orientation in TableData row/column bounding boxes (#622)
* fix: vertical-rows-and-columns

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added Orientation to TableData to deal with vertical tables

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* moved to ROT_0, etc

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the convention of the rotation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-29 17:16:08 +02:00
b20b5677c0 fix(html-serializer): emit XHTML-compatible metadata attributes (#613)
* fix(html-serializer): emit XHTML-compatible metadata attributes

HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.

Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.

Fixes #598.

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>

* fix(html-serializer): rebase changes to 'main' and add tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(html-serializer): remove redundant CSS rule

Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-05-29 10:01:30 +02:00
Panos Vagenas 430473ae54 chore(doclang): add document index support (#616)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-05-18 16:51:58 +02:00
Panos Vagenas 9d4b288a33 chore(doclang): align lists, tables, groups, headings (#610)
* chore(doclang): align lists, tables, groups, headings

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add aligned virtual text support to lists & tables

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-05-18 12:36:02 +02:00
Peter W. J. Staar 014948b0e8 feat: updated the HTML serialization (#609)
* updated the HTML serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-12 16:49:08 +02:00
Ahmed NassarandAhmed Nassar AHN@zurich.ibm.com 048f1720f6 fix(doclang): default DoclangDeserializer to page 1 (#590)
* fix(doclang): default DoclangDeserializer to page 1 and allow start offset

Doclang VLM output was assigning provenance to page 0 while DoclingDocument and ground truth use 1-based page numbers, which broke page-aligned matching.

- Default internal _page_no to 1 and add deserialize(..., page_no=1).
- Replace hardcoded self._page_no = 0 with the page_no argument.
- Update roundtrip_list_item_with_inline deserialized fixture for page 1.

* DCO Remediation Commit for Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>

I, Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 0c34abd5c3

Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>

---------

Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
Co-authored-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
2026-04-21 17:54:48 +02:00
Matteo b72af126b7 fix(DocLang): fix chemistry serialization (#584)
added correct serialization of chemistry into doclang

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-04-17 07:33:45 +02:00
Panos Vagenas 00c3bb223d feat(Doclang): add newline handling (#575)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-04-01 17:13:37 +02:00
Peter W. J. StaarandPanos Vagenas f20068db91 feat: add transforms in the hierarchy (#572)
* feat: add transforms in the hierarchy

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* implement hierachizing & flattening based on shift/tree operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflict

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-04-01 17:12:06 +02:00
Panos Vagenas c9b51520d2 fix(Doclang): improve checkbox serialization & deserialization (#570)
Wrap in `<text>` if needed.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 12:31:07 +02:00
Panos Vagenas a1535bc1d8 fix(Doclang): fix serialization order in text items (#571)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:44:45 +02:00
Panos Vagenas fe9bbfbb0f feat(Doclang): add content layer support (#568)
* feat(Doclang): add content layer support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename layer attribute

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:40:49 +02:00
fb3b603bc4 feat: add handwriting support (#561)
* feat: Add HANDWRITTEN_TEXT label support

Add full integration for the HANDWRITTEN_TEXT document item label.

Changes:
- tokens.py: Add HANDWRITTEN_TEXT to DocumentToken enum and mapping
- document.py: Add HANDWRITTEN_TEXT to DEFAULT_EXPORT_LABELS

Also adds test/test_handwritten_text_label.py for integration tests.

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: acbdfa3902e74da44a5844ad0cecab8657da4904

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

* add handwriting support to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: 000ccc55c5

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

---------

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-26 08:43:29 +01:00
Panos Vagenas 159eb8f021 fix(Doclang): fix group serialization (#566)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-25 14:31:32 +01:00
Panos Vagenas 4807381050 chore: improve key-value migration (#559)
* fix: improve picture KV migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* consider page nr in bbox indexing & comparisons

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unused internal method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve further cases

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix: support arbitrary cell ids

(Until now a cell's id was assumed to be its position in `.cells`)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-20 14:32:30 +01:00
Cesar Berrospi Ramis de575b7c3a chore: add item label in outline serializer (#558)
* chore(serializer): add item label to outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): leverage pydantic for validating and dumping in outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): test '_format_indented_text_line' from outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): pass kwargs to all outline serializers for format consistency

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-20 13:16:57 +01:00
Cesar Berrospi Ramis afa5bd9e3a chore: document-level metadata serialization via body field (#551)
* feat: add 'meta' field to DoclingDocument at root level

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(metadata): reuse DoclingDocument objects with fixtures

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add document-level meta to the outline serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: add helper function to compare or regenerate ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: regenerate (upgrade) ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): set default indent to 2 spaces in outline serializer

To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): add line break after ref in outline md serialization

To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: rollback adding 'meta' field to DoclingDocument

Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-17 14:55:52 +01:00
Panos Vagenas b93d5a3920 feat: introduce field data model incl. Doclang serialization (#519)
* feat: introduce new kv data model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add form-table test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update invoice test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include pre-migration YAML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add nesting test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to field naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve prov migration & location serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable proper nested serialization without inline

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add field marker and test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize marker naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* updated API

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align with TextItem conventions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make FieldItem a DocItem

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend KV migration scope, extend tree manipulation operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend kv migration tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test with KV nesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve key splitting and location assignment

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unnecessary files

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_content parameter, expose export_to_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts, add save_as_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:37:42 +01:00
Peter W. J. StaarandCesar Berrospi Ramis 8d7859eeec feat: make an experimental outline serializer (#415)
* feat: make an experimental outline serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* work ongoing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working outline serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* style(outline): align style of outline serializer to docling-core's

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): ensure 'include_non_meta' works as designed by CommonParams

Ensure the optional parameter 'include_non_meta' works as designed by CommonParams when used by OutlineDocSerializer.
Add regression tests for OutlineDocSerializer.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): outline serializer to optionally return in JSON format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): enable a custom set of labels for TOC serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): add summary to title in test data

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): ensure outline serializer keeps structured fields in JSON format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): refactor outline serializer to create proper markdown

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): use pydantic models for the outine serializer JSON representation

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: rename outline serialization module to align it with others

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add indented text format for outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 15:14:40 +01:00
samiuc e363c951d8 feat: add plain-text serializer (#522)
* feat: add plain-text serializer

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

* fix code coverage

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

---------

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>
2026-03-06 10:20:21 +01:00
Ivan Traus 2debe0836f fix: prevent hang in export_to_markdown() on nested RichTableCells (#525)
* fix: prevent hang in export_to_markdown() on nested RichTableCells

When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got &#124;-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.

Fix: add two module-level helpers:

  _cell_content_has_table(item, doc) — walks the item subtree and
  returns True if any node is a TableItem.

  _mark_subtree_visited(item, doc, visited) — adds all nodes in the
  subtree to the shared 'visited' set that the document serializer uses
  to prevent double-emitting items.

In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.

For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.

Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* style: apply ruff formatting to markdown serializer

Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* test(markdown): cover descendant table detection in rich-cell helper

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): preserve nested table content as flattened text

Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items.  This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): use TextItem instead of DocItem in subtree text collector

Signed-off-by: Ivan Traus <ivan@liminary.io>

* refactor(markdown): flatten only nested tables, preserve sibling rich formatting

Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.

Signed-off-by: Ivan Traus <ivan@liminary.io>

---------

Signed-off-by: Ivan Traus <ivan@liminary.io>
2026-03-05 09:54:27 +01:00
Cesar Berrospi Ramis b8ef7bad1b feat: add WebVTT export and save functionality (#523)
* feat(vtt): export and save to WebVTT format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(vtt): omit empty blocks in parsing

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(vtt): add tests for exporting and saving to WebVTT

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-24 10:17:17 +01:00
Panos Vagenas aec74d4eb0 fix(Doclang): align image mode, defaulting to placeholder (#506)
* fix(Doclang): align image mode, defaulting to placeholder

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update crop test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:38:27 +01:00
Panos Vagenas 134cf75b69 test(Doclang): add chart serialization test (#498)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:31:20 +01:00
Panos Vagenas 1d969d4e37 fix: fix document re-indexing (#510)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-12 10:11:14 +01:00
Panos Vagenas 193c25f083 fix(Doclang): fix image URI serialization (#504)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-04 09:39:03 +01:00
Panos Vagenas 8005892f6c fix(DocTags): fix deserialization to populate picture meta fields (#505)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-03 16:50:34 +01:00
Matteo de2b729617 fix(DocTags): fix picture classification deserialization (#500)
* fixed picture classification deserialization from doctags

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

* fixed picture classification deserialization from doctags

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-02-03 10:58:11 +01:00
Panos Vagenas 1d8b78cbad fix(Doclang): fix checkbox serialization (#503)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-03 09:54:26 +01:00
Panos Vagenas ad86b85b92 chore: rename IDocTags to Doclang (#494)
* chore: rename IDocTags to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename remaining file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-30 16:41:09 +01:00
Peter W. J. Staar fd27df1f07 fix(html): visualize picture meta as html collapsible (#497)
* feat: make meta of pictures cleaner in HTML

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-01-30 14:45:26 +01:00
Panos Vagenas 549a2f1472 fix(IDocTags): fix default location resolution handling (#492)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-29 13:52:59 +01:00
Panos Vagenas 62f8d4d838 feat(IDocTags): add rich table support (#491)
* feat(IDocTags): add rich table serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-28 13:44:57 +01:00
Cesar Berrospi Ramis c8f3c01a61 feat: model and serializer for audio tracks (#426)
* refactor: move WebVTT data model from docling

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): deal with HTML entities in cue text spans

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): support more WebVTT models

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(DoclingDocument): create a new provenance model for media file types

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make WebVTTTimestamp public

Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set languages to a list of strings in ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): add test for ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make all WebVTT classes public for reuse

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): preserve newlines as WebVTTLineTerminator

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set ProvenanceTrack time fields as float

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): ensure start time offsets are in sequence

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): improve regex to remove note,region,style blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): parse the WebVTT file title

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): rebase to latest changes in idoctags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* feat(webvtt): add WebVTT serializer

Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): add 'text/vtt' as extra mimetype

Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): fix test with STYLE and NOTE blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style(webvtt): apply X | Y annotation instead of Optional, Union

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): simplify TrackProvenance model with tags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): align class and field names to new 'source' type

Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(DoclingDocument): drop the validation on field assignment

Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

refactor(webvtt): drop cue span classes, 'lang' and 'c' tags

Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-27 14:45:03 +01:00
Panos Vagenas fdcdfd1f27 feat(IDocTags): add content wrapping for handling whitespace (#489)
* feat(IDocTags): add content wrapping for handling whitespace

Also:
- fix caption serialiazation
- consolidate indendation via `pretty_indentation` param

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix content tag value

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-23 15:20:54 +01:00
e5c0015fbe fix(IDocTags): align code labels with Linguist (#484)
* fix(IDocTags): align code labels with Linguist

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fixed bug code blocks programming language not parsed

Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* it works

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* removed debugging line from test

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>
Co-authored-by: mao <mao@login-c.zuvela.ibm.com>
Co-authored-by: Matteo Omenetti <omenetti.matteo@gmail.com>
2026-01-23 15:20:25 +01:00
Panos Vagenas d9e8d37a91 fix(IDocTags): fix InlineGroup serialization and deserialization (#477)
fix(IDocTags): fix `InlineGroup` serialization & deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 16:45:46 +01:00
Panos Vagenas 17bd21ee14 feat(IDocTags): add fine-grained content serialization filtering (#476)
* feat(IDocTags): add fine-grained content serialization filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update token analysis example

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 13:49:29 +01:00
Panos Vagenas 0c9c70db76 feat(IDocTags): add XML escape modes (#471)
* feat: add XML escape modes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove workaround for IDE debugging

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix inline handling, descope entities mode for now

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix escaping/CDATA interference with inline groups, formatting, code languages; add underline tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup, test expansion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-12 12:46:26 +01:00
Panos Vagenas 51a01a62c9 fix: skip annotation migration if respective meta present (#469)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-08 18:02:18 +01:00
SivaandPanos Vagenas 1d2e0c7ebe feat(DocItem): Add comments field for linking annotations to document… (#465)
* feat(DocItem): Add comments field for linking annotations to document items

Implements support for linking comments (from Word/PPT documents) to their
annotated content using the established FloatingItem/RefItem pattern.

Changes:
- Add `comments: List[RefItem]` field to DocItem class
- Update `_update_breadth_first_with_lookup()` to handle comment references on deletion
- Bump CURRENT_VERSION to 1.9.0
- Fix version comparison bug (string vs integer for minor version)
- Add 4 new tests for comments functionality
- Update test data files for new schema

Closes: docling-project/docling#464
Related: docling-project/docling#2834

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>

* improve comment Pydantic serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_comment, update tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce fine-granular references with span ranges

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify last test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-08 05:00:57 +01:00
Peter W. J. Staar dda9c885a3 feat: idoctags serialization and deserialization matching the iso proposal (#457)
* Working on ISO standard for IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the pre-commit, still tons of work to do

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated idoctag token class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* need to start fixing the bugs and add proper tests for idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the testing for dcotags and idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the location tokens in idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the DocumentToken

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the idt quot issue

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Added IDocTagsSerializationMode

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the deserializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the captions for floating items

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* still need to fix some seserialization tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* finally all is working and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* made the OTSL serialization and deserialization self-contained in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed some unclean code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* did some fixes and clean up

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests, expanded to deserialize to deal with nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the deserialization for formatting

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the formatting and the complex inline groups in nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added get_category

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* make IS_SELFCLOSING a set

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the regex

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the if ... return into if ... elif ... return

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the facets parsing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the footnotes for Tables and Pictures in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with latest doctags token table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed private name

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-17 14:38:10 +01:00