Commit Graph
247 Commits
Author SHA1 Message Date
Panos Vagenas fe9bbfbb0f feat(Doclang): add content layer support (#568)
* feat(Doclang): add content layer support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename layer attribute

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:40:49 +02:00
fb3b603bc4 feat: add handwriting support (#561)
* feat: Add HANDWRITTEN_TEXT label support

Add full integration for the HANDWRITTEN_TEXT document item label.

Changes:
- tokens.py: Add HANDWRITTEN_TEXT to DocumentToken enum and mapping
- document.py: Add HANDWRITTEN_TEXT to DEFAULT_EXPORT_LABELS

Also adds test/test_handwritten_text_label.py for integration tests.

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: acbdfa3902e74da44a5844ad0cecab8657da4904

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

* add handwriting support to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: 000ccc55c5

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

---------

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-26 08:43:29 +01:00
Panos Vagenas 0cfb663275 fix: extend validation to address duplicate refs (#565)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-25 15:43:28 +01:00
Panos Vagenas 159eb8f021 fix(Doclang): fix group serialization (#566)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-25 14:31:32 +01:00
Ahmed NassarandMatteoOmenetti 91ee7e2302 fix(Doclang): suppress empty elements in Doclang serialization (#554)
* fix(Doclang): suppress empty elements in Doclang serialization

Add `suppress_empty_elements` parameter to `DoclangParams` that, when
enabled, omits text items, picture groups, tables, and inline elements
that produce no content instead of emitting empty open/close tag pairs.
This avoids cluttering OCR-oriented output with vacant tags.

Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>

* added tests

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
Co-authored-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-03-20 14:49:59 +01:00
Panos Vagenas 4807381050 chore: improve key-value migration (#559)
* fix: improve picture KV migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* consider page nr in bbox indexing & comparisons

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unused internal method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve further cases

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix: support arbitrary cell ids

(Until now a cell's id was assumed to be its position in `.cells`)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-20 14:32:30 +01:00
Cesar Berrospi Ramis de575b7c3a chore: add item label in outline serializer (#558)
* chore(serializer): add item label to outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): leverage pydantic for validating and dumping in outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): test '_format_indented_text_line' from outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): pass kwargs to all outline serializers for format consistency

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-20 13:16:57 +01:00
samiuc 3e030edc6f fix: expose traverse_pictures in export_to_markdown and export_to_text (#557)
* fix: expose traverse_pictures in export_to_markdown and export_to_text

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

* address review comment

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

---------

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>
2026-03-20 08:12:54 +01:00
Cesar Berrospi Ramis afa5bd9e3a chore: document-level metadata serialization via body field (#551)
* feat: add 'meta' field to DoclingDocument at root level

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(metadata): reuse DoclingDocument objects with fixtures

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add document-level meta to the outline serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: add helper function to compare or regenerate ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: regenerate (upgrade) ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): set default indent to 2 spaces in outline serializer

To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): add line break after ref in outline md serialization

To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: rollback adding 'meta' field to DoclingDocument

Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-17 14:55:52 +01:00
Panos Vagenas c57e50ac43 fix: improve rich table cell validation (#550)
Extend validation to also cover case of missing reference to
rich child on table item.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-16 17:13:53 +01:00
Panos Vagenas b56f75190f chore(Doclang): remove inline element (#517)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:58:42 +01:00
Panos Vagenas b93d5a3920 feat: introduce field data model incl. Doclang serialization (#519)
* feat: introduce new kv data model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add form-table test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update invoice test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include pre-migration YAML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add nesting test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to field naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve prov migration & location serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable proper nested serialization without inline

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add field marker and test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize marker naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* updated API

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align with TextItem conventions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make FieldItem a DocItem

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend KV migration scope, extend tree manipulation operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend kv migration tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test with KV nesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve key splitting and location assignment

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unnecessary files

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_content parameter, expose export_to_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts, add save_as_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:37:42 +01:00
Peter W. J. StaarandCesar Berrospi Ramis 8d7859eeec feat: make an experimental outline serializer (#415)
* feat: make an experimental outline serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* work ongoing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working outline serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* style(outline): align style of outline serializer to docling-core's

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): ensure 'include_non_meta' works as designed by CommonParams

Ensure the optional parameter 'include_non_meta' works as designed by CommonParams when used by OutlineDocSerializer.
Add regression tests for OutlineDocSerializer.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): outline serializer to optionally return in JSON format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): enable a custom set of labels for TOC serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): add summary to title in test data

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): ensure outline serializer keeps structured fields in JSON format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): refactor outline serializer to create proper markdown

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): use pydantic models for the outine serializer JSON representation

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: rename outline serialization module to align it with others

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add indented text format for outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 15:14:40 +01:00
Cesar Berrospi Ramis af50f1cb07 feat: profile a document or collection (#511)
* feat: profile a document or collection

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(profiler): add deciles and histograms

Add deciles and histograms to the Docling collection statistics.
Add an example script to plot histograms.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(profiler): add option to plot log frequencies in histogram

Add the option to plot the histogram frequencies in logarithmic scale.
Extend README with documentation on the document profiler.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(profiler): cover missing lines in doc_profiler with tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 13:36:38 +01:00
b435090fdf feat: split html table to headers and body (#532)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* remove unnecessary lines

Signed-off-by: odelliab <odelliab@il.ibm.com>

* split html tables

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add bs4 dependency

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* typo

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add beautifulsoup tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor(serializer): drop dependency 'beautifulsoup4' in HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): refactor tests for HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 10:40:05 +01:00
Anish Raghavendra e00125c477 feat: handle wide table outliers with LineBasedTokenChunker (#536)
* feat: handle large prefixes by splitting into multiple chunks in LineBasedTokenChunker

Signed-off-by: z404 <anishr890@gmail.com>

* feat: add option to omit table headers on row overflow

Signed-off-by: z404 <anishr890@gmail.com>

* chore: add warning when prefix is omitted on overflow

Signed-off-by: z404 <anishr890@gmail.com>

* refactor: simplify prefix chunking logic by removing redundant method

Signed-off-by: z404 <anishr890@gmail.com>

* fix: ensure prefix appears as standalone chunk when all lines overflow

Signed-off-by: z404 <anishr890@gmail.com>

* docs: add usage examples for LineBasedTokenChunker with prefix handling

Signed-off-by: z404 <anishr890@gmail.com>

* docs: remove examples and describe parameters for LineBasedTokenChunker

Signed-off-by: z404 <anishr890@gmail.com>

* docs: simplify class docstring by removing parameter details

Signed-off-by: z404 <anishr890@gmail.com>

---------

Signed-off-by: z404 <anishr890@gmail.com>
2026-03-13 07:29:46 +01:00
Michele Dolfi 4eb0d20d04 feat: Loosen dependency version constraints (#534)
* update deps

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix code to support all versions and regen tests

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-03-09 05:27:48 +01:00
Cesar Berrospi Ramis a661bb10cb fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens (#533)
* fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests: add a pytest fixture to test LineBasedTokenChunker efficiently

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: apply style conventions to test_line_chunker.py test module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-06 12:52:30 +01:00
samiuc e363c951d8 feat: add plain-text serializer (#522)
* feat: add plain-text serializer

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

* fix code coverage

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

---------

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>
2026-03-06 10:20:21 +01:00
Ivan Traus 2debe0836f fix: prevent hang in export_to_markdown() on nested RichTableCells (#525)
* fix: prevent hang in export_to_markdown() on nested RichTableCells

When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got &#124;-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.

Fix: add two module-level helpers:

  _cell_content_has_table(item, doc) — walks the item subtree and
  returns True if any node is a TableItem.

  _mark_subtree_visited(item, doc, visited) — adds all nodes in the
  subtree to the shared 'visited' set that the document serializer uses
  to prevent double-emitting items.

In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.

For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.

Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* style: apply ruff formatting to markdown serializer

Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* test(markdown): cover descendant table detection in rich-cell helper

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): preserve nested table content as flattened text

Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items.  This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): use TextItem instead of DocItem in subtree text collector

Signed-off-by: Ivan Traus <ivan@liminary.io>

* refactor(markdown): flatten only nested tables, preserve sibling rich formatting

Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.

Signed-off-by: Ivan Traus <ivan@liminary.io>

---------

Signed-off-by: Ivan Traus <ivan@liminary.io>
2026-03-05 09:54:27 +01:00
ea359bcc63 feat: table aware chunking (#527)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-04 16:25:57 +01:00
Matvei SmirnovandCesar Berrospi Ramis c566268e0a fix: rich table triplet serialization (#425)
* fix: rich table triplet serialization

Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: remove kwargs from 'export_to_dataframe' signature

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-26 11:39:34 +01:00
Cesar Berrospi Ramis 73b07572cf fix: support single-column table default serialization (#526)
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-24 12:23:01 +01:00
Cesar Berrospi Ramis b8ef7bad1b feat: add WebVTT export and save functionality (#523)
* feat(vtt): export and save to WebVTT format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(vtt): omit empty blocks in parsing

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(vtt): add tests for exporting and saving to WebVTT

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-24 10:17:17 +01:00
Ultizan 6032c7c175 fix: accept relative URIs in PdfHyperlink without validation failure (#520)
PDF hyperlinks may contain relative paths, internal bookmarks, or
fragment-only references that are not valid absolute URLs. The strict
AnyUrl validation on PdfHyperlink.uri caused the entire page preprocess
stage to fail when such URIs were encountered, resulting in empty
documents and lost content.

Change uri type to Union[AnyUrl, str] with a field_validator that
attempts AnyUrl parsing first (preserving structured metadata like
scheme/host/path) and falls back to str for non-absolute URIs.

Signed-off-by: Ultizan <ultizan@gmail.com>
2026-02-23 16:11:38 +01:00
Christoph Auer 6a04db77aa fix: shift KV/Form graph cell page numbers during DoclingDocument.concatenate (#521)
fix: When concatenating docs, adjust page numbers the GraphCell elements appear on

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-02-23 10:35:16 +01:00
Cesar Berrospi Ramis a3b6e3fb89 fix(chunker): propagate 'traverse_pictures' parameter to chunker (#518)
* fix(chunker): propagate 'traverse_pictures' parameter from serializer to chunker

Propagate 'traverse_pictures' parameter from the 'serializer_provider' to the HierarchicalChunker, to ensure
that chunks include any TextItem children from PictureItem.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(chunker): simplify imports

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-20 20:37:43 +01:00
Panos Vagenas 9ba605d225 fix(Doclang): fix table cell content deserialization (#512)
* fix(DocLang): fix table cell `<content>` deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove irrelevant file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:42:30 +01:00
Panos Vagenas aec74d4eb0 fix(Doclang): align image mode, defaulting to placeholder (#506)
* fix(Doclang): align image mode, defaulting to placeholder

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update crop test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:38:27 +01:00
Panos Vagenas 134cf75b69 test(Doclang): add chart serialization test (#498)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:31:20 +01:00
Panos Vagenas 1d969d4e37 fix: fix document re-indexing (#510)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-12 10:11:14 +01:00
Panos Vagenas 193c25f083 fix(Doclang): fix image URI serialization (#504)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-04 09:39:03 +01:00
Panos Vagenas 8005892f6c fix(DocTags): fix deserialization to populate picture meta fields (#505)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-03 16:50:34 +01:00
Cesar Berrospi Ramis 04cf44b2c5 fix(serialization): add 'traverse_pictures' parameter to serializers (#501)
* fix(serializer): add 'traverse_pictures' to CommonParams

Expose 'traverse_pictures' parameter to serialier common parameters to control
whether to traverse into PictureItem objects to serialize their children.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): add test for 'traverse_pictures' parameter

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-03 15:07:55 +01:00
Matteo de2b729617 fix(DocTags): fix picture classification deserialization (#500)
* fixed picture classification deserialization from doctags

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

* fixed picture classification deserialization from doctags

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-02-03 10:58:11 +01:00
Panos Vagenas 1d8b78cbad fix(Doclang): fix checkbox serialization (#503)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-03 09:54:26 +01:00
Panos Vagenas ad86b85b92 chore: rename IDocTags to Doclang (#494)
* chore: rename IDocTags to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename remaining file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-30 16:41:09 +01:00
Peter W. J. Staar fd27df1f07 fix(html): visualize picture meta as html collapsible (#497)
* feat: make meta of pictures cleaner in HTML

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-01-30 14:45:26 +01:00
Cesar Berrospi Ramis 3b0b909c89 fix(markdown): add an option to compact table serialization (#495)
* fix(markdown): add an option to compact table serialization

Add an option in MarkdownTableSerializer to remove padding in tables in markdown serialization.
Propagate the option to MarkdownDocSerializer and DoclingDocument.
Remove unnecessary use of 'mode' argument in 'open' function when mode is 'r'.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(markdown): keep alignment marks in '_compact_table'

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-29 16:23:54 +01:00
Panos Vagenas 549a2f1472 fix(IDocTags): fix default location resolution handling (#492)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-29 13:52:59 +01:00
Panos Vagenas 62f8d4d838 feat(IDocTags): add rich table support (#491)
* feat(IDocTags): add rich table serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-28 13:44:57 +01:00
Cesar Berrospi Ramis c8f3c01a61 feat: model and serializer for audio tracks (#426)
* refactor: move WebVTT data model from docling

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): deal with HTML entities in cue text spans

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): support more WebVTT models

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(DoclingDocument): create a new provenance model for media file types

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make WebVTTTimestamp public

Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set languages to a list of strings in ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): add test for ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make all WebVTT classes public for reuse

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): preserve newlines as WebVTTLineTerminator

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set ProvenanceTrack time fields as float

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): ensure start time offsets are in sequence

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): improve regex to remove note,region,style blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): parse the WebVTT file title

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): rebase to latest changes in idoctags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* feat(webvtt): add WebVTT serializer

Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): add 'text/vtt' as extra mimetype

Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): fix test with STYLE and NOTE blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style(webvtt): apply X | Y annotation instead of Optional, Union

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): simplify TrackProvenance model with tags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): align class and field names to new 'source' type

Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(DoclingDocument): drop the validation on field assignment

Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

refactor(webvtt): drop cue span classes, 'lang' and 'c' tags

Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-27 14:45:03 +01:00
Panos Vagenas fdcdfd1f27 feat(IDocTags): add content wrapping for handling whitespace (#489)
* feat(IDocTags): add content wrapping for handling whitespace

Also:
- fix caption serialiazation
- consolidate indendation via `pretty_indentation` param

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix content tag value

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-23 15:20:54 +01:00
e5c0015fbe fix(IDocTags): align code labels with Linguist (#484)
* fix(IDocTags): align code labels with Linguist

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fixed bug code blocks programming language not parsed

Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* it works

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* removed debugging line from test

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>
Co-authored-by: mao <mao@login-c.zuvela.ibm.com>
Co-authored-by: Matteo Omenetti <omenetti.matteo@gmail.com>
2026-01-23 15:20:25 +01:00
Cesar Berrospi Ramis 334575cd08 fix(serialization): escape pipe symbol in single cell md serialization (#485)
fix(serialization): escape pipe characters in single cell markdown serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-22 13:06:04 +01:00
Peter W. J. Staar a6d2be687e fix: Fix transparency rendering in all visualizers (#481)
* fix: fixed the transparency

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-01-20 09:35:26 +01:00
Panos Vagenas d9e8d37a91 fix(IDocTags): fix InlineGroup serialization and deserialization (#477)
fix(IDocTags): fix `InlineGroup` serialization & deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 16:45:46 +01:00
Panos Vagenas 17bd21ee14 feat(IDocTags): add fine-grained content serialization filtering (#476)
* feat(IDocTags): add fine-grained content serialization filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update token analysis example

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 13:49:29 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
Panos Vagenas 0c9c70db76 feat(IDocTags): add XML escape modes (#471)
* feat: add XML escape modes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove workaround for IDE debugging

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix inline handling, descope entities mode for now

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix escaping/CDATA interference with inline groups, formatting, code languages; add underline tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup, test expansion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-12 12:46:26 +01:00