Commit Graph
34 Commits
Author SHA1 Message Date
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
b20b5677c0 fix(html-serializer): emit XHTML-compatible metadata attributes (#613)
* fix(html-serializer): emit XHTML-compatible metadata attributes

HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.

Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.

Fixes #598.

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>

* fix(html-serializer): rebase changes to 'main' and add tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(html-serializer): remove redundant CSS rule

Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-05-29 10:01:30 +02:00
Peter W. J. Staar 01f8d03cda fix: render RichTableCell descendants inside the cell (#621)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-26 16:46:10 +02:00
Cesar Berrospi Ramis 6cbdee9626 fix: prevent numeric precision loss in Markdown table serialization (#588)
* fix: preserve numeric precision in markdown serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: apply style conventions to test_serialization.py module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-04-15 17:16:57 +02:00
b435090fdf feat: split html table to headers and body (#532)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* remove unnecessary lines

Signed-off-by: odelliab <odelliab@il.ibm.com>

* split html tables

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add bs4 dependency

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* typo

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add beautifulsoup tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor(serializer): drop dependency 'beautifulsoup4' in HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): refactor tests for HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 10:40:05 +01:00
Ivan Traus 2debe0836f fix: prevent hang in export_to_markdown() on nested RichTableCells (#525)
* fix: prevent hang in export_to_markdown() on nested RichTableCells

When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got &#124;-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.

Fix: add two module-level helpers:

  _cell_content_has_table(item, doc) — walks the item subtree and
  returns True if any node is a TableItem.

  _mark_subtree_visited(item, doc, visited) — adds all nodes in the
  subtree to the shared 'visited' set that the document serializer uses
  to prevent double-emitting items.

In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.

For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.

Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* style: apply ruff formatting to markdown serializer

Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* test(markdown): cover descendant table detection in rich-cell helper

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): preserve nested table content as flattened text

Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items.  This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): use TextItem instead of DocItem in subtree text collector

Signed-off-by: Ivan Traus <ivan@liminary.io>

* refactor(markdown): flatten only nested tables, preserve sibling rich formatting

Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.

Signed-off-by: Ivan Traus <ivan@liminary.io>

---------

Signed-off-by: Ivan Traus <ivan@liminary.io>
2026-03-05 09:54:27 +01:00
Cesar Berrospi Ramis b8ef7bad1b feat: add WebVTT export and save functionality (#523)
* feat(vtt): export and save to WebVTT format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(vtt): omit empty blocks in parsing

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(vtt): add tests for exporting and saving to WebVTT

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-24 10:17:17 +01:00
Cesar Berrospi Ramis 04cf44b2c5 fix(serialization): add 'traverse_pictures' parameter to serializers (#501)
* fix(serializer): add 'traverse_pictures' to CommonParams

Expose 'traverse_pictures' parameter to serialier common parameters to control
whether to traverse into PictureItem objects to serialize their children.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): add test for 'traverse_pictures' parameter

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-03 15:07:55 +01:00
Cesar Berrospi Ramis 3b0b909c89 fix(markdown): add an option to compact table serialization (#495)
* fix(markdown): add an option to compact table serialization

Add an option in MarkdownTableSerializer to remove padding in tables in markdown serialization.
Propagate the option to MarkdownDocSerializer and DoclingDocument.
Remove unnecessary use of 'mode' argument in 'open' function when mode is 'r'.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(markdown): keep alignment marks in '_compact_table'

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-29 16:23:54 +01:00
Cesar Berrospi Ramis c8f3c01a61 feat: model and serializer for audio tracks (#426)
* refactor: move WebVTT data model from docling

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): deal with HTML entities in cue text spans

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): support more WebVTT models

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(DoclingDocument): create a new provenance model for media file types

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make WebVTTTimestamp public

Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set languages to a list of strings in ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): add test for ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make all WebVTT classes public for reuse

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): preserve newlines as WebVTTLineTerminator

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set ProvenanceTrack time fields as float

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): ensure start time offsets are in sequence

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): improve regex to remove note,region,style blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): parse the WebVTT file title

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): rebase to latest changes in idoctags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* feat(webvtt): add WebVTT serializer

Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): add 'text/vtt' as extra mimetype

Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): fix test with STYLE and NOTE blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style(webvtt): apply X | Y annotation instead of Optional, Union

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): simplify TrackProvenance model with tags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): align class and field names to new 'source' type

Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(DoclingDocument): drop the validation on field assignment

Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

refactor(webvtt): drop cue span classes, 'lang' and 'c' tags

Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-27 14:45:03 +01:00
Cesar Berrospi Ramis 334575cd08 fix(serialization): escape pipe symbol in single cell md serialization (#485)
fix(serialization): escape pipe characters in single cell markdown serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-22 13:06:04 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
Peter W. J. Staar dda9c885a3 feat: idoctags serialization and deserialization matching the iso proposal (#457)
* Working on ISO standard for IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the pre-commit, still tons of work to do

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated idoctag token class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* need to start fixing the bugs and add proper tests for idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the testing for dcotags and idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the location tokens in idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the DocumentToken

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the idt quot issue

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Added IDocTagsSerializationMode

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the deserializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the captions for floating items

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* still need to fix some seserialization tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* finally all is working and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* made the OTSL serialization and deserialization self-contained in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed some unclean code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* did some fixes and clean up

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests, expanded to deserialize to deal with nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the deserialization for formatting

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the formatting and the complex inline groups in nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added get_category

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* make IS_SELFCLOSING a set

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the regex

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the if ... return into if ... elif ... return

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the facets parsing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the footnotes for Tables and Pictures in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with latest doctags token table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed private name

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-17 14:38:10 +01:00
Peter W. J. Staar 9a42a3c860 feat: updating the idoctags serializer (#450)
* feat: updating the idoctags serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* feat: adding IDocTagsToken class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the list serializer in the doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added missing document serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added tests for content suppression

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests for serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* deal with differences between xml serialization in mac and linux

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (3)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (4)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixing the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed all the tests, except the no-content with lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the no-content in lists with doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-10 11:39:40 +01:00
Cesar Berrospi Ramis 8feb09f1cf fix: markdown serialization of hyperlink with code (#434)
* chore: leverage fixtures in tests and add type annotations

Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(markdown): serialize hyperlink as code always with single backticks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-11-21 14:05:29 +01:00
Panos Vagenas c9e5fb4a1c feat(experimental): add new DocTags serializer (#412)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-20 13:06:38 +01:00
Panos Vagenas 3d13b02756 fix: improve meta migration and warning handling (#417)
* fix: improve meta migration and warning handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove commented-out code

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 16:25:14 +01:00
Panos Vagenas 2ee3cacdd6 feat: add metadata model hierarchy (#408)
* feat: add metadata model hierarchy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add deprecation, add first migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend annotations migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update with feedback

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose main prediction

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* ideas on enforcing separation between standard and custom fields

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add custom field setter method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update Markdown serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert description, add include_non_meta, showcase custom serializer for summaries

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify customization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix reference exclusion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* eliminate serialization dupliation between meta & (legacy) annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove old file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix item used in get_parts for meta ser

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* serialize GroupItem meta prior to content, DocItem meta after content

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore ser order for all nodeitems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* move meta serialization into DocSerializer.serialize() to maintain seamless chunking integration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add allow- & block-lists for meta names, add std field name enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML serializer, document meta field names, rename SMILES field

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make TabularChartMetaField.title optional, expose new classes through __init__.py, add MetaUtils

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add DocTags serialization, revert smiles to smi to prevent confusion with plural

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-30 11:15:14 +01:00
Erik abfa81f065 feat: rendering of checkboxes in markdown (#381)
* added support for rendering checkboxes

* DCO Remediation Commit for Erik <erik@apption.com>

Signed-off-by: Erik <erik@apption.com>

* DCO Remediation Commit for Erik <erik@apption.com>

I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 7af470451b

Signed-off-by: Erik <erik@apption.com>

* added code suggestions for better checkboxes

* updated test data

* fixed formatting

* DCO Remediation Commit for Erik <erik@apption.com>

I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: fc7a57e1f7
I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 7ea737ecd4
I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 5c431b52c8

Signed-off-by: Erik <erik@apption.com>

---------

Signed-off-by: Erik <erik@apption.com>
2025-10-01 11:16:31 +02:00
Panos Vagenas 9df72083c0 fix(markdown): fix single-row table serialization (#385)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-10 16:11:34 +02:00
Panos Vagenas b60ac192ef feat: add support for heading with inline in HTML & DocTags (#379)
* test: refactor serialization tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* further refactor serialization tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename inline test file for markdown

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify Markdown handling of inline in heading / list item

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix input test data & make it more interesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML & DocTags support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize and refactor HTML ser case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize MD serializer too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-03 16:56:58 +02:00
Panos Vagenas 1d04154378 feat: add rich table cells (#368)
* feat: add rich table cells

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* propagate cell text resolution, cover row deletions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add doctags, fix referential integrity, expand tests, reenable mypy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify / remove serialize_cell

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update rich table cell refs in doc indexing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update notebook

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose new classes in `docling_core.types.doc`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-27 10:07:47 +02:00
Panos Vagenas 807d972f05 fix(HTML): fix nested list serialization edge cases (#367)
* fix(HTML): (initial state without fix) fix nested list serialization edge cases

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* actual fix

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-13 15:06:54 +02:00
Panos Vagenas 14a4fdee87 feat: remodel lists, add MD & HTML ser. params, enable unset marker (#339)
* feat: remodel lists

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* prepare test document separate markers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add MD/HTML serializer options, expand test data

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* create list groups where not in place

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add auto-increment logic

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore UnorderedList as deprecated alias (for backwards compatibility)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable unset marker case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make ordered markers able to be "unset" (empty) too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename default marker mode to auto

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-27 14:52:25 +02:00
Panos Vagenas c9374e8fe4 fix(markdown): add heading formatting, fix code & formula formatting (#336)
* feat(markdown): add formatted headings (show behavior before change)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add change and updated test data

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix when formatting & escaping are applied (depending on type)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-20 14:10:48 +02:00
Panos Vagenas dcc198f7c6 feat(doctags): add enclosing bbox to inline (#302)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-06 16:35:56 +02:00
Panos Vagenas d8a5256b2c feat: add table annotations (#304)
* feat: add table annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* refactor annotation types

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expand to HTML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce annotation serializer

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* Update dummy_doc.yaml

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-06-05 15:38:46 +02:00
Panos VagenasandMichele Dolfi f067c51c48 feat: add annotations in MD & HTML serialization (#295)
* feat: include annotations in MD & HTML serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* (HTML) move annotations into figcaptions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add explicit beginning/end markers, fix case of excluded refs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve annotation marking, extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* wrap captions (#305)

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* revert temp test changes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2025-05-27 10:49:13 +02:00
Panos Vagenas c49a50e76b fix(markdown): fix case of empty page break string (#298)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-20 10:51:24 +02:00
Panos Vagenas 6a7eb537eb feat: provide visualizer option in HTML split view (#294)
* feat: provide visualizer option in HTML split view

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* loosen test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-16 11:22:12 +02:00
Panos Vagenas d05fe08546 feat: promote serializers to stable API (#276)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-30 11:26:39 +02:00
d0a49da6de fix: HTML serialization for single image documents (#261)
* fix for HTML serialization for single image documents

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* minor refactor, add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-04-16 16:40:27 +02:00
Maxim LysakandMaksym Lysak caa8aeefae feat: Chart tabular data serialization for HTML serializer (#258)
* Add chart tabular data serialization to HTML serializer

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixing pre-commit issues

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixed table serialization for tabular chart data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* test for loading doctags with chart data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Improved loading doctags test with chart example, added tests for chart serialization into html and md

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-04-15 13:31:50 +02:00
Panos Vagenas 98c60bb65b perf: fix serialization performance (#249)
* perf: fix serialization performance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add missing leading HTML tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-11 15:37:52 +02:00