* fix(html-serializer): emit XHTML-compatible metadata attributes
HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.
Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.
Fixes#598.
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
* fix(html-serializer): rebase changes to 'main' and add tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(html-serializer): remove redundant CSS rule
Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: prevent hang in export_to_markdown() on nested RichTableCells
When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got |-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.
Fix: add two module-level helpers:
_cell_content_has_table(item, doc) — walks the item subtree and
returns True if any node is a TableItem.
_mark_subtree_visited(item, doc, visited) — adds all nodes in the
subtree to the shared 'visited' set that the document serializer uses
to prevent double-emitting items.
In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.
For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.
Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* style: apply ruff formatting to markdown serializer
Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* test(markdown): cover descendant table detection in rich-cell helper
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): preserve nested table content as flattened text
Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items. This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): use TextItem instead of DocItem in subtree text collector
Signed-off-by: Ivan Traus <ivan@liminary.io>
* refactor(markdown): flatten only nested tables, preserve sibling rich formatting
Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.
Signed-off-by: Ivan Traus <ivan@liminary.io>
---------
Signed-off-by: Ivan Traus <ivan@liminary.io>
* feat(vtt): export and save to WebVTT format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(vtt): omit empty blocks in parsing
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(vtt): add tests for exporting and saving to WebVTT
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): add 'traverse_pictures' to CommonParams
Expose 'traverse_pictures' parameter to serialier common parameters to control
whether to traverse into PictureItem objects to serialize their children.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(serializer): add test for 'traverse_pictures' parameter
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(markdown): add an option to compact table serialization
Add an option in MarkdownTableSerializer to remove padding in tables in markdown serialization.
Propagate the option to MarkdownDocSerializer and DoclingDocument.
Remove unnecessary use of 'mode' argument in 'open' function when mode is 'r'.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(markdown): keep alignment marks in '_compact_table'
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor: move WebVTT data model from docling
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(webvtt): deal with HTML entities in cue text spans
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): support more WebVTT models
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(DoclingDocument): create a new provenance model for media file types
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): make WebVTTTimestamp public
Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): set languages to a list of strings in ProvenanceTrack
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests(webvtt): add test for ProvenanceTrack
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): make all WebVTT classes public for reuse
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): preserve newlines as WebVTTLineTerminator
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): set ProvenanceTrack time fields as float
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): ensure start time offsets are in sequence
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): improve regex to remove note,region,style blocks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): parse the WebVTT file title
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): rebase to latest changes in idoctags
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat(webvtt): add WebVTT serializer
Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(webvtt): add 'text/vtt' as extra mimetype
Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests(webvtt): fix test with STYLE and NOTE blocks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style(webvtt): apply X | Y annotation instead of Optional, Union
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): simplify TrackProvenance model with tags
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): align class and field names to new 'source' type
Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(DoclingDocument): drop the validation on field assignment
Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
refactor(webvtt): drop cue span classes, 'lang' and 'c' tags
Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* Added ruff to dev dependencies
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added ruff settings to pyproject.toml as in docling
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Cleanup uf pyproject.toml
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Copied settings for ruff pre-commit hooks from docling
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Excluded test/data/** from ruff formatting / linting
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* ruff format
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* ruff check --fix
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Ignored some more rules
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Fixed the rest of the errors that would only concern 1 - 3 files
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added another ignore related to df for DataFrame names
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Modified CONTRIBUTING.md such that black / isort are replaced by ruff
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added UP045 to ignore list such that Optional[...] does not raise
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved .flake8 configs to pyproject.toml
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved autoflake to be used with ruff
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Removed flake8 from .pre-commit hooks
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Adding PD901 to ignore list such that pre-commit hooks run fully again
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Replaced dtype | None syntax by Optional[dtype] in remaining places
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* chore: fix 'test' ref in pyproject
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove typing List, Set, Tuple, Dict
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove UP015 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove UP034 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: normalize dashes in comments and docstrings
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove PD901 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove C403 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove C403, C413, C416 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove E203, F811 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* Working on ISO standard for IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the pre-commit, still tons of work to do
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated idoctag token class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* need to start fixing the bugs and add proper tests for idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* refactored the testing for dcotags and idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updating the location tokens in idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the DocumentToken
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the idt quot issue
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* Added IDocTagsSerializationMode
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* working on the deserializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the captions for floating items
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* still need to fix some seserialization tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* finally all is working and reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* made the OTSL serialization and deserialization self-contained in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed some unclean code
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* did some fixes and clean up
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests, expanded to deserialize to deal with nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added the deserialization for formatting
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the formatting and the complex inline groups in nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added get_category
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* make IS_SELFCLOSING a set
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the regex
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the if ... return into if ... elif ... return
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the facets parsing
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the footnotes for Tables and Pictures in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated with latest doctags token table
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed private name
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: updating the idoctags serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: adding IDocTagsToken class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the list serializer in the doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added missing document serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added tests for content suppression
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests for serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ran pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* deal with differences between xml serialization in mac and linux
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (3)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (4)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixing the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed all the tests, except the no-content with lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the no-content in lists with doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* chore: leverage fixtures in tests and add type annotations
Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(markdown): serialize hyperlink as code always with single backticks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* Add chart tabular data serialization to HTML serializer
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* Fixing pre-commit issues
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* Fixed table serialization for tabular chart data
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* test for loading doctags with chart data
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* Improved loading doctags test with chart example, added tests for chart serialization into html and md
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
---------
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>