* test(doclang): add simple multi-page roundtrip test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add container-level threading
- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with v0.5 threading and heads
- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): switch to suppressing namespace and version by default
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with DocLang 0.4
- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add basic virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): extend virtual `<text>` support, unify el. head parsing
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align code/picture labels to DocLang, add LabelMode
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): emit bare `<value>` for read-only field values
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] align with DocLang v0.5
- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage
- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] drop legacy vocabulary
- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat: add keywords and topics meta fields
Introduce two simple, additive meta fields:
- KeywordsMetaField — unique list of keyword strings
- TopicsMetaField — unique list of topic / subject strings
Both attach to BaseMeta (node-level) and to the new DocumentMeta
(document-level), which is wired into DoclingDocument.meta. HTML,
Markdown, and Doclang serializers render the new fields as comma-
separated values. Schema bump is intentionally avoided since the
fields are optional and backward-compatible.
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* test: cover keywords and topics meta fields
Add round-trip, dedup, has_content, HTML escape, and Markdown
rendering tests for KeywordsMetaField, TopicsMetaField, and
DocumentMeta. Document-level meta absent-by-default is also covered.
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* style: satisfy ruff UP038 and import order
Convert tuple-form isinstance checks to PEP 604 unions and
reorder a misplaced import flagged by ruff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* fix: align with HTML attribute format from #613
Post-rebase onto main, the HTML metadata serializer now emits
`data-meta-name="<field>"` instead of valueless `data-meta-<field>`
and applies a single trailing html.escape pass. Drop the per-type
escape from the Keywords/Topics branch (was producing double
escaping after rebase) and update the two test assertions to the
new attribute shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* revert: remove DocLang keywords/topics serialization
Per discussion, DocLang is a standard whose extension is out of
scope for this PR. Drop the KeywordsMetaField/TopicsMetaField
branches from DoclangMetaSerializer and their imports; HTML and
Markdown serialization remain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* refactor: address review feedback on keywords/topics meta fields
- Replace MetaFieldName inline comments with Attributes docstring (TableFormerMode pattern)
- Swap list[str]+manual dedup → UniqueList[str] in KeywordsMetaField and TopicsMetaField
- Add Field(description, examples) to all BaseMeta fields distinguishing entities/keywords/topics
- Remove DocumentMeta class and DoclingDocument.meta field; document-level meta goes via body
- Consolidate keywords/topics tests into test_semantic_base_meta_fields_roundtrip_and_html_rendering
- Regenerate docs/DoclingDocument.json
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(meta): reuse validator between keywords and topics
Revert the deduplication as part of a validator.
Share the validator between keywords and topics meta classes.
Adopt the standard Google style for enum classes.
Adapt the regression tests to the changes above.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Ana Daniele <ana.daniele@ibm.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: vertical-rows-and-columns
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ran pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added Orientation to TableData to deal with vertical tables
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated docs
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* moved to ROT_0, etc
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the convention of the rotation
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix(html-serializer): emit XHTML-compatible metadata attributes
HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.
Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.
Fixes#598.
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
* fix(html-serializer): rebase changes to 'main' and add tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(html-serializer): remove redundant CSS rule
Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
When a `TableItem` produced empty serialized text, `TripletTableSerializer`
was still populating the shared `visited` set via `RichTableCell._get_text()`.
Downstream items were then treated as already chunked and the chunker
emitted 0 chunks (e.g. on DOCX layout/header-only tables).
This change:
- Isolates `visited` while exporting the table dataframe and only merges
the local copy back into the shared set when non-empty text is produced.
- Promotes the header row into a data row when the dataframe has 0 data
rows but non-empty column names, so header-only tables still emit text.
- Handles single-row single-column tables by emitting the cell text
directly instead of an empty triplet list.
- Adds `_flatten_table_text` as a fallback for tables whose triplet form
collapses to empty text.
- Guards against appending empty text to serialization parts.
Adds regression tests covering header-only tables, single-row
single-column tables, end-to-end `HierarchicalChunker`/`HybridChunker`
runs on header-only / single-cell rich tables, and verifies that empty
tables do not consume their children's refs.
Closesdocling-project/docling#3335
Signed-off-by: Manohar Mallipudi <manohar.m4475@yahoo.com>
* feat: extending BaseMeta with language, entities, etc for docling-agent
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ignore serialization of empty Meta in HTML
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* addressing the good comments from Cesar
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix(doclang): default DoclangDeserializer to page 1 and allow start offset
Doclang VLM output was assigning provenance to page 0 while DoclingDocument and ground truth use 1-based page numbers, which broke page-aligned matching.
- Default internal _page_no to 1 and add deserialize(..., page_no=1).
- Replace hardcoded self._page_no = 0 with the page_no argument.
- Update roundtrip_list_item_with_inline deserialized fixture for page 1.
* DCO Remediation Commit for Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
I, Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 0c34abd5c3
Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
---------
Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
Co-authored-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
* feat(serializer): add MsExcelMarkdownDocSerializer for sheet-name headings
Add `MsExcelMarkdownFallbackSerializer` and `MsExcelMarkdownDocSerializer`
to the serializer package so that `GroupLabel.SHEET` groups are rendered as
level-2 Markdown headings when exporting Excel-sourced documents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* DCO Remediation Commit for Smeet23 <smeetagrawal2003@gmail.com>
I, Smeet23 <smeetagrawal2003@gmail.com>, hereby add my Signed-off-by to this commit: 2a3808e5dc
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
* style: apply ruff formatter to markdown_excel.py
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
---------
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>