* test(doclang): add simple multi-page roundtrip test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add container-level threading
- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with v0.5 threading and heads
- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): switch to suppressing namespace and version by default
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with DocLang 0.4
- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add basic virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): extend virtual `<text>` support, unify el. head parsing
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align code/picture labels to DocLang, add LabelMode
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): emit bare `<value>` for read-only field values
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] align with DocLang v0.5
- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage
- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] drop legacy vocabulary
- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* fix(doclang): default DoclangDeserializer to page 1 and allow start offset
Doclang VLM output was assigning provenance to page 0 while DoclingDocument and ground truth use 1-based page numbers, which broke page-aligned matching.
- Default internal _page_no to 1 and add deserialize(..., page_no=1).
- Replace hardcoded self._page_no = 0 with the page_no argument.
- Update roundtrip_list_item_with_inline deserialized fixture for page 1.
* DCO Remediation Commit for Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
I, Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 0c34abd5c3
Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
---------
Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
Co-authored-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
* fix(Doclang): suppress empty elements in Doclang serialization
Add `suppress_empty_elements` parameter to `DoclangParams` that, when
enabled, omits text items, picture groups, tables, and inline elements
that produce no content instead of emitting empty open/close tag pairs.
This avoids cluttering OCR-oriented output with vacant tags.
Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
* added tests
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
---------
Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
Co-authored-by: MatteoOmenetti <omenetti.matteo@gmail.com>
* chore(serializer): add item label to outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): leverage pydantic for validating and dumping in outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(serializer): test '_format_indented_text_line' from outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): pass kwargs to all outline serializers for format consistency
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat: add 'meta' field to DoclingDocument at root level
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(metadata): reuse DoclingDocument objects with fixtures
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): add document-level meta to the outline serialization
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test: add helper function to compare or regenerate ground truth files
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test: regenerate (upgrade) ground truth files
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(serializer): set default indent to 2 spaces in outline serializer
To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(serializer): add line break after ref in outline md serialization
To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor: rollback adding 'meta' field to DoclingDocument
Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat: make an experimental outline serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* work ongoing
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* working outline serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* style(outline): align style of outline serializer to docling-core's
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): ensure 'include_non_meta' works as designed by CommonParams
Ensure the optional parameter 'include_non_meta' works as designed by CommonParams when used by OutlineDocSerializer.
Add regression tests for OutlineDocSerializer.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): outline serializer to optionally return in JSON format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): enable a custom set of labels for TOC serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(serializer): add summary to title in test data
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): ensure outline serializer keeps structured fields in JSON format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): refactor outline serializer to create proper markdown
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(serializer): use pydantic models for the outine serializer JSON representation
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test: rename outline serialization module to align it with others
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(serializer): add indented text format for outline serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* Added ruff to dev dependencies
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added ruff settings to pyproject.toml as in docling
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Cleanup uf pyproject.toml
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Copied settings for ruff pre-commit hooks from docling
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Excluded test/data/** from ruff formatting / linting
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* ruff format
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* ruff check --fix
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Ignored some more rules
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Fixed the rest of the errors that would only concern 1 - 3 files
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added another ignore related to df for DataFrame names
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Modified CONTRIBUTING.md such that black / isort are replaced by ruff
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added UP045 to ignore list such that Optional[...] does not raise
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved .flake8 configs to pyproject.toml
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved autoflake to be used with ruff
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Removed flake8 from .pre-commit hooks
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Adding PD901 to ignore list such that pre-commit hooks run fully again
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Replaced dtype | None syntax by Optional[dtype] in remaining places
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* chore: fix 'test' ref in pyproject
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove typing List, Set, Tuple, Dict
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove UP015 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove UP034 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: normalize dashes in comments and docstrings
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove PD901 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove C403 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove C403, C413, C416 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove E203, F811 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* Working on ISO standard for IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the pre-commit, still tons of work to do
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated idoctag token class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* need to start fixing the bugs and add proper tests for idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* refactored the testing for dcotags and idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updating the location tokens in idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the DocumentToken
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the idt quot issue
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* Added IDocTagsSerializationMode
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* working on the deserializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the captions for floating items
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* still need to fix some seserialization tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* finally all is working and reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* made the OTSL serialization and deserialization self-contained in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed some unclean code
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* did some fixes and clean up
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests, expanded to deserialize to deal with nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added the deserialization for formatting
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the formatting and the complex inline groups in nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added get_category
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* make IS_SELFCLOSING a set
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the regex
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the if ... return into if ... elif ... return
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the facets parsing
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the footnotes for Tables and Pictures in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated with latest doctags token table
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed private name
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: updating the idoctags serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: adding IDocTagsToken class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the list serializer in the doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added missing document serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added tests for content suppression
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests for serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ran pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* deal with differences between xml serialization in mac and linux
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (3)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (4)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixing the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed all the tests, except the no-content with lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the no-content in lists with doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* chore: remove unnecesary copyright headers
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* reformatted, still need to fix a test ...
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* docs: Recommend installing all groups for mypy to work properly
Signed-off-by: Eugene <fogaprod@gmail.com>
* chore: Fix some incomplete type defs
Signed-off-by: Eugene <fogaprod@gmail.com>
---------
Signed-off-by: Eugene <fogaprod@gmail.com>
* Add chart tabular data serialization to HTML serializer
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* Fixing pre-commit issues
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* Fixed table serialization for tabular chart data
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* test for loading doctags with chart data
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
* Improved loading doctags test with chart example, added tests for chart serialization into html and md
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
---------
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>