Commit Graph
65 Commits
Author SHA1 Message Date
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
Panos Vagenas 5d740c260a feat(doclang): promote serializer and deserializer (#633)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 15:23:21 +02:00
Panos Vagenas ba110e8671 chore(doclang): add v0.5 container-level threading (#631)
* test(doclang): add simple multi-page roundtrip test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add container-level threading

- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with v0.5 threading and heads

- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 11:17:14 +02:00
Panos Vagenas 28342dcc6c chore(doclang): align with DocLang v0.5 foundation (#630)
* chore(doclang): switch to suppressing namespace and version by default

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with DocLang 0.4

- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add basic virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): extend virtual `<text>` support, unify el. head parsing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align code/picture labels to DocLang, add LabelMode

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): emit bare `<value>` for read-only field values

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] align with DocLang v0.5

- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage

- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] drop legacy vocabulary

- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-08 16:42:35 +02:00
Panos Vagenas 430473ae54 chore(doclang): add document index support (#616)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-05-18 16:51:58 +02:00
Panos Vagenas 9d4b288a33 chore(doclang): align lists, tables, groups, headings (#610)
* chore(doclang): align lists, tables, groups, headings

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add aligned virtual text support to lists & tables

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-05-18 12:36:02 +02:00
48c5b97593 fix(DocLang): fix chemistry serialization. (#607)
* fix chemistry serialization. use molecule meta instead of classification

* DCO Remediation Commit for VWE@zurich.ibm.com;1G0002848;Valery Weber <vwe@login4.bluevela.rmf.ibm.com>

I, VWE@zurich.ibm.com;1G0002848;Valery Weber <vwe@login4.bluevela.rmf.ibm.com>, hereby add my Signed-off-by to this commit: e77b7e9478

Signed-off-by: VWE@zurich.ibm.com;1G0002848;Valery Weber <vwe@login3.bluevela.rmf.ibm.com>

---------

Signed-off-by: VWE@zurich.ibm.com;1G0002848;Valery Weber <vwe@login3.bluevela.rmf.ibm.com>
Co-authored-by: VWE@zurich.ibm.com;1G0002848;Valery Weber <vwe@login4.bluevela.rmf.ibm.com>
Co-authored-by: VWE@zurich.ibm.com;1G0002848;Valery Weber <vwe@login3.bluevela.rmf.ibm.com>
2026-05-08 13:43:26 +02:00
Ahmed NassarandAhmed Nassar AHN@zurich.ibm.com 048f1720f6 fix(doclang): default DoclangDeserializer to page 1 (#590)
* fix(doclang): default DoclangDeserializer to page 1 and allow start offset

Doclang VLM output was assigning provenance to page 0 while DoclingDocument and ground truth use 1-based page numbers, which broke page-aligned matching.

- Default internal _page_no to 1 and add deserialize(..., page_no=1).
- Replace hardcoded self._page_no = 0 with the page_no argument.
- Update roundtrip_list_item_with_inline deserialized fixture for page 1.

* DCO Remediation Commit for Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>

I, Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 0c34abd5c3

Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>

---------

Signed-off-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
Co-authored-by: Ahmed Nassar AHN@zurich.ibm.com <AHN@zurich.ibm.com>
2026-04-21 17:54:48 +02:00
Matteo b72af126b7 fix(DocLang): fix chemistry serialization (#584)
added correct serialization of chemistry into doclang

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-04-17 07:33:45 +02:00
Cesar Berrospi Ramis 18f573899b feat(ouline): extend OutlineDocSerializer with filtering capabilities (#580)
* feat(outline): add filtering params to OutlineDocSerializer

Add start_item and max_level parameters for flexible outline generation.
Refactor get_parts to reduce complexity below Ruff C901 threshold.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(outline): preserve spans in serialization results

Pass span_source to create_ser_result in all outline serializers.
Add test_outline_serialization_spans to verify get_unique_doc_items().

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(outline): reduce complexity and improve code quality

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-04-09 09:56:02 +02:00
Panos Vagenas 00c3bb223d feat(Doclang): add newline handling (#575)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-04-01 17:13:37 +02:00
Panos Vagenas c9b51520d2 fix(Doclang): improve checkbox serialization & deserialization (#570)
Wrap in `<text>` if needed.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 12:31:07 +02:00
Panos Vagenas a1535bc1d8 fix(Doclang): fix serialization order in text items (#571)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:44:45 +02:00
Panos Vagenas fe9bbfbb0f feat(Doclang): add content layer support (#568)
* feat(Doclang): add content layer support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename layer attribute

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:40:49 +02:00
fb3b603bc4 feat: add handwriting support (#561)
* feat: Add HANDWRITTEN_TEXT label support

Add full integration for the HANDWRITTEN_TEXT document item label.

Changes:
- tokens.py: Add HANDWRITTEN_TEXT to DocumentToken enum and mapping
- document.py: Add HANDWRITTEN_TEXT to DEFAULT_EXPORT_LABELS

Also adds test/test_handwritten_text_label.py for integration tests.

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: acbdfa3902e74da44a5844ad0cecab8657da4904

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

* add handwriting support to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: 000ccc55c5

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

---------

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-26 08:43:29 +01:00
Panos Vagenas 159eb8f021 fix(Doclang): fix group serialization (#566)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-25 14:31:32 +01:00
Ahmed NassarandMatteoOmenetti 91ee7e2302 fix(Doclang): suppress empty elements in Doclang serialization (#554)
* fix(Doclang): suppress empty elements in Doclang serialization

Add `suppress_empty_elements` parameter to `DoclangParams` that, when
enabled, omits text items, picture groups, tables, and inline elements
that produce no content instead of emitting empty open/close tag pairs.
This avoids cluttering OCR-oriented output with vacant tags.

Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>

* added tests

Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Ahmed Nassar <nassarofficial@gmail.com>
Signed-off-by: MatteoOmenetti <omenetti.matteo@gmail.com>
Co-authored-by: MatteoOmenetti <omenetti.matteo@gmail.com>
2026-03-20 14:49:59 +01:00
Cesar Berrospi Ramis de575b7c3a chore: add item label in outline serializer (#558)
* chore(serializer): add item label to outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): leverage pydantic for validating and dumping in outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): test '_format_indented_text_line' from outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): pass kwargs to all outline serializers for format consistency

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-20 13:16:57 +01:00
Cesar Berrospi Ramis afa5bd9e3a chore: document-level metadata serialization via body field (#551)
* feat: add 'meta' field to DoclingDocument at root level

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(metadata): reuse DoclingDocument objects with fixtures

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add document-level meta to the outline serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: add helper function to compare or regenerate ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: regenerate (upgrade) ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): set default indent to 2 spaces in outline serializer

To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): add line break after ref in outline md serialization

To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: rollback adding 'meta' field to DoclingDocument

Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-17 14:55:52 +01:00
Panos Vagenas b56f75190f chore(Doclang): remove inline element (#517)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:58:42 +01:00
Panos Vagenas b93d5a3920 feat: introduce field data model incl. Doclang serialization (#519)
* feat: introduce new kv data model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add form-table test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update invoice test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include pre-migration YAML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add nesting test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to field naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve prov migration & location serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable proper nested serialization without inline

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add field marker and test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize marker naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* updated API

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align with TextItem conventions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make FieldItem a DocItem

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend KV migration scope, extend tree manipulation operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend kv migration tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test with KV nesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve key splitting and location assignment

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unnecessary files

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_content parameter, expose export_to_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts, add save_as_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:37:42 +01:00
Peter W. J. StaarandCesar Berrospi Ramis 8d7859eeec feat: make an experimental outline serializer (#415)
* feat: make an experimental outline serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* work ongoing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working outline serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* style(outline): align style of outline serializer to docling-core's

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): ensure 'include_non_meta' works as designed by CommonParams

Ensure the optional parameter 'include_non_meta' works as designed by CommonParams when used by OutlineDocSerializer.
Add regression tests for OutlineDocSerializer.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): outline serializer to optionally return in JSON format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): enable a custom set of labels for TOC serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): add summary to title in test data

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(serializer): ensure outline serializer keeps structured fields in JSON format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): refactor outline serializer to create proper markdown

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): use pydantic models for the outine serializer JSON representation

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: rename outline serialization module to align it with others

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add indented text format for outline serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 15:14:40 +01:00
Panos Vagenas 9ba605d225 fix(Doclang): fix table cell content deserialization (#512)
* fix(DocLang): fix table cell `<content>` deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove irrelevant file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:42:30 +01:00
Panos Vagenas aec74d4eb0 fix(Doclang): align image mode, defaulting to placeholder (#506)
* fix(Doclang): align image mode, defaulting to placeholder

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update crop test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-13 10:38:27 +01:00
Panos Vagenas 2793dda9d7 fix: switch XML parsing (#509)
* fix: switch xml parsing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* move types package to dev

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-11 14:39:34 +01:00
Panos Vagenas 193c25f083 fix(Doclang): fix image URI serialization (#504)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-04 09:39:03 +01:00
Panos Vagenas 1d8b78cbad fix(Doclang): fix checkbox serialization (#503)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-02-03 09:54:26 +01:00
Panos Vagenas ad86b85b92 chore: rename IDocTags to Doclang (#494)
* chore: rename IDocTags to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename remaining file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-30 16:41:09 +01:00
Panos Vagenas 549a2f1472 fix(IDocTags): fix default location resolution handling (#492)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-29 13:52:59 +01:00
Panos Vagenas 62f8d4d838 feat(IDocTags): add rich table support (#491)
* feat(IDocTags): add rich table serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-28 13:44:57 +01:00
Panos Vagenas fdcdfd1f27 feat(IDocTags): add content wrapping for handling whitespace (#489)
* feat(IDocTags): add content wrapping for handling whitespace

Also:
- fix caption serialiazation
- consolidate indendation via `pretty_indentation` param

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix content tag value

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-23 15:20:54 +01:00
e5c0015fbe fix(IDocTags): align code labels with Linguist (#484)
* fix(IDocTags): align code labels with Linguist

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fixed bug code blocks programming language not parsed

Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* it works

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* removed debugging line from test

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>
Co-authored-by: mao <mao@login-c.zuvela.ibm.com>
Co-authored-by: Matteo Omenetti <omenetti.matteo@gmail.com>
2026-01-23 15:20:25 +01:00
Michele Dolfi 01373c4aab fix: drop python 3.9 and pin built tree-sitter versions (#487)
* drop python 3.9

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply linter for python 3.10 changes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-01-22 19:40:35 +01:00
Panos Vagenas d9e8d37a91 fix(IDocTags): fix InlineGroup serialization and deserialization (#477)
fix(IDocTags): fix `InlineGroup` serialization & deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 16:45:46 +01:00
Panos Vagenas 17bd21ee14 feat(IDocTags): add fine-grained content serialization filtering (#476)
* feat(IDocTags): add fine-grained content serialization filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update token analysis example

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 13:49:29 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
Panos Vagenas 0c9c70db76 feat(IDocTags): add XML escape modes (#471)
* feat: add XML escape modes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove workaround for IDE debugging

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix inline handling, descope entities mode for now

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix escaping/CDATA interference with inline groups, formatting, code languages; add underline tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup, test expansion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-12 12:46:26 +01:00
Peter W. J. Staar dda9c885a3 feat: idoctags serialization and deserialization matching the iso proposal (#457)
* Working on ISO standard for IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the pre-commit, still tons of work to do

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated idoctag token class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* need to start fixing the bugs and add proper tests for idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the testing for dcotags and idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the location tokens in idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the DocumentToken

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the idt quot issue

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Added IDocTagsSerializationMode

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the deserializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the captions for floating items

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* still need to fix some seserialization tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* finally all is working and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* made the OTSL serialization and deserialization self-contained in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed some unclean code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* did some fixes and clean up

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests, expanded to deserialize to deal with nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the deserialization for formatting

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the formatting and the complex inline groups in nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added get_category

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* make IS_SELFCLOSING a set

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the regex

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the if ... return into if ... elif ... return

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the facets parsing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the footnotes for Tables and Pictures in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with latest doctags token table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed private name

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-17 14:38:10 +01:00
Peter W. J. Staar 9a42a3c860 feat: updating the idoctags serializer (#450)
* feat: updating the idoctags serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* feat: adding IDocTagsToken class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the list serializer in the doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added missing document serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added tests for content suppression

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests for serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* deal with differences between xml serialization in mac and linux

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (3)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (4)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixing the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed all the tests, except the no-content with lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the no-content in lists with doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-10 11:39:40 +01:00
Panos Vagenas aa5c668050 feat(experimental): extend IDocTags tokens (#439)
feat: extend IDocTags tokens [WIP]

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-27 11:44:02 +01:00
Peter W. J. Staar 928aaed9ab chore: remove unnecesary copyright headers (#440)
* chore: remove unnecesary copyright headers

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted, still need to fix a test ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-27 11:17:23 +01:00
Panos Vagenas c9e5fb4a1c feat(experimental): add new DocTags serializer (#412)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-20 13:06:38 +01:00
Panos Vagenas d05fe08546 feat: promote serializers to stable API (#276)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-30 11:26:39 +02:00
Eugene c30ada64d6 chore: Types fixes (#267)
* docs: Recommend installing all groups for mypy to work properly

Signed-off-by: Eugene <fogaprod@gmail.com>

* chore: Fix some incomplete type defs

Signed-off-by: Eugene <fogaprod@gmail.com>

---------

Signed-off-by: Eugene <fogaprod@gmail.com>
2025-04-22 10:29:20 +02:00
d0a49da6de fix: HTML serialization for single image documents (#261)
* fix for HTML serialization for single image documents

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* minor refactor, add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-04-16 16:40:27 +02:00
Maxim LysakandMaksym Lysak caa8aeefae feat: Chart tabular data serialization for HTML serializer (#258)
* Add chart tabular data serialization to HTML serializer

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixing pre-commit issues

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixed table serialization for tabular chart data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* test for loading doctags with chart data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Improved loading doctags test with chart example, added tests for chart serialization into html and md

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-04-15 13:31:50 +02:00
Panos Vagenas 928e5c5046 fix: fix page breaking in case page starts with group (#253)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-14 18:12:47 +02:00
Panos Vagenas 15d2f2c1d5 fix: fix code handling in HTML serialization (#251)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-14 07:10:14 +02:00
Panos Vagenas 98c60bb65b perf: fix serialization performance (#249)
* perf: fix serialization performance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add missing leading HTML tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-11 15:37:52 +02:00
Panos Vagenas ab78e0bd9c fix: fix page filtering issue (#247)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-10 17:04:09 +02:00