Commit Graph
50 Commits
Author SHA1 Message Date
738abeee00 feat: make inline serializers concatenate runs faithfully (#693)
* fix(doclang): preserve leading text before a lone formatting tag

`_get_children_simple_text_block` guarded against a second *text* fragment
but the Element branch just overwrote `result`, so
`<text>2<superscript>nd</superscript></text>` returned "nd" and the leading
`2` was silently discarded. Guard both branches so the element is routed to
`_parse_inline_group`, which already handles mixed children.

* feat: inline runs carry their own whitespace; serializers concatenate faithfully

The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a
hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`,
`O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never
be removed while the separator lives in the serializer.

Serializers:
- join inline parts with `""`; DocLang keeps its delimiter between the element
  head and the body only, where `<content>` is the whitespace channel
- hoist edge whitespace around the *complete* decoration stack (formatting and
  hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim
  emits `**bold ** tail`, which CommonMark renders as literal asterisks;
  hoisting inside one formatter would let the hyperlink recapture it. DocLang
  opts out: it keeps whitespace inside the markup, in `<content>`.

DocLang deserializer:
- `_get_text` stripped every fragment and joined with `""`, collapsing
  `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer
  block boundary only, and never trims whitespace delivered by `<content>`.
- bare text nodes use the standard XML heuristic: a whitespace run containing a
  newline is pretty-print indentation, one without it is content. This is what
  hand-written and VLM-emitted DocLang needs, since it will not use `<content>`.
- element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`)
  no longer bleed into body text, where they were also double-modelled in
  `item.meta`.

Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator
re-injects the legacy separator for documents stamped below it, preserving their
previous *visible* output including its extra-space bugs (`H 2 O` stays
`H 2 O`). It must run before `check_version_is_compatible`, which rewrites
`version` and destroys the evidence. Versionless input is left alone. The
validator works on a copy, so the caller's dict is untouched and re-validating
the same raw value cannot compound separators.

A separator can never land inside a delimiter. Code and formula runs are wrapped
in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when
both neighbours are delimited it becomes a plain run of its own, because
`` `a``b` `` and `$a$$b$` are both corrupt.

Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join
-- anyone with a persisted vector index gets silent retrieval drift, so that
needs its own line in the release notes.

DocTags code is untouched. Its golden gains one empty `<text></text>` for the
legacy fixture because that document now carries an extra whitespace run, which
DocTags renders empty by its own pre-existing strip-per-run behaviour.

Ships as a minor, not a major: `feat!:` would make python-semantic-release
(7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package.

* DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 64404b3709d9e3ba7caf08957cf2ef19df29459f
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 66040cb7b96f77699c9d74051500d645e7e758ed

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(compat): register 1.11->1.10 projector for the whitespace-contract bump

Integrates the 1.11.0 schema bump with the downgrade-projector framework
(#703) landed upstream. 1.11 and 1.10 are structurally identical; the only
difference is that 1.11 inline runs own their boundary whitespace, so the
projector just re-stamps the version (a 1.10 client re-applies its own join,
yielding the documented harmless double space).

- add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10
- add the auto-generated 1.11 JSON Schema snapshot
- re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff)
- update the stale dclq CLI expectation (old join produced 'public  class';
  faithful join now yields 'public class')

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(dclq): use typer's vendored click; require typer>=0.25

typer >=0.25 vendors its own copy of click (typer._click) and raises those
exceptions instead of the external click package's. The dclq CLI referenced
the external click, so on typer >=0.25 bad input escaped the exit-code
handlers and the context-flag callback (-C/-A/-B/--context-scope) silently
no-op'd because the isinstance(param, click.Option) check no longer matched.

Import the exception and Parameter types from typer._click, duck-type the
option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused
external click dependency.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(doclang): keep inherited formatting on <content> runs in inline groups

<content> is the explicit whitespace-preserving text wrapper, so inside an
inline group it is a run, not a nested item: it must inherit the formatting
accumulated while descending through <bold>/<italic>/... . Dispatching it
through _dispatch_element dropped that formatting, which surfaced after the
rebase as <bold><content>x </content></bold> coming back unformatted and
broke the doclang round-trip for whitespace_inside_bold / _link.

Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

* chore(compat): point schema 1.11.0 at the next minor release

The whitespace-contract bump was first written against v2.86.0; main is now
at v2.104.0, so record v2.105.0 as the placeholder release that ships schema
1.11.0. Adjust to the actual version at release time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* test: rebaseline goldens for the inline whitespace contract

Regenerated with DOCLING_GEN_TEST_DATA=1. The only change is the schema
version stamp 1.10.0 -> 1.11.0 in goldens added or moved on main
(including the six test/ -> tests/ renames and misplaced_list_items.out.yaml).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: join inline runs faithfully in the plain-text reader

_collect_subtree_text (TextItem._get_plain_text, FloatingItem.caption_text,
nested rich table cells) still joined the runs of an InlineGroup with " ",
so mixed captions read "Table 1 : results". It now concatenates the runs of
an inline group; every other join, including a host text item's own text
and its children, keeps its space. Children of a text item are block
structure (producers put a section's paragraphs under its heading), so the
host text stays apart from its inline group, as on main; add_inline_group
documents this.

Tests that asserted the invented run spacing now expect the contract
output, fixtures that built runs without boundary whitespace now carry it,
and a new test pins the host-text boundary for headings and captions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: note that load_from_doctags cannot restore inline run whitespace

DocTags stores inline runs trimmed, so the boundary whitespace the inline
whitespace contract expects is not in the source. Document the limitation
instead of guessing separators back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(md): keep the host separator before a list item's inline group

The Markdown list serializer glued an inline group onto the preceding
list item with no separator, so a list item "Intro" whose child group
starts with "more" exported "- Intromore". Captions, headings and the
plain-text reader keep a space between host text and its inline group;
the list serializer now does the same when the group is the child of a
text item with its own text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): keep migrated separators between code or formula runs in page filters

A legacy document with two adjacent code or formula runs gets a standalone
" " run between them on load. It had no provenance, so filter(page_nrs=...)
and export_to_markdown(page_no=...) dropped it and rendered `$a$$b$`.

The separator now takes the left run's location when both runs are on the
same page, so it survives a page filter exactly when both runs do. Runs on
different pages keep the unlocated separator, which matches the old join.
Inline group locations in DocLang and DocTags are a union of the runs, so
they do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): count empty inline hosts and nested inline groups as legacy runs

The legacy separator migration counted only direct text children with
non-empty text as runs. An empty-text item whose content is a child inline
group (a formula or footnote host) and an inline group nested directly in
another were skipped, so `left`, host(`x`), `right` loaded as
`left $x$right` instead of the old `left $x$ right`. The DocLang
deserializer writes both shapes for nested mixed-content elements.

A child now counts as a run when it or its subtree has text. Only a plain
run with its own text takes the separator; an empty host must keep its
empty text to render its group, so it is treated like a code or formula run
and the separator stays outside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs(doclang): note that pre-contract archives load without legacy separators

DocLang has no schema version, so a .dclx written before schema 1.11.0
cannot be told apart from a current one and is not migrated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore(dclq): move the vendored-click fix to its own PR

This reverts 6a8b8374; the change is unrelated to the whitespace contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): don't run the legacy inline migration in project_to

project_to validated the projected dict under its target stamp. For 1.10.0 that ran the load-time separator migration on runs that already carry their whitespace, so 'Normal ', 'bold', ' tail' came back as 'Normal  ', 'bold ', ' tail'. Validate under the source stamp and restore the target stamp afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): strip inline runs when projecting 1.11 to 1.10

A 1.10 client joins inline runs with " ", so runs that keep their boundary whitespace rendered with double spaces and with the space inside the markup (**bold **, which CommonMark does not render as bold). The projector now strips each inline-group child's text to the shape old producers emitted. Whitespace-only runs become "", which old serializers skip, so no refs move. orig and text outside inline groups are left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): reuse the stripped separator run when migrating

Between two code or formula runs the migration inserts a plain " " run. The 1.11 -> 1.10 projector strips it to "", and the next migration skipped the empty run and inserted another, so every projection round trip added a run. Refill an empty plain run between the two neighbours instead. Old serializers skipped empty runs, so legacy output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 7600953bc0
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: f0e26b6722

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(doclang): read and write whitespace per DocLang 0.9

- Deserializer applies spec rules to the DOM before parsing: a
  whitespace sequence reads as one space, none at block edges,
  <content> is literal.
- Serializer keeps each text block body on one line; text with
  non-single-space whitespace goes in <content>, unwrapped where
  the rules read it the same bare.
- Emit version 0.9; validate against doclang 0.9.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix(doclang): keep inline code and formula in virtual text

In a list item or cell with text of its own, <code> and <formula> are
inline, so whitespace around them reads as a space under DocLang 0.9.

- Reader applies the whitespace rules across them instead of treating
  them as blocks.
- Pretty-printer writes such a virtual text on one line, so it no
  longer adds spaces next to inline code or formulas.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(compat): point schema 1.11.0 at v2.106.0

v2.105.0 was released without the whitespace contract, so record the
next minor release as the one that ships schema 1.11.0.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* docs(compat): name schema 1.11 as current in the projectors header

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix(doclang): harden the DocLang 0.9 whitespace reader and writer

- read list items with leading inline code or formula as virtual text
- keep whitespace-only content between inline code runs
- keep blank lines in content when pretty printing
- keep inline handwriting and rtl text as runs
- trim only XML whitespace when reading text
- keep content literal in headings, captions and list items
- read table cells with inline elements as virtual text
- keep text beside a whitespace-only formatted run
- read spaces around an empty content as one sequence
- read a checkbox as part of virtual text
- keep a whitespace-only inline code run

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(doclang): preserve checkbox spaces and formatted whitespace runs

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-11-03 10:38:54 +01:00
Cesar Berrospi Ramis f756342105 feat(compat): DoclingDocument downgrade projector system (#703)
* feat(compat): add DoclingDocument version-downgrade projector system

Adds docling_core/compat.py with a register_projector decorator,
project_to() API, and five retroactive projectors (1.10→1.9→1.8→1.7→
1.6→1.5). Adds scripts/check_compat_projectors.py as a pre-commit hook
that detects unannounced structural breaking changes and enforces that
every schema minor bump ships a projector, a JSON Schema snapshot, and a
test class. 36 new tests in test/test_compat.py; CONTRIBUTING.md
documents the change-classification table and 7-step checklist.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: upgrade uv.lock dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-08-18 10:29:48 +02:00
maxandmaxtaran2010 500a6e7d12 docs: fix typos in docstrings and comments (#677)
Corrects several misspellings in docstrings and inline comments:
- explicity -> explicit (ProvenanceItem docstring + generated schema)
- horisontal -> horizontal
- ommitted -> omitted
- to serializes -> to serialize
- succesive -> successive
- questionaires -> questionnaires
- optionaly -> optionally

No functional changes; identifiers and enum values are untouched.

Signed-off-by: maxtaran2010 <ocotifuzo727@gmail.com>
Co-authored-by: maxtaran2010 <ocotifuzo727@gmail.com>
2026-07-09 08:49:20 +02:00
Peter W. J. Staar 8951e6f202 chore: removing all legacy / deepsearch code (#644)
* What was removed

  The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.

  Deleted packages/modules
  - docling_core/types/legacy_doc/ (whole folder)
  - docling_core/types/rec/ (whole folder)
  - docling_core/utils/legacy.py (legacy conversion functions)
  - docling_core/utils/validate.py (RAW/ANN/OCR CLI)

  Edited production code
  - types/__init__.py — dropped the Record export
  - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
  validate_unique_list/validate_datetime
  - utils/generate_docs.py — dropped Record from MODELS

  Removed resources/artifacts
  - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json

  Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
  replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.

  Verification

  - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
  - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
  - Full project suite: 496 passed.
  - generate_docs.py now emits only DoclingDocument.json and Generic.json.

  The full plan is saved at _plans/remove-legacy-doc-and-rec.md.

  Two notes worth flagging:
  - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
  legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
  bump downstream.
  - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
  approval flow.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the checks.yaml

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the minimal_document_schema_flat.json

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/resources

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the files and updated the imports according to Cesar's suggestion

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the ruff linter

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-18 17:00:52 +02:00
129507ebd2 feat(meta): add keywords and topics metadata fields (#617)
* feat: add keywords and topics meta fields

Introduce two simple, additive meta fields:
- KeywordsMetaField — unique list of keyword strings
- TopicsMetaField — unique list of topic / subject strings

Both attach to BaseMeta (node-level) and to the new DocumentMeta
(document-level), which is wired into DoclingDocument.meta. HTML,
Markdown, and Doclang serializers render the new fields as comma-
separated values. Schema bump is intentionally avoided since the
fields are optional and backward-compatible.

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* test: cover keywords and topics meta fields

Add round-trip, dedup, has_content, HTML escape, and Markdown
rendering tests for KeywordsMetaField, TopicsMetaField, and
DocumentMeta. Document-level meta absent-by-default is also covered.

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* style: satisfy ruff UP038 and import order

Convert tuple-form isinstance checks to PEP 604 unions and
reorder a misplaced import flagged by ruff.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* fix: align with HTML attribute format from #613

Post-rebase onto main, the HTML metadata serializer now emits
`data-meta-name="<field>"` instead of valueless `data-meta-<field>`
and applies a single trailing html.escape pass. Drop the per-type
escape from the Keywords/Topics branch (was producing double
escaping after rebase) and update the two test assertions to the
new attribute shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* revert: remove DocLang keywords/topics serialization

Per discussion, DocLang is a standard whose extension is out of
scope for this PR. Drop the KeywordsMetaField/TopicsMetaField
branches from DoclangMetaSerializer and their imports; HTML and
Markdown serialization remain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* refactor: address review feedback on keywords/topics meta fields

- Replace MetaFieldName inline comments with Attributes docstring (TableFormerMode pattern)
- Swap list[str]+manual dedup → UniqueList[str] in KeywordsMetaField and TopicsMetaField
- Add Field(description, examples) to all BaseMeta fields distinguishing entities/keywords/topics
- Remove DocumentMeta class and DoclingDocument.meta field; document-level meta goes via body
- Consolidate keywords/topics tests into test_semantic_base_meta_fields_roundtrip_and_html_rendering
- Regenerate docs/DoclingDocument.json

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(meta): reuse validator between keywords and topics

Revert the deduplication as part of a validator.
Share the validator between keywords and topics meta classes.
Adopt the standard Google style for enum classes.
Adapt the regression tests to the changes above.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Ana Daniele <ana.daniele@ibm.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-05 13:06:37 +02:00
Peter W. J. Staar c1c9ea80a5 feat: support vertical orientation in TableData row/column bounding boxes (#622)
* fix: vertical-rows-and-columns

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added Orientation to TableData to deal with vertical tables

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* moved to ROT_0, etc

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the convention of the rotation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-29 17:16:08 +02:00
Peter W. J. Staar 9187158ef8 feat: extending BaseMeta with language and entities for docling-agent (#611)
* feat: extending BaseMeta with language, entities, etc for docling-agent

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ignore serialization of empty Meta in HTML

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* addressing the good comments from Cesar

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-19 15:43:35 +02:00
Peter W. J. Staar 46a9b5a329 feat: add latex and Tikz as codelabels (#579)
* feat: add latex and Tikz as codelabels

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-04-09 06:46:18 +02:00
Panos Vagenas 0bd5d8e649 feat: add code representation meta field (#573)
* feat: add code representation meta field

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add code language

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore custom field API changes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 17:43:39 +02:00
fb3b603bc4 feat: add handwriting support (#561)
* feat: Add HANDWRITTEN_TEXT label support

Add full integration for the HANDWRITTEN_TEXT document item label.

Changes:
- tokens.py: Add HANDWRITTEN_TEXT to DocumentToken enum and mapping
- document.py: Add HANDWRITTEN_TEXT to DEFAULT_EXPORT_LABELS

Also adds test/test_handwritten_text_label.py for integration tests.

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: acbdfa3902e74da44a5844ad0cecab8657da4904

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

* add handwriting support to Doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* DCO Remediation Commit for Vittorio Pippi <vpi@wila.zurich.ibm.com>

I, Vittorio Pippi <vpi@wila.zurich.ibm.com>, hereby add my Signed-off-by to this commit: 000ccc55c5

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>

---------

Signed-off-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Vittorio Pippi <vpi@wila.zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-26 08:43:29 +01:00
Panos Vagenas b93d5a3920 feat: introduce field data model incl. Doclang serialization (#519)
* feat: introduce new kv data model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add form-table test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update invoice test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include pre-migration YAML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add nesting test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to field naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve prov migration & location serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable proper nested serialization without inline

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add field marker and test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize marker naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* updated API

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align with TextItem conventions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make FieldItem a DocItem

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend KV migration scope, extend tree manipulation operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend kv migration tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test with KV nesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve key splitting and location assignment

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unnecessary files

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_content parameter, expose export_to_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts, add save_as_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:37:42 +01:00
Cesar Berrospi Ramis c8f3c01a61 feat: model and serializer for audio tracks (#426)
* refactor: move WebVTT data model from docling

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): deal with HTML entities in cue text spans

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): support more WebVTT models

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(DoclingDocument): create a new provenance model for media file types

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make WebVTTTimestamp public

Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set languages to a list of strings in ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): add test for ProvenanceTrack

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): make all WebVTT classes public for reuse

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): preserve newlines as WebVTTLineTerminator

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): set ProvenanceTrack time fields as float

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): ensure start time offsets are in sequence

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): improve regex to remove note,region,style blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): parse the WebVTT file title

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(webvtt): rebase to latest changes in idoctags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* feat(webvtt): add WebVTT serializer

Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(webvtt): add 'text/vtt' as extra mimetype

Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests(webvtt): fix test with STYLE and NOTE blocks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style(webvtt): apply X | Y annotation instead of Optional, Union

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): simplify TrackProvenance model with tags

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(webvtt): align class and field names to new 'source' type

Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(DoclingDocument): drop the validation on field assignment

Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

refactor(webvtt): drop cue span classes, 'lang' and 'c' tags

Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-27 14:45:03 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
SivaandPanos Vagenas 1d2e0c7ebe feat(DocItem): Add comments field for linking annotations to document… (#465)
* feat(DocItem): Add comments field for linking annotations to document items

Implements support for linking comments (from Word/PPT documents) to their
annotated content using the established FloatingItem/RefItem pattern.

Changes:
- Add `comments: List[RefItem]` field to DocItem class
- Update `_update_breadth_first_with_lookup()` to handle comment references on deletion
- Bump CURRENT_VERSION to 1.9.0
- Fix version comparison bug (string vs integer for minor version)
- Add 4 new tests for comments functionality
- Update test data files for new schema

Closes: docling-project/docling#464
Related: docling-project/docling#2834

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>

* improve comment Pydantic serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_comment, update tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce fine-granular references with span ranges

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify last test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-08 05:00:57 +01:00
Peter W. J. Staar 09ef91c272 fix: add JSON to CodeLanguageLabel (#413)
* fix: add JSON to CodeLanguageLabel

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-04 14:06:47 +01:00
Panos Vagenas 2ee3cacdd6 feat: add metadata model hierarchy (#408)
* feat: add metadata model hierarchy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add deprecation, add first migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend annotations migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update with feedback

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose main prediction

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* ideas on enforcing separation between standard and custom fields

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add custom field setter method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update Markdown serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert description, add include_non_meta, showcase custom serializer for summaries

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify customization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix reference exclusion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* eliminate serialization dupliation between meta & (legacy) annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove old file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix item used in get_parts for meta ser

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* serialize GroupItem meta prior to content, DocItem meta after content

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore ser order for all nodeitems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* move meta serialization into DocSerializer.serialize() to maintain seamless chunking integration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add allow- & block-lists for meta names, add std field name enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML serializer, document meta field names, rename SMILES field

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make TabularChartMetaField.title optional, expose new classes through __init__.py, add MetaUtils

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add DocTags serialization, revert smiles to smi to prevent confusion with plural

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-30 11:15:14 +01:00
Maxim LysakandMaksym Lysak b13267f18b feat: Introduction of fillable TableCell (#384)
* New properties for TableCell, corner_header and fillable

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Updated docs

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* bumped docling document version to 1.7.0

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-09-09 15:48:42 +02:00
Panos Vagenas 1d04154378 feat: add rich table cells (#368)
* feat: add rich table cells

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* propagate cell text resolution, cover row deletions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add doctags, fix referential integrity, expand tests, reenable mypy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify / remove serialize_cell

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update rich table cell refs in doc indexing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update notebook

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose new classes in `docling_core.types.doc`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-27 10:07:47 +02:00
eb2538eb3a feat: added different content-layers (#345)
* feat: added different content-layers

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* renamed concealed to invisible

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Update docling_core/types/doc/document.py

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>

* Update docs/DoclingDocument.json

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
Co-authored-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-07-02 17:28:53 +02:00
Panos Vagenas 14a4fdee87 feat: remodel lists, add MD & HTML ser. params, enable unset marker (#339)
* feat: remodel lists

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* prepare test document separate markers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add MD/HTML serializer options, expand test data

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* create list groups where not in place

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add auto-increment logic

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore UnorderedList as deprecated alias (for backwards compatibility)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable unset marker case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make ordered markers able to be "unset" (empty) too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename default marker mode to auto

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-27 14:52:25 +02:00
Christoph AuerandPanos Vagenas aa430d3776 feat: New labels for CVAT annotation (#314)
* New labels and utility for CVAT annotation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add EMPTY_VALUE to TextItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* align usage of content layer param (#326)

* align usage of content layer param

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* parametrize content layers in visualizers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-06-11 19:25:13 +02:00
Panos Vagenas ae961299a5 feat: add subscript & superscript formatting (#319)
* feat: add subscript & superscript formatting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-06 09:01:32 +02:00
Panos Vagenas d8a5256b2c feat: add table annotations (#304)
* feat: add table annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* refactor annotation types

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expand to HTML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce annotation serializer

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* Update dummy_doc.yaml

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-06-05 15:38:46 +02:00
Cesar Berrospi Ramis 4a174b5679 chore: fix deprecation warnings (#303)
* chore: fix deprecation warnings

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: disregard deprecated captions from hierarchical chunker in hybrid chunker

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: update poetry lock file

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-05-26 15:58:08 +02:00
Christoph Auer aa957cf4b6 fix: Updates for labels and methods to support document GT annotation (#293)
Update labels and get_color

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2025-05-14 14:35:17 +02:00
Peter W. J. Staar 2f0f12160b feat: adding the label picture_group (#283)
* feat: adding the label picture_group

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* renamed group to area

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-05-06 09:43:35 +02:00
Maxim LysakandMaksym Lysak e9259a5f87 feat: Support of DocTags charts (serialization and deserialization) (#229)
* Doctags charts deserialization into Docling Document, added PictureTabularChartData

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Correction for chart label mapping

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* simplified label mapping

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Improved docummentation comment

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Added stacked_bar_chart to labels and tokens, as well as adjusted doctags_load

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* doctags serializer for charts

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* - Fixed DocTags table deserialization
- Updated tests

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* cleaning

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Allowing DocItemLabel.CHART as a label for PictureItem

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-04-07 12:27:48 +02:00
Panos Vagenas a7cdc87411 feat: add serializers, text formatting, update Markdown export (#182)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-13 17:54:32 +01:00
Panos Vagenas 2abaf9b537 feat: add inline groups, revamp Markdown export incl. list groups (#156)
* feat: add inline groups, update markdown export

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* rename parameter for clarity

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* drop redundant `traverse_inline`, use only `visited`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update API usage after rebase

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor refactor

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend nesting logic to list groups

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* (MD export) add inline formulas, placeholders for `KeyValueItem`, `FormItem`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include 3rd-level list item in test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update test file with more recent conversion result

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update list construction pattern so that sublists are added to actual ListItems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-02-25 09:48:10 +01:00
d622800750 feat: Introduce Key-Value and Forms items (#158)
* Draft KeyValueItem content

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* added add_key_value_item method

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add KeyValueLink

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update an add_key_value_item argument

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* remove KeyOrValueCellType and KeyValueLinkType

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update tests for KeyValueItem

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add union method to create bbox that covers all the given bboxes

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* added the , and rewrote it to make it more general with a GraphItem, as well as having cell- and link-labels

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the code due to MyPy and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the reference docs with form-key

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the circle

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the figure, testing now ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added square in image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed square in image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* tests should go through

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* update poetry.lock versions

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* fix import issue

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add validator for links in GraphData

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added field_validator

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rename add_key_values and add_form

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* rename union to enclosingbbox

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
2025-02-19 09:50:45 +01:00
Christoph Auer 7267c3f571 fix: Fix inheritance of CodeItem for backward compatibility (#162)
* fix: Fix inheritance of CodeItem for backward compatibility

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update docs

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2025-02-14 17:48:34 +01:00
Peter W. J. StaarandMichele Dolfi 916323fb55 feat: Redefine CodeItem as floating object with captions (#160)
* updated the CodeItem with captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* use FloatItem and add captions to add_code interface

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2025-02-14 11:34:40 +01:00
Christoph AuerandPanos Vagenas 786f0c6833 feat: Add ContentLayer attribute to designate items to body or furniture (#148)
* feat: Add ContentLayer attribute to designate items to body or furniture

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce safer data gen mechanism, update chunking test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Do not make test rely on order in yaml

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore: format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: legacy_to_docling_doc must use content_layer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add content_layer in iterate_items

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Bump format version, add model_validator for old page_header,page_footer in body

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: Change to before model_validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Address review comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-02-10 10:58:41 +01:00
c940aa5ca9 feat: Add CodeItem as pydantic type, update export methods and APIs (#129)
* added code item

* added code item

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* add constraints to allow numpy > 2.1.0 on python3.13 and others

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Add CodeItem to ContentItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* added CodeItem in ContentItem tagged union.

* added enum for programming languages

* removed double CodeItem in ContentItem Union

* fixed type of code_language in CodeItem class

* fixed sorting of programming languages, not sorted anymore by value of string but variable name

---------

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2025-01-17 16:59:33 +01:00
Peter W. J. Staar 5101dd8845 feat: added the new label comment_section in the groups (#114)
* added the new label comment_section in the groups

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-12-17 14:23:18 +01:00
Christoph Auer aeaf89de10 feat: Add group labels for form and key-value areas (#110)
Add group labels for form and key-value

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2024-12-16 13:40:01 +01:00
Panos Vagenas 047a1960af fix: improve doc item typing (#105)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-13 12:51:53 +01:00
Cesar Berrospi Ramis 266e7fc603 build: set pydantic version to 2.10.3 (#93)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-12-04 14:26:22 +01:00
ef49fd3f34 feat: adding HTML export to DoclingDocument, adding export of images in png with links to Markdown & HTML (#69)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2024-11-27 05:27:23 +01:00
MatteoandMatteo-Omenetti 36b7bea53a feat: added pydantic models to store charts data (pie, bar, stacked bar, line, scatter) (#52)
* added pydantic models to store charts data (pie, bar, line)

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed chart models hierarchy structure, added StackedBarChart class

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added scatter chart and addresed Peter's comments

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fixed names of classes

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

---------

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Co-authored-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
2024-10-29 15:49:09 +01:00
Panos Vagenas e12d6a70c3 fix: fix legacy doc ref (#48)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-18 15:21:01 +02:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Cesar Berrospi Ramis a4ddd142ee fix: change order of JSON Schema to search mapper transformations (#32)
* fix: change order of JSON Schema to search mapper transformations

In the JSON Schema to search mapper, the suppression flags need to be addressed first,
otherwise the jsonref.replace_refs function may remove them.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* build: update dependencies

Updating pydantic from 2.8.2 to 2.9.2 triggers a change in JSON Schema from models:
the  lists with only 1 element get flatten.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: improve verbose in JSON Schema to search mapper test

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-09-26 13:49:18 +02:00
Peter W. J. Staar 891530f595 feat: add export to doctags for document components (#25)
* feat: refactoring doc-tokens in new file and add new export function to table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted and fixed bugs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on table exporting in document tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated code and tests, need to decide what to commit as test-cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the test-files

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (boundingbox to figure and table)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (making BaseCell the base-class of BaseText)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (re-use of function in BaseCell)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed issues from review (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-20 14:16:33 +02:00
Peter W. J. Staar acdf816081 feat: added the XML export (#16)
* added the XML export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted all

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the DocumentTokens class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the strict-text in the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added page-tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the location/page tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-09 17:41:58 +02:00
Michele Dolfi f450c8cbfd fix: use same base type for all components (#10)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-08-28 11:21:05 +02:00
Panos Vagenas ec3cff97e5 fix: make page number strictly positive (#8)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-31 13:36:49 +02:00
Cesar Berrospi Ramis faf472c168 fix: set type to optional (#7)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-23 14:07:42 +02:00
Cesar Berrospi Ramis 656f56380f fix: fix definition issues in record type (#2)
* fix(test): set static typing compatible to python 3.10

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* style: enforce some pre-commit hooks on tests

Enforce pre-commit hooks black, isort, autoflake, and mypy on test modules.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* fix(rec): fix definition issues in attribute, predicate, subject

Remove duplicate generic types across base and predicate modules.
Create an identifier class for subject names.
Remove unnecessary type variables in attribute model.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* docs: refer to Docling data objects

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-17 17:24:31 +02:00
Cesar Berrospi Ramis 9134ffee93 Initial commit
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-15 12:55:20 +02:00