Commit Graph
136 Commits
Author SHA1 Message Date
Christoph Auer 3a3f47ec1b fix: preserve superscripts and subscripts in Markdown export
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-09-14 10:55:53 +02:00
Anusha AcharyaandCesar Berrospi Ramis 9b23d315af fix(serializer): derive markdown table header rows from column_header flags (#723)
* fix(serializer): derive markdown table header rows from column_header flags

MarkdownTableSerializer hardcoded grid row 0 as the only header row, while
TableItem._export_to_dataframe_with_options, the HTML serializer and the OTSL
export all derive the header rows from TableCell.column_header. A table with a
stacked header therefore serialized to markdown with its lower header rows
demoted to data rows, and HybridChunker repeated only the first of them at the
top of each chunk of an oversized table.

Take the header row count from the flags and flatten those rows into the single
header row GFM allows, dropping the repeated text a cell spanning several header
rows produces. When no cell in the table carries column_header the previous
behavior is kept, since backends that never set the flag cannot be told apart
from genuinely headerless tables.

Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>

* test(serializer): cover tables whose header flags start below row 0

The empty-header-block branch of _flatten_header_rows was the one line of the
patch left uncovered. Pin the behavior it produces: with no leading run of
flagged rows there is no header to promote, so the header is empty and every
row stays in the body rather than one being consumed.

Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>

* fix(serializer): join stacked header cells with a documented separator

Review feedback: a blank space loses the boundary between header levels, so
"native backend" over "TTS" is indistinguishable from a single cell reading
"native backend TTS". Join with " - " instead, kept as a module constant, and
document the header-resolution contract on MarkdownTableSerializer.

Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>

* fix(serializer): only count header rows a column header starts on

TableData.grid repeats a row-spanning cell into every row it covers, so a
header cell with row_span > 1 leaves a column_header-flagged cell in the data
rows beneath it. Counting every row that holds a flagged cell therefore pulled
those data rows into the header block, where they were flattened into the
column names and dropped from the body entirely.

Require that a header cell starts on the row, matching the same fix applied to
the DataFrame export in #756.

Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>

* perf(serializer): avoid iterating grid twice; style and tests improvements

In '_count_header_rows' of the Markdown serializer, avoid iterating
the entire grid twice in all scenarios.
Strengthen docstrings in markdown serializer.
Add test for single header row with flags.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-09-10 14:03:23 +02:00
Cesar Berrospi Ramis 0c46294fac style: modernize type hints with PEP 604 + dependency upgrades (#737)
* style: modernize type hints with PEP 604

Enforce Ruff UP007 and UP045 to align with PEP 604 (union types as X | Y).

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: upgrade 'uv.lock' dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-09-04 06:31:04 +02:00
Cesar Berrospi Ramis 7a6c5a4114 fix(latex): prevent inline-group content from being serialized twice (#743)
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-09-04 06:30:55 +02:00
ccdffa7844 fix(markdown): escape referenced figures' filepaths and ensure cross-platform path handling (Windows / POSIX) (#698)
* fix regression from removal of #122 by #182

Signed-off-by: Laurier Lavoie-Giasson <laurier@laurier.dev>

* change classmethod to staticmethod

Signed-off-by: Laurier Lavoie-Giasson <laurier@laurier.dev>

* fix(markdown): apply PR comments

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Laurier Lavoie-Giasson <laurier@laurier.dev>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Laurier Lavoie-Giasson <laurier@laurier.dev>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-09-02 10:16:02 +02:00
bdfca5b89e fix: convert headings to plain text in table cells (#540)
* fix: convert headings to plain text in table cells (issue #2722)

Signed-off-by: 温可 <wenke@openclaw.ai>

* fix(serializer): accept 'in_table_cell' parameter in PlainTextTextSerializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): add test for headings in rich table cells

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs(serializer): add/improve docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): reduce verbosity with inline comments

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: 温可 <wenke@openclaw.ai>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: 温可 <wenke@openclaw.ai>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-09-02 10:15:51 +02:00
Peter W. J. Staar a222564ef8 fix: updated the doclang serializer for groups (#736)
* fix: updated the doclang serializer for groups

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-09-01 12:42:17 +02:00
Panos Vagenas dedc35da4c feat(markdown): make image class suppressible (#726)
* feat(markdown): make image class suppressible

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* propagate to `save_as_markdown()`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-08-20 10:42:26 +02:00
Panos Vagenas b957d966d8 fix(markdown): suppress FieldRegionItem and FieldItem serialization (#724)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-08-19 15:54:02 +02:00
Cesar Berrospi Ramis 289c2e50fb fix(markdown): serialize newlines as GFM hard line breaks (#721)
* fix(markdown): serialize newlines as GFM hard line breaks

Single \n in body text and list items is now serialized as '  \n'
(two trailing spaces), which GFM renders as a hard line break.
Double \n\n is preserved as a paragraph break.
Newlines in headings are collapsed to a space, since headings cannot
span multiple lines.
PlainTextDocSerializer is unaffected via overridable instance methods.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(deps): upgrade uv.lock dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-08-19 11:03:45 +02:00
Jordan WhiteandCesar Berrospi Ramis ec0069c6ec perf: contextualize function() skip excluded fields (#581)
* contextualize function() skip excluded fields

This change proposes that we use model_dump to skip/avoid serializing
fields that are in excluded_embed. This should reduce unnecesary
computation and is also more precisely what I believe we are intending
to do here in that we do not need to unnecessarily filter the results
below in the for-loop.

Signed-off-by: Jordan White <jordan.page.white@gmail.com>

* style(chunker): fix style issue C416

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): resolve excluded_embed to field names

model_dump(exclude=...) requires Python attribute names, not
aliases. excluded_embed stores alias keys, which happen to
match today but would silently fail to exclude fields in
subclasses where they differ, leaking excluded fields into the
embedding context.

Adds BaseMeta._excluded_embed_field_names() to resolve each
entry through the alias-to-field-name map. contextualize() now
calls this helper instead of using excluded_embed directly.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): fix pydantic deprecation

Deprecated in pydantic v2.11, removed in v3: model_fields can only be accessed
from class object, not the instance.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(chunker): add test to verify contextualize() excludes the right fields

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: update uv.lock dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Jordan White <jordan.page.white@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-08-19 10:00:09 +02:00
Matvei SmirnovandCesar Berrospi Ramis 965a6bb6a8 fix(chunker): forward content layers from serializer params to doc traversal (#593)
fix(chunker): included content layers

Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-08-17 10:53:11 +02:00
Omar Faig Orujlu 4252788e6a fix(serializer): use block-level spacing for page breaks inside lists (#717)
A page break falling inside a ListGroup was emitted as an ordinary list
part and joined with the list separator "\n", while one falling between
top-level items was joined with "\n\n". The placeholder is substituted
into the already joined text, so the spacing was fixed by whichever
scope emitted the break and could not be renegotiated downstream.

Exporting a document globally and page by page therefore disagreed by
two bytes whenever a list straddled a page boundary, although the
content matched.

MarkdownListSerializer now separates a page break part from its
neighbours with "\n\n", so the marker forms a block in either scope. To
let it recognise such a part, DocSerializer.serialize() returns the
_PageBreakSerResult as such instead of rebuilding a plain result and
dropping the type.

Output is unchanged for callers leaving page_break_placeholder unset,
as no page break parts are produced at all in that case.

Signed-off-by: OmarFaig <omerfaiq7@gmail.com>
2026-08-14 16:07:01 +02:00
Christoph Auer cf2ac70632 feat: dclq tool (#696)
* feat: add initial dlgrep implementation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* refactor: migrate CLI to Typer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: expand README

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: improve README examples

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: streamline dlgrep output

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: make text context output grep-compatible

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Improve XPath and inline context output

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* dlgrep support extensions

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Use git sources for doclang and docling-core

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: emit native JSON records

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove search class filter

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Make section output addressable

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Improve README examples

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: build and publish dlgrep from this repository

dlgrep becomes a uv workspace member under packages/dlgrep, published as
its own wheel and released in lockstep with docling-core: same version,
exact `docling-core==` pin, both wheels built from the same commit. This
removes the git-branch dependency dlgrep needed to track unreleased
docling-core changes.

- workspace wiring; ruff, mypy and pytest config inherited from the root,
  dlgrep in the dev group so `uv sync` covers it
- release.sh writes both versions and the pin from one TARGET_VERSION
- build-packages.sh builds into per-package dist dirs; pypi.yml publishes
  docling-core first, then dlgrep
- check_lockstep.py guards against version drift, which uv cannot catch on
  its own: the workspace source replaces the pin with a local path, so a
  wrong pin resolves and locks cleanly
- checks.yml and pypi.yml install both wheels together, which makes a
  mismatched pin fail before anything is published
- declare click and lxml, previously only transitive
- read __version__ from package metadata instead of a hardcoded literal
- flag dlgrep as experimental in its help output, since the shared version
  number would otherwise imply a maturity it does not have

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* ci: simplify lockstep release tooling

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore(dlgrep): drop handoff notes, keep only the CLI plan

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* refactor(doclang): isolate source mapping recorder

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* build(dlgrep): move dependencies to core extra

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(dlq): rename CLI and add list command

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(dclq): rename tool and consolidate plan

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(dclq): bound archive document reads

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-07-31 13:38:43 +02:00
Christoph Auer 4afc5e6789 fix: preserve literal tag-shaped text in OTSL cells (#695)
Fix OTSL delimited content interpreted as XML elements

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-07-27 16:45:29 +02:00
Max Hsu b869c67cf8 fix(doclang): clamp deep section headers instead of raising (#688)
DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.

The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.

Fixes #685

Signed-off-by: Max Hsu <maxmilian@gmail.com>
2026-07-27 13:48:31 +02:00
Akhil Trivedi 26a50ab140 fix(doclang): sanitize XML-illegal text characters (#687)
Signed-off-by: AkhilTrivediX <118957648+AkhilTrivediX@users.noreply.github.com>
2026-07-27 13:48:15 +02:00
Peter W. J. Staar 129811fe49 fix: DocLang deserialization of XML-sensitive OTSL cell content (#689)
fix: doclang deserialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-07-24 16:23:43 +02:00
Rehan KhanandCesar Berrospi Ramis 9ceb26665f fix: lazy load transformers in HybridChunker (#468)
* fix: lazy load transformers in HybridChunker

Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): guard check at point of use

Guard import check at the point of use in hybrid chunker.
Follow the identical pattern for lazy load as in docling.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(chunker): add tests for lazy-loading with HybridChunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: normalize import calls from DoclingDocument

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-22 11:01:30 +02:00
Panos Vagenas 224eeeab60 fix(doclang): enforce image size budgets for archive assets and ImageRef (#684)
* fix(doclang): enforce image size budgets for archive assets and ImageRef

- Cap `.dclx` `pages/` and `assets/` rasters with `max_image_decoded_size` before Pillow opens them
- Apply the same limit to Path and `file://` ImageRef loads
- Add tests covering oversize archive images and path-based ImageRefs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump doclang to 0.7.3 and clarify ImageRef remote-fetch guidance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-07-15 10:47:13 +02:00
Panos Vagenas aee0674a28 fix(doclang): add deserialize size, depth, and element budgets (#683)
- Cap DocLang XML input bytes before `parseString`
- Bound nesting depth and element count after DOM construction
- Expose limits via settings / `deserialize_str` kwargs and cover with unit tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-07-14 15:49:17 +02:00
Cesar Berrospi Ramisandhardik-kansal 7775df1176 fix(chunker): preserve table structure across chunks in markdown (#678)
* fix(chunker): preserve markdown table structure when splitting across chunks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: extract shared fixtures and serializer providers in test_hybrid_chunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: replace inline comments with docstrings in segment and get_header_and_body_lines

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs(chunker): fix misleading docstrings in LineBasedTokenChunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: normalize import calls from DoclingDocument

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): correctly identify table preamble when caption starts with '|'

Co-Authored-By: hardik-kansal <hkhardik4321@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: hardik-kansal <hkhardik4321@gmail.com>
2026-07-14 10:50:03 +02:00
maxandmaxtaran2010 500a6e7d12 docs: fix typos in docstrings and comments (#677)
Corrects several misspellings in docstrings and inline comments:
- explicity -> explicit (ProvenanceItem docstring + generated schema)
- horisontal -> horizontal
- ommitted -> omitted
- to serializes -> to serialize
- succesive -> successive
- questionaires -> questionnaires
- optionaly -> optionally

No functional changes; identifiers and enum values are untouched.

Signed-off-by: maxtaran2010 <ocotifuzo727@gmail.com>
Co-authored-by: maxtaran2010 <ocotifuzo727@gmail.com>
2026-07-09 08:49:20 +02:00
Panos Vagenas 3704874312 feat(doclang): add support for .dclx archive, <description>, <summary> (#658)
* feat(doclang): add .dclx archive save and load support

- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): add support for `<description>` and `<summary>`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-25 18:06:10 +02:00
Peter W. J. Staar 8951e6f202 chore: removing all legacy / deepsearch code (#644)
* What was removed

  The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.

  Deleted packages/modules
  - docling_core/types/legacy_doc/ (whole folder)
  - docling_core/types/rec/ (whole folder)
  - docling_core/utils/legacy.py (legacy conversion functions)
  - docling_core/utils/validate.py (RAW/ANN/OCR CLI)

  Edited production code
  - types/__init__.py — dropped the Record export
  - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
  validate_unique_list/validate_datetime
  - utils/generate_docs.py — dropped Record from MODELS

  Removed resources/artifacts
  - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json

  Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
  replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.

  Verification

  - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
  - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
  - Full project suite: 496 passed.
  - generate_docs.py now emits only DoclingDocument.json and Generic.json.

  The full plan is saved at _plans/remove-legacy-doc-and-rec.md.

  Two notes worth flagging:
  - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
  legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
  bump downstream.
  - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
  approval flow.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the checks.yaml

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the minimal_document_schema_flat.json

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/resources

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the files and updated the imports according to Cesar's suggestion

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the ruff linter

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-18 17:00:52 +02:00
Panos Vagenas 9dddcf7137 fix: clamp provenance bboxes and DocLang locations to valid page bounds (#643)
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:44:20 +02:00
Panos Vagenas 34bd98eb3d feat(doclang): default include_version to True (#642)
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:20:20 +02:00
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
Panos Vagenas 0b7690509e feat(doclang): align with DocLang v0.6, add field region deser. (#636)
* feat(doclang): align with DocLang v0.6 picture model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): enhance picture metadata handling with custom fields

- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): deserialize field regions and add round-trip fixtures

- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-11 12:30:09 +02:00
Panos Vagenas 5d740c260a feat(doclang): promote serializer and deserializer (#633)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 15:23:21 +02:00
129507ebd2 feat(meta): add keywords and topics metadata fields (#617)
* feat: add keywords and topics meta fields

Introduce two simple, additive meta fields:
- KeywordsMetaField — unique list of keyword strings
- TopicsMetaField — unique list of topic / subject strings

Both attach to BaseMeta (node-level) and to the new DocumentMeta
(document-level), which is wired into DoclingDocument.meta. HTML,
Markdown, and Doclang serializers render the new fields as comma-
separated values. Schema bump is intentionally avoided since the
fields are optional and backward-compatible.

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* test: cover keywords and topics meta fields

Add round-trip, dedup, has_content, HTML escape, and Markdown
rendering tests for KeywordsMetaField, TopicsMetaField, and
DocumentMeta. Document-level meta absent-by-default is also covered.

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* style: satisfy ruff UP038 and import order

Convert tuple-form isinstance checks to PEP 604 unions and
reorder a misplaced import flagged by ruff.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* fix: align with HTML attribute format from #613

Post-rebase onto main, the HTML metadata serializer now emits
`data-meta-name="<field>"` instead of valueless `data-meta-<field>`
and applies a single trailing html.escape pass. Drop the per-type
escape from the Keywords/Topics branch (was producing double
escaping after rebase) and update the two test assertions to the
new attribute shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* revert: remove DocLang keywords/topics serialization

Per discussion, DocLang is a standard whose extension is out of
scope for this PR. Drop the KeywordsMetaField/TopicsMetaField
branches from DoclangMetaSerializer and their imports; HTML and
Markdown serialization remain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

* refactor: address review feedback on keywords/topics meta fields

- Replace MetaFieldName inline comments with Attributes docstring (TableFormerMode pattern)
- Swap list[str]+manual dedup → UniqueList[str] in KeywordsMetaField and TopicsMetaField
- Add Field(description, examples) to all BaseMeta fields distinguishing entities/keywords/topics
- Remove DocumentMeta class and DoclingDocument.meta field; document-level meta goes via body
- Consolidate keywords/topics tests into test_semantic_base_meta_fields_roundtrip_and_html_rendering
- Regenerate docs/DoclingDocument.json

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(meta): reuse validator between keywords and topics

Revert the deduplication as part of a validator.
Share the validator between keywords and topics meta classes.
Adopt the standard Google style for enum classes.
Adapt the regression tests to the changes above.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Ana Daniele <ana.daniele@ibm.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-05 13:06:37 +02:00
b20b5677c0 fix(html-serializer): emit XHTML-compatible metadata attributes (#613)
* fix(html-serializer): emit XHTML-compatible metadata attributes

HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.

Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.

Fixes #598.

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>

* fix(html-serializer): rebase changes to 'main' and add tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(html-serializer): remove redundant CSS rule

Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-05-29 10:01:30 +02:00
Manohar Mallipudi 507df717c1 fix(chunking): handle empty TripletTableSerializer output without consuming children (#596)
When a `TableItem` produced empty serialized text, `TripletTableSerializer`
was still populating the shared `visited` set via `RichTableCell._get_text()`.
Downstream items were then treated as already chunked and the chunker
emitted 0 chunks (e.g. on DOCX layout/header-only tables).

This change:
- Isolates `visited` while exporting the table dataframe and only merges
  the local copy back into the shared set when non-empty text is produced.
- Promotes the header row into a data row when the dataframe has 0 data
  rows but non-empty column names, so header-only tables still emit text.
- Handles single-row single-column tables by emitting the cell text
  directly instead of an empty triplet list.
- Adds `_flatten_table_text` as a fallback for tables whose triplet form
  collapses to empty text.
- Guards against appending empty text to serialization parts.

Adds regression tests covering header-only tables, single-row
single-column tables, end-to-end `HierarchicalChunker`/`HybridChunker`
runs on header-only / single-cell rich tables, and verifies that empty
tables do not consume their children's refs.

Closes docling-project/docling#3335

Signed-off-by: Manohar Mallipudi <manohar.m4475@yahoo.com>
2026-05-28 15:41:47 +02:00
Peter W. J. Staar 01f8d03cda fix: render RichTableCell descendants inside the cell (#621)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-26 16:46:10 +02:00
Peter W. J. Staar 9187158ef8 feat: extending BaseMeta with language and entities for docling-agent (#611)
* feat: extending BaseMeta with language, entities, etc for docling-agent

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ignore serialization of empty Meta in HTML

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* addressing the good comments from Cesar

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-19 15:43:35 +02:00
Peter W. J. Staar b6c3d6fb14 feat: updated the table visualization (#615)
updated the table visualization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-18 07:30:34 +02:00
Peter W. J. Staar 014948b0e8 feat: updated the HTML serialization (#609)
* updated the HTML serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-12 16:49:08 +02:00
Smeet AgrawalandClaude Sonnet 4.6 9dc882dc48 feat(serializer): add MsExcelMarkdownDocSerializer for sheet-name headings (#587)
* feat(serializer): add MsExcelMarkdownDocSerializer for sheet-name headings

Add `MsExcelMarkdownFallbackSerializer` and `MsExcelMarkdownDocSerializer`
to the serializer package so that `GroupLabel.SHEET` groups are rendered as
level-2 Markdown headings when exporting Excel-sourced documents.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* DCO Remediation Commit for Smeet23 <smeetagrawal2003@gmail.com>

I, Smeet23 <smeetagrawal2003@gmail.com>, hereby add my Signed-off-by to this commit: 2a3808e5dc

Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>

* style: apply ruff formatter to markdown_excel.py

Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>

---------

Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 07:01:13 +02:00
Cesar Berrospi Ramis 6cbdee9626 fix: prevent numeric precision loss in Markdown table serialization (#588)
* fix: preserve numeric precision in markdown serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: apply style conventions to test_serialization.py module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-04-15 17:16:57 +02:00
f2a61868d4 feat: DocChunk expansion (#549)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* chunk expansion + test

Signed-off-by: odelliab <odelliab@il.ibm.com>

* small fixes

Signed-off-by: odelliab <odelliab@il.ibm.com>

* small fixes

Signed-off-by: odelliab <odelliab@il.ibm.com>

* remove unnecessary code

Signed-off-by: odelliab <odelliab@il.ibm.com>

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <91875866+odelliab@users.noreply.github.com>, hereby add my Signed-off-by to this commit: 6aa0019fe0

Signed-off-by: odelliab <odelliab@il.ibm.com>

* change names

Signed-off-by: odelliab <odelliab@il.ibm.com>

* remove some tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* consolidate tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* fix failing test

Signed-off-by: odelliab <odelliab@il.ibm.com>

* test: regenerate ground truth for hybrid chunker test

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: extract chunk expansion into TreeChunkExpander and PageChunkExpander

- Create chunk_expander.py with TreeChunkExpander and PageChunkExpander classes
- Remove DocChunk expansion methods
- Improve docstrings
- Refactor tests: test_doc_chunk_expansion.py → test_chunk_expander.py

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-04-15 17:16:28 +02:00
Cesar Berrospi Ramis 0a3b2787e0 fix(markdown): remove assert statements to support Python optimization mode (#548)
fix(markdown): remove 'assert' statement in runtime code

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-17 11:25:39 +01:00
Cesar Berrospi Ramis af50f1cb07 feat: profile a document or collection (#511)
* feat: profile a document or collection

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(profiler): add deciles and histograms

Add deciles and histograms to the Docling collection statistics.
Add an example script to plot histograms.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(profiler): add option to plot log frequencies in histogram

Add the option to plot the histogram frequencies in logarithmic scale.
Extend README with documentation on the document profiler.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(profiler): cover missing lines in doc_profiler with tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 13:36:38 +01:00
b435090fdf feat: split html table to headers and body (#532)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* remove unnecessary lines

Signed-off-by: odelliab <odelliab@il.ibm.com>

* split html tables

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add bs4 dependency

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* typo

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add beautifulsoup tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor(serializer): drop dependency 'beautifulsoup4' in HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): refactor tests for HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 10:40:05 +01:00
Anish Raghavendra e00125c477 feat: handle wide table outliers with LineBasedTokenChunker (#536)
* feat: handle large prefixes by splitting into multiple chunks in LineBasedTokenChunker

Signed-off-by: z404 <anishr890@gmail.com>

* feat: add option to omit table headers on row overflow

Signed-off-by: z404 <anishr890@gmail.com>

* chore: add warning when prefix is omitted on overflow

Signed-off-by: z404 <anishr890@gmail.com>

* refactor: simplify prefix chunking logic by removing redundant method

Signed-off-by: z404 <anishr890@gmail.com>

* fix: ensure prefix appears as standalone chunk when all lines overflow

Signed-off-by: z404 <anishr890@gmail.com>

* docs: add usage examples for LineBasedTokenChunker with prefix handling

Signed-off-by: z404 <anishr890@gmail.com>

* docs: remove examples and describe parameters for LineBasedTokenChunker

Signed-off-by: z404 <anishr890@gmail.com>

* docs: simplify class docstring by removing parameter details

Signed-off-by: z404 <anishr890@gmail.com>

---------

Signed-off-by: z404 <anishr890@gmail.com>
2026-03-13 07:29:46 +01:00
Michele Dolfi 4eb0d20d04 feat: Loosen dependency version constraints (#534)
* update deps

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix code to support all versions and regen tests

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-03-09 05:27:48 +01:00
Cesar Berrospi Ramis a661bb10cb fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens (#533)
* fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests: add a pytest fixture to test LineBasedTokenChunker efficiently

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: apply style conventions to test_line_chunker.py test module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-06 12:52:30 +01:00
samiuc e363c951d8 feat: add plain-text serializer (#522)
* feat: add plain-text serializer

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

* fix code coverage

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

---------

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>
2026-03-06 10:20:21 +01:00
Ivan Traus 2debe0836f fix: prevent hang in export_to_markdown() on nested RichTableCells (#525)
* fix: prevent hang in export_to_markdown() on nested RichTableCells

When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got &#124;-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.

Fix: add two module-level helpers:

  _cell_content_has_table(item, doc) — walks the item subtree and
  returns True if any node is a TableItem.

  _mark_subtree_visited(item, doc, visited) — adds all nodes in the
  subtree to the shared 'visited' set that the document serializer uses
  to prevent double-emitting items.

In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.

For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.

Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* style: apply ruff formatting to markdown serializer

Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* test(markdown): cover descendant table detection in rich-cell helper

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): preserve nested table content as flattened text

Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items.  This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.

Signed-off-by: Ivan Traus <ivan@liminary.io>

* fix(markdown): use TextItem instead of DocItem in subtree text collector

Signed-off-by: Ivan Traus <ivan@liminary.io>

* refactor(markdown): flatten only nested tables, preserve sibling rich formatting

Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.

Signed-off-by: Ivan Traus <ivan@liminary.io>

---------

Signed-off-by: Ivan Traus <ivan@liminary.io>
2026-03-05 09:54:27 +01:00
ea359bcc63 feat: table aware chunking (#527)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-04 16:25:57 +01:00
Matvei SmirnovandCesar Berrospi Ramis c566268e0a fix: rich table triplet serialization (#425)
* fix: rich table triplet serialization

Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: remove kwargs from 'export_to_dataframe' signature

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-26 11:39:34 +01:00