* fix(serializer): derive markdown table header rows from column_header flags
MarkdownTableSerializer hardcoded grid row 0 as the only header row, while
TableItem._export_to_dataframe_with_options, the HTML serializer and the OTSL
export all derive the header rows from TableCell.column_header. A table with a
stacked header therefore serialized to markdown with its lower header rows
demoted to data rows, and HybridChunker repeated only the first of them at the
top of each chunk of an oversized table.
Take the header row count from the flags and flatten those rows into the single
header row GFM allows, dropping the repeated text a cell spanning several header
rows produces. When no cell in the table carries column_header the previous
behavior is kept, since backends that never set the flag cannot be told apart
from genuinely headerless tables.
Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>
* test(serializer): cover tables whose header flags start below row 0
The empty-header-block branch of _flatten_header_rows was the one line of the
patch left uncovered. Pin the behavior it produces: with no leading run of
flagged rows there is no header to promote, so the header is empty and every
row stays in the body rather than one being consumed.
Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>
* fix(serializer): join stacked header cells with a documented separator
Review feedback: a blank space loses the boundary between header levels, so
"native backend" over "TTS" is indistinguishable from a single cell reading
"native backend TTS". Join with " - " instead, kept as a module constant, and
document the header-resolution contract on MarkdownTableSerializer.
Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>
* fix(serializer): only count header rows a column header starts on
TableData.grid repeats a row-spanning cell into every row it covers, so a
header cell with row_span > 1 leaves a column_header-flagged cell in the data
rows beneath it. Counting every row that holds a flagged cell therefore pulled
those data rows into the header block, where they were flattened into the
column names and dropped from the body entirely.
Require that a header cell starts on the row, matching the same fix applied to
the DataFrame export in #756.
Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>
* perf(serializer): avoid iterating grid twice; style and tests improvements
In '_count_header_rows' of the Markdown serializer, avoid iterating
the entire grid twice in all scenarios.
Strengthen docstrings in markdown serializer.
Add test for single header row with flags.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Anusha Acharya <acharyaanusha123@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: modernize type hints with PEP 604
Enforce Ruff UP007 and UP045 to align with PEP 604 (union types as X | Y).
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: upgrade 'uv.lock' dependencies
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: updated the doclang serializer for groups
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix(markdown): serialize newlines as GFM hard line breaks
Single \n in body text and list items is now serialized as ' \n'
(two trailing spaces), which GFM renders as a hard line break.
Double \n\n is preserved as a paragraph break.
Newlines in headings are collapsed to a space, since headings cannot
span multiple lines.
PlainTextDocSerializer is unaffected via overridable instance methods.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(deps): upgrade uv.lock dependencies
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* contextualize function() skip excluded fields
This change proposes that we use model_dump to skip/avoid serializing
fields that are in excluded_embed. This should reduce unnecesary
computation and is also more precisely what I believe we are intending
to do here in that we do not need to unnecessarily filter the results
below in the for-loop.
Signed-off-by: Jordan White <jordan.page.white@gmail.com>
* style(chunker): fix style issue C416
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(chunker): resolve excluded_embed to field names
model_dump(exclude=...) requires Python attribute names, not
aliases. excluded_embed stores alias keys, which happen to
match today but would silently fail to exclude fields in
subclasses where they differ, leaking excluded fields into the
embedding context.
Adds BaseMeta._excluded_embed_field_names() to resolve each
entry through the alias-to-field-name map. contextualize() now
calls this helper instead of using excluded_embed directly.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(chunker): fix pydantic deprecation
Deprecated in pydantic v2.11, removed in v3: model_fields can only be accessed
from class object, not the instance.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(chunker): add test to verify contextualize() excludes the right fields
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: update uv.lock dependencies
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Jordan White <jordan.page.white@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
A page break falling inside a ListGroup was emitted as an ordinary list
part and joined with the list separator "\n", while one falling between
top-level items was joined with "\n\n". The placeholder is substituted
into the already joined text, so the spacing was fixed by whichever
scope emitted the break and could not be renegotiated downstream.
Exporting a document globally and page by page therefore disagreed by
two bytes whenever a list straddled a page boundary, although the
content matched.
MarkdownListSerializer now separates a page break part from its
neighbours with "\n\n", so the marker forms a block in either scope. To
let it recognise such a part, DocSerializer.serialize() returns the
_PageBreakSerResult as such instead of rebuilding a plain result and
dropping the type.
Output is unchanged for callers leaving page_break_placeholder unset,
as no page break parts are produced at all in that case.
Signed-off-by: OmarFaig <omerfaiq7@gmail.com>
* feat: add initial dlgrep implementation
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* refactor: migrate CLI to Typer
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* docs: expand README
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* docs: improve README examples
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* fix: streamline dlgrep output
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* fix: make text context output grep-compatible
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Improve XPath and inline context output
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* dlgrep support extensions
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Use git sources for doclang and docling-core
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* feat: emit native JSON records
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Remove search class filter
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Make section output addressable
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Improve README examples
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* feat: build and publish dlgrep from this repository
dlgrep becomes a uv workspace member under packages/dlgrep, published as
its own wheel and released in lockstep with docling-core: same version,
exact `docling-core==` pin, both wheels built from the same commit. This
removes the git-branch dependency dlgrep needed to track unreleased
docling-core changes.
- workspace wiring; ruff, mypy and pytest config inherited from the root,
dlgrep in the dev group so `uv sync` covers it
- release.sh writes both versions and the pin from one TARGET_VERSION
- build-packages.sh builds into per-package dist dirs; pypi.yml publishes
docling-core first, then dlgrep
- check_lockstep.py guards against version drift, which uv cannot catch on
its own: the workspace source replaces the pin with a local path, so a
wrong pin resolves and locks cleanly
- checks.yml and pypi.yml install both wheels together, which makes a
mismatched pin fail before anything is published
- declare click and lxml, previously only transitive
- read __version__ from package metadata instead of a hardcoded literal
- flag dlgrep as experimental in its help output, since the shared version
number would otherwise imply a maturity it does not have
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* ci: simplify lockstep release tooling
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* chore(dlgrep): drop handoff notes, keep only the CLI plan
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* refactor(doclang): isolate source mapping recorder
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* build(dlgrep): move dependencies to core extra
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* feat(dlq): rename CLI and add list command
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* feat(dclq): rename tool and consolidate plan
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* fix(dclq): bound archive document reads
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.
The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.
Fixes#685
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* fix: lazy load transformers in HybridChunker
Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(chunker): guard check at point of use
Guard import check at the point of use in hybrid chunker.
Follow the identical pattern for lazy load as in docling.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(chunker): add tests for lazy-loading with HybridChunker
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: normalize import calls from DoclingDocument
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(doclang): enforce image size budgets for archive assets and ImageRef
- Cap `.dclx` `pages/` and `assets/` rasters with `max_image_decoded_size` before Pillow opens them
- Apply the same limit to Path and `file://` ImageRef loads
- Add tests covering oversize archive images and path-based ImageRefs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* bump doclang to 0.7.3 and clarify ImageRef remote-fetch guidance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- Cap DocLang XML input bytes before `parseString`
- Bound nesting depth and element count after DOM construction
- Expose limits via settings / `deserialize_str` kwargs and cover with unit tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add .dclx archive save and load support
- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add support for `<description>` and `<summary>`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* What was removed
The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.
Deleted packages/modules
- docling_core/types/legacy_doc/ (whole folder)
- docling_core/types/rec/ (whole folder)
- docling_core/utils/legacy.py (legacy conversion functions)
- docling_core/utils/validate.py (RAW/ANN/OCR CLI)
Edited production code
- types/__init__.py — dropped the Record export
- utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
validate_unique_list/validate_datetime
- utils/generate_docs.py — dropped Record from MODELS
Removed resources/artifacts
- resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json
Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.
Verification
- No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
- from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
- Full project suite: 496 passed.
- generate_docs.py now emits only DoclingDocument.json and Generic.json.
The full plan is saved at _plans/remove-legacy-doc-and-rec.md.
Two notes worth flagging:
- Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
bump downstream.
- I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
approval flow.
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the checks.yaml
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the minimal_document_schema_flat.json
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/resources
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the files and updated the imports according to Cesar's suggestion
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the ruff linter
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): align with DocLang v0.6 picture model
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): enhance picture metadata handling with custom fields
- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): deserialize field regions and add round-trip fixtures
- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat: add keywords and topics meta fields
Introduce two simple, additive meta fields:
- KeywordsMetaField — unique list of keyword strings
- TopicsMetaField — unique list of topic / subject strings
Both attach to BaseMeta (node-level) and to the new DocumentMeta
(document-level), which is wired into DoclingDocument.meta. HTML,
Markdown, and Doclang serializers render the new fields as comma-
separated values. Schema bump is intentionally avoided since the
fields are optional and backward-compatible.
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* test: cover keywords and topics meta fields
Add round-trip, dedup, has_content, HTML escape, and Markdown
rendering tests for KeywordsMetaField, TopicsMetaField, and
DocumentMeta. Document-level meta absent-by-default is also covered.
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* style: satisfy ruff UP038 and import order
Convert tuple-form isinstance checks to PEP 604 unions and
reorder a misplaced import flagged by ruff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* fix: align with HTML attribute format from #613
Post-rebase onto main, the HTML metadata serializer now emits
`data-meta-name="<field>"` instead of valueless `data-meta-<field>`
and applies a single trailing html.escape pass. Drop the per-type
escape from the Keywords/Topics branch (was producing double
escaping after rebase) and update the two test assertions to the
new attribute shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* revert: remove DocLang keywords/topics serialization
Per discussion, DocLang is a standard whose extension is out of
scope for this PR. Drop the KeywordsMetaField/TopicsMetaField
branches from DoclangMetaSerializer and their imports; HTML and
Markdown serialization remain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* refactor: address review feedback on keywords/topics meta fields
- Replace MetaFieldName inline comments with Attributes docstring (TableFormerMode pattern)
- Swap list[str]+manual dedup → UniqueList[str] in KeywordsMetaField and TopicsMetaField
- Add Field(description, examples) to all BaseMeta fields distinguishing entities/keywords/topics
- Remove DocumentMeta class and DoclingDocument.meta field; document-level meta goes via body
- Consolidate keywords/topics tests into test_semantic_base_meta_fields_roundtrip_and_html_rendering
- Regenerate docs/DoclingDocument.json
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(meta): reuse validator between keywords and topics
Revert the deduplication as part of a validator.
Share the validator between keywords and topics meta classes.
Adopt the standard Google style for enum classes.
Adapt the regression tests to the changes above.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Ana Daniele <ana.daniele@ibm.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(html-serializer): emit XHTML-compatible metadata attributes
HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.
Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.
Fixes#598.
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
* fix(html-serializer): rebase changes to 'main' and add tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(html-serializer): remove redundant CSS rule
Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
When a `TableItem` produced empty serialized text, `TripletTableSerializer`
was still populating the shared `visited` set via `RichTableCell._get_text()`.
Downstream items were then treated as already chunked and the chunker
emitted 0 chunks (e.g. on DOCX layout/header-only tables).
This change:
- Isolates `visited` while exporting the table dataframe and only merges
the local copy back into the shared set when non-empty text is produced.
- Promotes the header row into a data row when the dataframe has 0 data
rows but non-empty column names, so header-only tables still emit text.
- Handles single-row single-column tables by emitting the cell text
directly instead of an empty triplet list.
- Adds `_flatten_table_text` as a fallback for tables whose triplet form
collapses to empty text.
- Guards against appending empty text to serialization parts.
Adds regression tests covering header-only tables, single-row
single-column tables, end-to-end `HierarchicalChunker`/`HybridChunker`
runs on header-only / single-cell rich tables, and verifies that empty
tables do not consume their children's refs.
Closesdocling-project/docling#3335
Signed-off-by: Manohar Mallipudi <manohar.m4475@yahoo.com>
* feat: extending BaseMeta with language, entities, etc for docling-agent
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ignore serialization of empty Meta in HTML
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* addressing the good comments from Cesar
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat(serializer): add MsExcelMarkdownDocSerializer for sheet-name headings
Add `MsExcelMarkdownFallbackSerializer` and `MsExcelMarkdownDocSerializer`
to the serializer package so that `GroupLabel.SHEET` groups are rendered as
level-2 Markdown headings when exporting Excel-sourced documents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* DCO Remediation Commit for Smeet23 <smeetagrawal2003@gmail.com>
I, Smeet23 <smeetagrawal2003@gmail.com>, hereby add my Signed-off-by to this commit: 2a3808e5dc
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
* style: apply ruff formatter to markdown_excel.py
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
---------
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: profile a document or collection
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(profiler): add deciles and histograms
Add deciles and histograms to the Docling collection statistics.
Add an example script to plot histograms.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(profiler): add option to plot log frequencies in histogram
Add the option to plot the histogram frequencies in logarithmic scale.
Extend README with documentation on the document profiler.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(profiler): cover missing lines in doc_profiler with tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests: add a pytest fixture to test LineBasedTokenChunker efficiently
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: apply style conventions to test_line_chunker.py test module
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: prevent hang in export_to_markdown() on nested RichTableCells
When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got |-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.
Fix: add two module-level helpers:
_cell_content_has_table(item, doc) — walks the item subtree and
returns True if any node is a TableItem.
_mark_subtree_visited(item, doc, visited) — adds all nodes in the
subtree to the shared 'visited' set that the document serializer uses
to prevent double-emitting items.
In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.
For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.
Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* style: apply ruff formatting to markdown serializer
Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* test(markdown): cover descendant table detection in rich-cell helper
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): preserve nested table content as flattened text
Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items. This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): use TextItem instead of DocItem in subtree text collector
Signed-off-by: Ivan Traus <ivan@liminary.io>
* refactor(markdown): flatten only nested tables, preserve sibling rich formatting
Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.
Signed-off-by: Ivan Traus <ivan@liminary.io>
---------
Signed-off-by: Ivan Traus <ivan@liminary.io>