Commit Graph
17 Commits
Author SHA1 Message Date
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
Manohar Mallipudi 507df717c1 fix(chunking): handle empty TripletTableSerializer output without consuming children (#596)
When a `TableItem` produced empty serialized text, `TripletTableSerializer`
was still populating the shared `visited` set via `RichTableCell._get_text()`.
Downstream items were then treated as already chunked and the chunker
emitted 0 chunks (e.g. on DOCX layout/header-only tables).

This change:
- Isolates `visited` while exporting the table dataframe and only merges
  the local copy back into the shared set when non-empty text is produced.
- Promotes the header row into a data row when the dataframe has 0 data
  rows but non-empty column names, so header-only tables still emit text.
- Handles single-row single-column tables by emitting the cell text
  directly instead of an empty triplet list.
- Adds `_flatten_table_text` as a fallback for tables whose triplet form
  collapses to empty text.
- Guards against appending empty text to serialization parts.

Adds regression tests covering header-only tables, single-row
single-column tables, end-to-end `HierarchicalChunker`/`HybridChunker`
runs on header-only / single-cell rich tables, and verifies that empty
tables do not consume their children's refs.

Closes docling-project/docling#3335

Signed-off-by: Manohar Mallipudi <manohar.m4475@yahoo.com>
2026-05-28 15:41:47 +02:00
Cesar Berrospi Ramis afa5bd9e3a chore: document-level metadata serialization via body field (#551)
* feat: add 'meta' field to DoclingDocument at root level

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(metadata): reuse DoclingDocument objects with fixtures

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add document-level meta to the outline serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: add helper function to compare or regenerate ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: regenerate (upgrade) ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): set default indent to 2 spaces in outline serializer

To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): add line break after ref in outline md serialization

To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: rollback adding 'meta' field to DoclingDocument

Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-17 14:55:52 +01:00
Matvei SmirnovandCesar Berrospi Ramis c566268e0a fix: rich table triplet serialization (#425)
* fix: rich table triplet serialization

Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: remove kwargs from 'export_to_dataframe' signature

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-26 11:39:34 +01:00
Cesar Berrospi Ramis 73b07572cf fix: support single-column table default serialization (#526)
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-24 12:23:01 +01:00
Cesar Berrospi Ramis a3b6e3fb89 fix(chunker): propagate 'traverse_pictures' parameter to chunker (#518)
* fix(chunker): propagate 'traverse_pictures' parameter from serializer to chunker

Propagate 'traverse_pictures' parameter from the 'serializer_provider' to the HierarchicalChunker, to ensure
that chunks include any TextItem children from PictureItem.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(chunker): simplify imports

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-20 20:37:43 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
Peter W. J. Staar 928aaed9ab chore: remove unnecesary copyright headers (#440)
* chore: remove unnecesary copyright headers

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted, still need to fix a test ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-27 11:17:23 +01:00
Panos Vagenas d05fe08546 feat: promote serializers to stable API (#276)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-30 11:26:39 +02:00
Panos Vagenas 23036e17fa feat: add serializer provider to chunkers (#239)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-09 12:45:44 +02:00
Panos Vagenas 5e4c0fd51f feat: integrate serialization API into chunkers (#221)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-08 14:04:40 +02:00
Christoph AuerandPanos Vagenas 786f0c6833 feat: Add ContentLayer attribute to designate items to body or furniture (#148)
* feat: Add ContentLayer attribute to designate items to body or furniture

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce safer data gen mechanism, update chunking test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Do not make test rely on order in yaml

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore: format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: legacy_to_docling_doc must use content_layer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add content_layer in iterate_items

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Bump format version, add model_validator for old page_header,page_footer in body

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: Change to before model_validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Address review comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-02-10 10:58:41 +01:00
Michele Dolfi 268c294cc9 fix: always write with utf8 encoding (#111)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-12-17 09:37:01 +01:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Panos Vagenas b5592ad747 fix: align chunk ref format with one used in Document (#37)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-07 09:07:08 +02:00
Panos Vagenas 4bde51528d feat: support heading as chunk metadata (#36)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-30 11:08:09 +02:00
Panos Vagenas 9698d30288 feat: add hierarchical chunker (#18)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-11 12:57:56 +02:00