Commit Graph
172 Commits
Author SHA1 Message Date
Panos Vagenas fdcdfd1f27 feat(IDocTags): add content wrapping for handling whitespace (#489)
* feat(IDocTags): add content wrapping for handling whitespace

Also:
- fix caption serialiazation
- consolidate indendation via `pretty_indentation` param

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix content tag value

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-23 15:20:54 +01:00
e5c0015fbe fix(IDocTags): align code labels with Linguist (#484)
* fix(IDocTags): align code labels with Linguist

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fixed bug code blocks programming language not parsed

Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* it works

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

* removed debugging line from test

Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: mao <mao@login-c.zuvela.ibm.com>
Signed-off-by: Matteo Omenetti <omenetti.matteo@gmail.com>
Co-authored-by: mao <mao@login-c.zuvela.ibm.com>
Co-authored-by: Matteo Omenetti <omenetti.matteo@gmail.com>
2026-01-23 15:20:25 +01:00
Peter W. J. Staar a6d2be687e fix: Fix transparency rendering in all visualizers (#481)
* fix: fixed the transparency

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-01-20 09:35:26 +01:00
Panos Vagenas d9e8d37a91 fix(IDocTags): fix InlineGroup serialization and deserialization (#477)
fix(IDocTags): fix `InlineGroup` serialization & deserialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 16:45:46 +01:00
Panos Vagenas 17bd21ee14 feat(IDocTags): add fine-grained content serialization filtering (#476)
* feat(IDocTags): add fine-grained content serialization filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update token analysis example

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-19 13:49:29 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
Panos Vagenas 0c9c70db76 feat(IDocTags): add XML escape modes (#471)
* feat: add XML escape modes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove workaround for IDE debugging

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix inline handling, descope entities mode for now

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix escaping/CDATA interference with inline groups, formatting, code languages; add underline tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup, test expansion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-12 12:46:26 +01:00
Panos Vagenas 51a01a62c9 fix: skip annotation migration if respective meta present (#469)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-08 18:02:18 +01:00
SivaandPanos Vagenas 1d2e0c7ebe feat(DocItem): Add comments field for linking annotations to document… (#465)
* feat(DocItem): Add comments field for linking annotations to document items

Implements support for linking comments (from Word/PPT documents) to their
annotated content using the established FloatingItem/RefItem pattern.

Changes:
- Add `comments: List[RefItem]` field to DocItem class
- Update `_update_breadth_first_with_lookup()` to handle comment references on deletion
- Bump CURRENT_VERSION to 1.9.0
- Fix version comparison bug (string vs integer for minor version)
- Add 4 new tests for comments functionality
- Update test data files for new schema

Closes: docling-project/docling#464
Related: docling-project/docling#2834

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>

* improve comment Pydantic serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_comment, update tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce fine-granular references with span ranges

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify last test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-08 05:00:57 +01:00
Panos Vagenas 8b082e9b9d feat: enable heading-only chunks for empty-section headings (#461)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-12-18 14:35:15 +01:00
Peter W. J. Staar dda9c885a3 feat: idoctags serialization and deserialization matching the iso proposal (#457)
* Working on ISO standard for IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the pre-commit, still tons of work to do

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated idoctag token class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* need to start fixing the bugs and add proper tests for idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the testing for dcotags and idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the location tokens in idoctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the DocumentToken

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the idt quot issue

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Added IDocTagsSerializationMode

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the deserializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the captions for floating items

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* still need to fix some seserialization tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* finally all is working and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* made the OTSL serialization and deserialization self-contained in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed some unclean code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* did some fixes and clean up

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests, expanded to deserialize to deal with nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the deserialization for formatting

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the formatting and the complex inline groups in nested lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added get_category

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* make IS_SELFCLOSING a set

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the regex

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the if ... return into if ... elif ... return

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the facets parsing

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the footnotes for Tables and Pictures in IDocTags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with latest doctags token table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed private name

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-17 14:38:10 +01:00
Peter W. J. Staar 9a42a3c860 feat: updating the idoctags serializer (#450)
* feat: updating the idoctags serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* feat: adding IDocTagsToken class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the list serializer in the doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added missing document serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added tests for content suppression

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests for serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* deal with differences between xml serialization in mac and linux

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (3)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (4)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixing the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed all the tests, except the no-content with lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the no-content in lists with doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-10 11:39:40 +01:00
Christoph AuerandPanos Vagenas a5e2d2dcc0 feat: Visualize reading-order inside pictures (#446)
* fix: robustify page filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat: Visualize reading-order inside pictures

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add EMPTY_VALUE to DEFAULT_EXPORT_LABELS

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-28 16:29:32 +01:00
Peter W. J. Staar 76e14fdb97 feat: add latex serializer (#445)
* feat: add latex serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed spurious copyright

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the multiline

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-28 14:43:37 +01:00
Peter W. J. StaarandPanos Vagenas 92d60b0948 feat: added the Azure Document Intelligence (#395)
* added the Azure Document Intelligence

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the formatting and MyPy, working on the tests now

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added test document

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* replace all  with pydantic dataclasses

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* minor azure serializer refactoring

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-27 11:32:15 +01:00
Cesar Berrospi Ramis 8feb09f1cf fix: markdown serialization of hyperlink with code (#434)
* chore: leverage fixtures in tests and add type annotations

Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(markdown): serialize hyperlink as code always with single backticks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-11-21 14:05:29 +01:00
Panos Vagenas c9e5fb4a1c feat(experimental): add new DocTags serializer (#412)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-20 13:06:38 +01:00
Panos Vagenas 7bd274b84a chore: fix invalid test filenames (#431)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-18 11:31:29 +01:00
Panos Vagenas 3d13b02756 fix: improve meta migration and warning handling (#417)
* fix: improve meta migration and warning handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove commented-out code

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 16:25:14 +01:00
3097645198 feat: add code chunking functionality (#398)
* initial code chunking for docling-core

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* include language detections, add code chunking into hierarchical chunker

* add serializer, internal marking of chunkers, typing

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* Update docling_core/transforms/chunker/hierarchical_chunker.py

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* run all pre-commit less pytest

* update test files for code ID

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* update uv.lock

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* revert to stricter treesitter versioning due to compatibility

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove language detection (to be run by client, i.e. docling)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align new dependency specs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address backticks, ABC, and supported languages feedback

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove Language class and reuse CodeLanguageLabel

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* refactoring and improvements

- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 10:28:54 +01:00
Panos Vagenas 2ee3cacdd6 feat: add metadata model hierarchy (#408)
* feat: add metadata model hierarchy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add deprecation, add first migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend annotations migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update with feedback

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose main prediction

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* ideas on enforcing separation between standard and custom fields

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add custom field setter method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update Markdown serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert description, add include_non_meta, showcase custom serializer for summaries

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify customization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix reference exclusion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* eliminate serialization dupliation between meta & (legacy) annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove old file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix item used in get_parts for meta ser

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* serialize GroupItem meta prior to content, DocItem meta after content

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore ser order for all nodeitems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* move meta serialization into DocSerializer.serialize() to maintain seamless chunking integration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add allow- & block-lists for meta names, add std field name enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML serializer, document meta field names, rename SMILES field

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make TabularChartMetaField.title optional, expose new classes through __init__.py, add MetaUtils

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add DocTags serialization, revert smiles to smi to prevent confusion with plural

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-30 11:15:14 +01:00
Michele Dolfi 47b70d3c24 feat: python 3.14 compatibility (#403)
update deps for python 3.14 compatibility

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-16 15:58:29 +02:00
Erik abfa81f065 feat: rendering of checkboxes in markdown (#381)
* added support for rendering checkboxes

* DCO Remediation Commit for Erik <erik@apption.com>

Signed-off-by: Erik <erik@apption.com>

* DCO Remediation Commit for Erik <erik@apption.com>

I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 7af470451b

Signed-off-by: Erik <erik@apption.com>

* added code suggestions for better checkboxes

* updated test data

* fixed formatting

* DCO Remediation Commit for Erik <erik@apption.com>

I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: fc7a57e1f7
I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 7ea737ecd4
I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 5c431b52c8

Signed-off-by: Erik <erik@apption.com>

---------

Signed-off-by: Erik <erik@apption.com>
2025-10-01 11:16:31 +02:00
Panos Vagenas 9df72083c0 fix(markdown): fix single-row table serialization (#385)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-10 16:11:34 +02:00
Maxim LysakandMaksym Lysak b13267f18b feat: Introduction of fillable TableCell (#384)
* New properties for TableCell, corner_header and fillable

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Updated docs

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* bumped docling document version to 1.7.0

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-09-09 15:48:42 +02:00
Panos Vagenas 2dc57c1769 fix: fix handling of generic groups in rich table cells (#383)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-09 09:55:55 +02:00
Panos Vagenas b60ac192ef feat: add support for heading with inline in HTML & DocTags (#379)
* test: refactor serialization tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* further refactor serialization tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename inline test file for markdown

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify Markdown handling of inline in heading / list item

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix input test data & make it more interesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML & DocTags support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize and refactor HTML ser case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize MD serializer too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-03 16:56:58 +02:00
Panos Vagenas 9dc526de53 feat: add page filtering to DoclingDocument (#378)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-02 15:56:35 +02:00
Panos Vagenas b2095b31de fix: use in-OTSL DocTags for rich table cells (#375)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-28 09:46:46 +02:00
Panos Vagenas 1d04154378 feat: add rich table cells (#368)
* feat: add rich table cells

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* propagate cell text resolution, cover row deletions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add doctags, fix referential integrity, expand tests, reenable mypy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify / remove serialize_cell

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update rich table cell refs in doc indexing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update notebook

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose new classes in `docling_core.types.doc`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-27 10:07:47 +02:00
Panos Vagenas bbe6243833 chore: add DoclingDocument validation rules (#349)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-19 14:14:13 +02:00
Panos Vagenas 99eabb3b01 feat: add document concatenation (#365)
* chore: add document concatenation

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* finalize exposed API, add tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-19 12:30:33 +02:00
Cesar Berrospi Ramisand15mbp 23badf2a1d fix: add forward slashes to singleton tags (#369)
* Add forward slash to singleton tags

See issue #354 on docling-project/docling-core

Signed-off-by: 15mbp <100952997+15mbp@users.noreply.github.com>

* test(HTML): fix ground truth data files

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: 15mbp <100952997+15mbp@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: 15mbp <100952997+15mbp@users.noreply.github.com>
2025-08-19 10:05:34 +02:00
Panos Vagenas 807d972f05 fix(HTML): fix nested list serialization edge cases (#367)
* fix(HTML): (initial state without fix) fix nested list serialization edge cases

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* actual fix

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-13 15:06:54 +02:00
Christoph Auer f9b3b499de fix: LayoutVisualizer should traverse pictures (#358)
LayoutVisualizer should traverse pictures

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2025-07-23 09:56:00 +02:00
Cesar Berrospi Ramis 5a7883cee5 fix: HTML serialization of nested lists (#359)
* fix(serializer): the HTML serializer should append nested lists to list items

For a valid HTML document, the serializer should ensure that sub-list items are indented
under their respective main list items.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* test: address several warning messages

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-07-23 09:31:02 +02:00
Panos Vagenas 3a0b7471ba feat: add page chunking (#337)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-07-15 11:03:11 +02:00
Ryan Fernandes cb59fd39bc feat: Additional DoclingDocument methods for use in MCP document manipulation (#344)
* insert_* methods and range manipulation methods

* DCO Remediation Commit for Ryan Fernandes <ryan@fernandes.us>

I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: cc0c4b283f

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* insert_node_items method with recursive _append_item_copies helper for cross-document manipulation

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* add_node_items complement to inser_node_items, refactored extract_items_range to be more robust, added insert/add_document

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* Extended table data editing functionality

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* Added new document manipulation tests

* Extra gt files

* Refactor for reusable code - table methods

* Private methods for more concise insert_* methods

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* Docstring changes

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* ValueError for failed insertions

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* DCO Remediation Commit for Ryan Fernandes <ryan@fernandes.us>

I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 01e8b9cb03
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 8c016d9d32
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 308900270a

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

---------

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
2025-07-09 10:53:04 +02:00
Panos Vagenas baa2cc3714 feat: enable precision control in float serialization (#352)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-07-08 17:00:29 +02:00
Panos Vagenas 14a4fdee87 feat: remodel lists, add MD & HTML ser. params, enable unset marker (#339)
* feat: remodel lists

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* prepare test document separate markers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add MD/HTML serializer options, expand test data

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* create list groups where not in place

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add auto-increment logic

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore UnorderedList as deprecated alias (for backwards compatibility)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable unset marker case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make ordered markers able to be "unset" (empty) too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename default marker mode to auto

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-27 14:52:25 +02:00
Panos Vagenas c9374e8fe4 fix(markdown): add heading formatting, fix code & formula formatting (#336)
* feat(markdown): add formatted headings (show behavior before change)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add change and updated test data

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix when formatting & escaping are applied (depending on type)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-20 14:10:48 +02:00
Panos Vagenas 78b796221d feat(viz): add reading order branch numbering, fix cross-page lists (#334)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-18 14:18:48 +02:00
Panos Vagenas 574779f4c7 chore: add ref number normalization (#322)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-17 12:31:39 +02:00
Panos Vagenas 5c99722b38 fix(HybridChunker): improve long heading handling (#333)
* fix(HybridChunker): improve long heading handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix warning wording

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-17 12:21:51 +02:00
Riccardo Maggioni 0fd3c1cbe6 feat: add parameter to choose of which pages export the doctags (#290)
* feat: add parameter to choose of which pages export the doctags

Signed-off-by: rickymaggio02 <riccardo.maggioni@googlemail.com>

* feat: added test and better description of the method

Signed-off-by: rickymaggio02 <riccardo.maggioni@googlemail.com>

---------

Signed-off-by: rickymaggio02 <riccardo.maggioni@googlemail.com>
2025-06-16 15:30:09 +02:00
Peter W. J. Staar 3b998795b8 feat: add improved table serializer and visualizer (#328)
* feat: add improved table serializer and visualizer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* visualizer for rows and columns

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-06-13 11:50:27 +02:00
Christoph AuerandPanos Vagenas aa430d3776 feat: New labels for CVAT annotation (#314)
* New labels and utility for CVAT annotation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add EMPTY_VALUE to TextItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* align usage of content layer param (#326)

* align usage of content layer param

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* parametrize content layers in visualizers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-06-11 19:25:13 +02:00
Peter W. J. Staar 45667c751b feat: Visualizer for table cells (#325)
* feat: visualizing the tables

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Added test for table visualizer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* cleaned up code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with content-layer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-06-11 16:53:09 +02:00
Panos Vagenas 01b27b57b3 fix: warn when adding misplaced ListItem via API (#321)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-08 20:04:40 +02:00
Panos Vagenas dcc198f7c6 feat(doctags): add enclosing bbox to inline (#302)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-06 16:35:56 +02:00