Commit Graph
73 Commits
Author SHA1 Message Date
Panos Vagenas 224eeeab60 fix(doclang): enforce image size budgets for archive assets and ImageRef (#684)
* fix(doclang): enforce image size budgets for archive assets and ImageRef

- Cap `.dclx` `pages/` and `assets/` rasters with `max_image_decoded_size` before Pillow opens them
- Apply the same limit to Path and `file://` ImageRef loads
- Add tests covering oversize archive images and path-based ImageRefs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump doclang to 0.7.3 and clarify ImageRef remote-fetch guidance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-07-15 10:47:13 +02:00
Panos Vagenas 9dddcf7137 fix: clamp provenance bboxes and DocLang locations to valid page bounds (#643)
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:44:20 +02:00
Panos Vagenas 6fd2c1a247 fix: collapse consecutive names in concatenate output (#645)
- when merging documents, successive identical source names are deduplicated in the resulting document name (e.g. `a + a + b` → `a + b`)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:25:37 +02:00
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
Peter W. J. Staar c1c9ea80a5 feat: support vertical orientation in TableData row/column bounding boxes (#622)
* fix: vertical-rows-and-columns

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added Orientation to TableData to deal with vertical tables

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* moved to ROT_0, etc

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the convention of the rotation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-29 17:16:08 +02:00
b20b5677c0 fix(html-serializer): emit XHTML-compatible metadata attributes (#613)
* fix(html-serializer): emit XHTML-compatible metadata attributes

HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.

Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.

Fixes #598.

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>

* fix(html-serializer): rebase changes to 'main' and add tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(html-serializer): remove redundant CSS rule

Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-05-29 10:01:30 +02:00
Panos Vagenas 2087d0f362 fix: refine ImageRef URI handling (#595)
* fix: refine ImageRef URI handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make URI handling configurable via settings

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-04-22 16:13:10 +02:00
Peter W. J. StaarandPanos Vagenas f20068db91 feat: add transforms in the hierarchy (#572)
* feat: add transforms in the hierarchy

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* implement hierachizing & flattening based on shift/tree operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflict

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-04-01 17:12:06 +02:00
Panos Vagenas 57187b2364 chore: address subtree moving within same parent (#564)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-30 10:41:54 +02:00
Panos Vagenas 0cfb663275 fix: extend validation to address duplicate refs (#565)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-25 15:43:28 +01:00
samiuc 3e030edc6f fix: expose traverse_pictures in export_to_markdown and export_to_text (#557)
* fix: expose traverse_pictures in export_to_markdown and export_to_text

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

* address review comment

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>

---------

Signed-off-by: samiuc <sami.ullah.chat@gmail.com>
2026-03-20 08:12:54 +01:00
Panos Vagenas c57e50ac43 fix: improve rich table cell validation (#550)
Extend validation to also cover case of missing reference to
rich child on table item.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-16 17:13:53 +01:00
Panos Vagenas b93d5a3920 feat: introduce field data model incl. Doclang serialization (#519)
* feat: introduce new kv data model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add form-table test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update invoice test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include pre-migration YAML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add nesting test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to field naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve prov migration & location serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable proper nested serialization without inline

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add field marker and test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize marker naming

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* updated API

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align with TextItem conventions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make FieldItem a DocItem

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend KV migration scope, extend tree manipulation operations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend kv migration tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add test with KV nesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve key splitting and location assignment

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor cleanup

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove unnecessary files

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_content parameter, expose export_to_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address conflicts, add save_as_doclang

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-03-13 15:37:42 +01:00
Cesar Berrospi Ramis b8ef7bad1b feat: add WebVTT export and save functionality (#523)
* feat(vtt): export and save to WebVTT format

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(vtt): omit empty blocks in parsing

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(vtt): add tests for exporting and saving to WebVTT

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-24 10:17:17 +01:00
Christoph Auer 6a04db77aa fix: shift KV/Form graph cell page numbers during DoclingDocument.concatenate (#521)
fix: When concatenating docs, adjust page numbers the GraphCell elements appear on

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-02-23 10:35:16 +01:00
Cesar Berrospi Ramis 3b0b909c89 fix(markdown): add an option to compact table serialization (#495)
* fix(markdown): add an option to compact table serialization

Add an option in MarkdownTableSerializer to remove padding in tables in markdown serialization.
Propagate the option to MarkdownDocSerializer and DoclingDocument.
Remove unnecessary use of 'mode' argument in 'open' function when mode is 'r'.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(markdown): keep alignment marks in '_compact_table'

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-29 16:23:54 +01:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
SivaandPanos Vagenas 1d2e0c7ebe feat(DocItem): Add comments field for linking annotations to document… (#465)
* feat(DocItem): Add comments field for linking annotations to document items

Implements support for linking comments (from Word/PPT documents) to their
annotated content using the established FloatingItem/RefItem pattern.

Changes:
- Add `comments: List[RefItem]` field to DocItem class
- Update `_update_breadth_first_with_lookup()` to handle comment references on deletion
- Bump CURRENT_VERSION to 1.9.0
- Fix version comparison bug (string vs integer for minor version)
- Add 4 new tests for comments functionality
- Update test data files for new schema

Closes: docling-project/docling#464
Related: docling-project/docling#2834

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>

* improve comment Pydantic serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add add_comment, update tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce fine-granular references with span ranges

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify last test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-01-08 05:00:57 +01:00
Peter W. J. Staar 9a42a3c860 feat: updating the idoctags serializer (#450)
* feat: updating the idoctags serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* feat: adding IDocTagsToken class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fix the get_special_tokens (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the list serializer in the doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added missing document serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added tests for content suppression

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests for serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added extra tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* ran pre-commit

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* deal with differences between xml serialization in mac and linux

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (3)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* revert to old doctags serialization (4)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixing the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed all the tests, except the no-content with lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the no-content in lists with doctags

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-12-10 11:39:40 +01:00
Panos Vagenas f8e09ecf69 fix: switch meta migration logging to info (#452)
* chore: switch meta migration logging to info

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* suppress deprecation warnings within migration, expand to TableItem, add tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-12-08 11:36:48 +01:00
Panos Vagenas 8bdeaa7238 fix: robustify page filtering (#437)
* fix: robustify page filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* cover validate_rules

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-26 13:53:15 +01:00
Cesar Berrospi Ramis 8feb09f1cf fix: markdown serialization of hyperlink with code (#434)
* chore: leverage fixtures in tests and add type annotations

Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(markdown): serialize hyperlink as code always with single backticks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-11-21 14:05:29 +01:00
Panos Vagenas 7bd274b84a chore: fix invalid test filenames (#431)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-18 11:31:29 +01:00
Cesar Berrospi Ramis 56b3c42c61 fix: DoclingDocument model validator should deal with any raw input (#419)
* fix: DoclingDocument model validator should deal with any raw input

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: mention 3.14 as valid Python version for development

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-11-13 16:37:11 +01:00
3097645198 feat: add code chunking functionality (#398)
* initial code chunking for docling-core

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* include language detections, add code chunking into hierarchical chunker

* add serializer, internal marking of chunkers, typing

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* Update docling_core/transforms/chunker/hierarchical_chunker.py

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* run all pre-commit less pytest

* update test files for code ID

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* update uv.lock

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* revert to stricter treesitter versioning due to compatibility

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove language detection (to be run by client, i.e. docling)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align new dependency specs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address backticks, ABC, and supported languages feedback

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove Language class and reuse CodeLanguageLabel

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* refactoring and improvements

- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 10:28:54 +01:00
Panos Vagenas 3e8d628eee fix: switch to safe YAML loader (#396)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-01 09:43:42 +02:00
Viroshaan Uthayamoorthy (KLP) 0512f4473d fix: add doc param to all export_to_dataframe() calls (#380)
* fix: add 'doc' to all export_to_dataframe calls

* DCO Remediation Commit for Viroshaan <viroshaan.uthayamoorthy@klp.no>

I, Viroshaan <viroshaan.uthayamoorthy@klp.no>, hereby add my Signed-off-by to this commit: 10317f8db7

Signed-off-by: Viroshaan <viroshaan.uthayamoorthy@klp.no>

---------

Signed-off-by: Viroshaan <viroshaan.uthayamoorthy@klp.no>
2025-09-09 14:42:57 +02:00
Panos Vagenas 2dc57c1769 fix: fix handling of generic groups in rich table cells (#383)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-09 09:55:55 +02:00
Panos Vagenas 9dc526de53 feat: add page filtering to DoclingDocument (#378)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-02 15:56:35 +02:00
Panos Vagenas 1d04154378 feat: add rich table cells (#368)
* feat: add rich table cells

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* propagate cell text resolution, cover row deletions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add doctags, fix referential integrity, expand tests, reenable mypy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify / remove serialize_cell

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update rich table cell refs in doc indexing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update notebook

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose new classes in `docling_core.types.doc`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-27 10:07:47 +02:00
Panos Vagenas bbe6243833 chore: add DoclingDocument validation rules (#349)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-19 14:14:13 +02:00
Panos Vagenas 99eabb3b01 feat: add document concatenation (#365)
* chore: add document concatenation

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* finalize exposed API, add tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-19 12:30:33 +02:00
Michele DolfiandCesar Berrospi Ramis 5afa99e028 fix: referenced artifacts relative to the document location (#361)
* fix relative references

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-07-30 13:00:03 +02:00
Cesar Berrospi Ramis 5a7883cee5 fix: HTML serialization of nested lists (#359)
* fix(serializer): the HTML serializer should append nested lists to list items

For a valid HTML document, the serializer should ensure that sub-list items are indented
under their respective main list items.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* test: address several warning messages

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-07-23 09:31:02 +02:00
Clément Doumouro ebd9147ced fix: only save applicable page images (#226)
fix(document): save document images page by page

Signed-off-by: Clément Doumouro <clement.doumouro@gmail.com>
2025-07-15 09:52:09 +02:00
Ryan Fernandes cb59fd39bc feat: Additional DoclingDocument methods for use in MCP document manipulation (#344)
* insert_* methods and range manipulation methods

* DCO Remediation Commit for Ryan Fernandes <ryan@fernandes.us>

I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: cc0c4b283f

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* insert_node_items method with recursive _append_item_copies helper for cross-document manipulation

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* add_node_items complement to inser_node_items, refactored extract_items_range to be more robust, added insert/add_document

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* Extended table data editing functionality

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* Added new document manipulation tests

* Extra gt files

* Refactor for reusable code - table methods

* Private methods for more concise insert_* methods

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* Docstring changes

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* ValueError for failed insertions

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

* DCO Remediation Commit for Ryan Fernandes <ryan@fernandes.us>

I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 01e8b9cb03
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 8c016d9d32
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 308900270a

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>

---------

Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
2025-07-09 10:53:04 +02:00
Panos Vagenas baa2cc3714 feat: enable precision control in float serialization (#352)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-07-08 17:00:29 +02:00
Panos Vagenas 14a4fdee87 feat: remodel lists, add MD & HTML ser. params, enable unset marker (#339)
* feat: remodel lists

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* prepare test document separate markers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add MD/HTML serializer options, expand test data

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* create list groups where not in place

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add auto-increment logic

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore UnorderedList as deprecated alias (for backwards compatibility)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enable unset marker case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make ordered markers able to be "unset" (empty) too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename default marker mode to auto

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-27 14:52:25 +02:00
Panos Vagenas 574779f4c7 chore: add ref number normalization (#322)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-17 12:31:39 +02:00
Riccardo Maggioni 0fd3c1cbe6 feat: add parameter to choose of which pages export the doctags (#290)
* feat: add parameter to choose of which pages export the doctags

Signed-off-by: rickymaggio02 <riccardo.maggioni@googlemail.com>

* feat: added test and better description of the method

Signed-off-by: rickymaggio02 <riccardo.maggioni@googlemail.com>

---------

Signed-off-by: rickymaggio02 <riccardo.maggioni@googlemail.com>
2025-06-16 15:30:09 +02:00
Panos Vagenas 01b27b57b3 fix: warn when adding misplaced ListItem via API (#321)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-08 20:04:40 +02:00
Panos Vagenas c383f64c44 fix: fix misplaced list items (#317)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-06 13:23:24 +02:00
Panos Vagenas ae961299a5 feat: add subscript & superscript formatting (#319)
* feat: add subscript & superscript formatting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* switch to enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-06 09:01:32 +02:00
samiucandsamiullahchattha c521766cb7 feat: add BoundingBox methods for overlap and union calculations (#311)
* feat(BoundingBox): add methods for overlap and union calculations

Signed-off-by: samiullahchattha <Sami.Ullah1@ibm.com>

* format files

Signed-off-by: samiullahchattha <Sami.Ullah1@ibm.com>

---------

Signed-off-by: samiullahchattha <Sami.Ullah1@ibm.com>
Co-authored-by: samiullahchattha <Sami.Ullah1@ibm.com>
2025-06-02 10:45:20 +02:00
Cesar Berrospi Ramis 4a174b5679 chore: fix deprecation warnings (#303)
* chore: fix deprecation warnings

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: disregard deprecated captions from hierarchical chunker in hybrid chunker

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: update poetry lock file

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-05-26 15:58:08 +02:00
Panos Vagenas 98c60bb65b perf: fix serialization performance (#249)
* perf: fix serialization performance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add missing leading HTML tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-11 15:37:52 +02:00
Peter W. J. StaarandPanos Vagenas 5d406008f2 feat: add HTML serializer (#232)
* iunitial attempt at HTML serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first version, to be tested thoroughly

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the new test

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rewrote carefully the export-to-html into new framework

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the inline list-items

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the inline code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* migrated the table html code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the picture HTML serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the html for Form and KeyValue

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first version of KeyValue serialisation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the key-value and form-region and added the GraphData serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* need to do some mypy work now

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* passed the mypy

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the get_excluded_refs function to obtain proper serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* enabled the captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed empty lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added initial split view and customised styles

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* cleaned up, now waiting for page-indices propagationg

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the styles and parameters with split_page

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* propagated the parameter split_page_view

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first fully working version

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests for html export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed dead code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test output

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rename parameter tag to class_name

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added serializers to table and picture

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed dead code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* various HTML serialization improvements (#242)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* added enum for different output styles

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-10 10:59:07 +02:00
Peter W. J. StaarandPanos Vagenas f2fe1c1bc6 feat: added initial delete and insert methods in DoclingDocument (#220)
* added intial delete and merge

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* work-in-progress for the delete option

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the delete_document_item

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* verified the model-validation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the delete item

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reproducible error in tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* before rebase with main

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the tests (2) and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the iterate_items to reduce code duplication

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* mapped path method to _split_ref_to_path

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* made get_parent_ref private method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with new public functions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the comments of the PR

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* factored out the internal tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the code and updated the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* update return types, extend unit test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add replace_item test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert unnecessary comments

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-04 14:34:25 +02:00
Panos Vagenas ff13a9385d feat: expose MD page break & DocTags minification (#213)
* feat: expose MD page break & DocTags minification

Also:
- aligned DocTags method naming
- fixed mutable default arguments

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* cover newly exposed params in testing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* drop DocTags-specific post-processing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-25 10:52:09 +01:00
Panos Vagenas 1f4d57ed18 feat: add DocTags serializers (#192)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-21 14:49:28 +01:00