Commit Graph
596 Commits
Author SHA1 Message Date
Peter W. J. Staar 76e14fdb97 feat: add latex serializer (#445)
* feat: add latex serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed spurious copyright

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the multiline

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-28 14:43:37 +01:00
github-actions[bot] e54524dd22 chore: bump version to 2.53.0 [skip ci] v2.53.0 2025-11-27 16:16:59 +00:00
Michele Dolfi edda70b2cf chore: use sample_doc fixture (#442)
use sample_doc fixture

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-11-27 15:59:41 +01:00
Panos Vagenas aa5c668050 feat(experimental): extend IDocTags tokens (#439)
feat: extend IDocTags tokens [WIP]

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-27 11:44:02 +01:00
Peter W. J. StaarandPanos Vagenas 92d60b0948 feat: added the Azure Document Intelligence (#395)
* added the Azure Document Intelligence

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the formatting and MyPy, working on the tests now

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added test document

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* replace all  with pydantic dataclasses

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* minor azure serializer refactoring

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-27 11:32:15 +01:00
Peter W. J. Staar 928aaed9ab chore: remove unnecesary copyright headers (#440)
* chore: remove unnecesary copyright headers

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted, still need to fix a test ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-27 11:17:23 +01:00
Ahmed NassarandAhmed Nassar 3b253c17f0 fix: chart title (#428)
* pic title fix

* DCO Remediation Commit for Ahmed Nassar <ahn@zurich.ibm.com>

I, Ahmed Nassar <ahn@zurich.ibm.com>, hereby add my Signed-off-by to this commit: da99c63b89

Signed-off-by: Ahmed Nassar <ahn@zurich.ibm.com>

---------

Signed-off-by: Ahmed Nassar <ahn@zurich.ibm.com>
Co-authored-by: Ahmed Nassar <ahn@zurich.ibm.com>
2025-11-27 11:01:39 +01:00
Panos Vagenas 8bdeaa7238 fix: robustify page filtering (#437)
* fix: robustify page filtering

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* cover validate_rules

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-26 13:53:15 +01:00
Cesar Berrospi Ramis 8feb09f1cf fix: markdown serialization of hyperlink with code (#434)
* chore: leverage fixtures in tests and add type annotations

Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(markdown): serialize hyperlink as code always with single backticks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-11-21 14:05:29 +01:00
github-actions[bot] 3fe8b5af52 chore: bump version to 2.52.0 [skip ci] v2.52.0 2025-11-20 14:15:55 +00:00
Panos Vagenas c9e5fb4a1c feat(experimental): add new DocTags serializer (#412)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-20 13:06:38 +01:00
c80b58369c feat: convert regions into TableData (#430)
* Introduction of regions.py - convert regions into TableData

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Cleaned up bbox helper functions, reusing more of the existing code

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Small fixes

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* refactored _bbox_intersection from regions.py into a method of BoundingBox.get_intersection_bbox

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* move region-based construction into `TableData` class

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-19 13:47:20 +01:00
Panos Vagenas 7bd274b84a chore: fix invalid test filenames (#431)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-18 11:31:29 +01:00
github-actions[bot] 6d48bf5307 chore: bump version to 2.51.1 [skip ci] v2.51.1 2025-11-14 13:30:10 +00:00
Panos Vagenas 57b019c58f chore: fix tree-sitter dependency constraints (#421)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-14 14:00:11 +01:00
Panos Vagenas bc0e96b9dc fix: improve meta migration (#422)
`setdefault` would not update anything if key was mapped to `None`.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-14 13:59:47 +01:00
Cesar Berrospi Ramis 56b3c42c61 fix: DoclingDocument model validator should deal with any raw input (#419)
* fix: DoclingDocument model validator should deal with any raw input

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: mention 3.14 as valid Python version for development

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-11-13 16:37:11 +01:00
github-actions[bot] d52bd37950 chore: bump version to 2.51.0 [skip ci] v2.51.0 2025-11-12 15:52:23 +00:00
Panos Vagenas 3d13b02756 fix: improve meta migration and warning handling (#417)
* fix: improve meta migration and warning handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove commented-out code

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 16:25:14 +01:00
Panos Vagenas 567d3ada57 fix: fix import handling of extra dependencies for chunking (#418)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 15:46:25 +01:00
3097645198 feat: add code chunking functionality (#398)
* initial code chunking for docling-core

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* include language detections, add code chunking into hierarchical chunker

* add serializer, internal marking of chunkers, typing

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* Update docling_core/transforms/chunker/hierarchical_chunker.py

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* run all pre-commit less pytest

* update test files for code ID

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* update uv.lock

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* revert to stricter treesitter versioning due to compatibility

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove language detection (to be run by client, i.e. docling)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align new dependency specs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address backticks, ABC, and supported languages feedback

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove Language class and reuse CodeLanguageLabel

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* refactoring and improvements

- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 10:28:54 +01:00
github-actions[bot] a54f6f075a chore: bump version to 2.50.1 [skip ci] v2.50.1 2025-11-04 13:19:32 +00:00
Peter W. J. Staar 09ef91c272 fix: add JSON to CodeLanguageLabel (#413)
* fix: add JSON to CodeLanguageLabel

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the docs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-04 14:06:47 +01:00
github-actions[bot] 9b9e8a7882 chore: bump version to 2.50.0 [skip ci] v2.50.0 2025-10-30 12:37:03 +00:00
Panos Vagenas 2ee3cacdd6 feat: add metadata model hierarchy (#408)
* feat: add metadata model hierarchy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add deprecation, add first migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend annotations migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update with feedback

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose main prediction

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* ideas on enforcing separation between standard and custom fields

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add custom field setter method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update Markdown serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert description, add include_non_meta, showcase custom serializer for summaries

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify customization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix reference exclusion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* eliminate serialization dupliation between meta & (legacy) annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove old file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix item used in get_parts for meta ser

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* serialize GroupItem meta prior to content, DocItem meta after content

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore ser order for all nodeitems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* move meta serialization into DocSerializer.serialize() to maintain seamless chunking integration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add allow- & block-lists for meta names, add std field name enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML serializer, document meta field names, rename SMILES field

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make TabularChartMetaField.title optional, expose new classes through __init__.py, add MetaUtils

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add DocTags serialization, revert smiles to smi to prevent confusion with plural

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-30 11:15:14 +01:00
Panos Vagenas a3feae0a1e feat: add split view & YAML support to CLI viewer (#407)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-10-21 20:21:20 +02:00
Michele Dolfi 1a023dad6f chore: fix discord (#406)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-17 17:08:41 +02:00
Maxim LysakandMaksym Lysak ada4068e7a feat: New picture classes for doctags (#404)
* New picture classes

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Updated to reflect actual data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* updated, harmonized picture class names

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* simplifying: pictogram is icon

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-10-17 16:58:09 +02:00
Michele Dolfi 9fa02d08e3 chore: publish release notes on Discord (#405)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-17 16:31:40 +02:00
github-actions[bot] 2ed4374c71 chore: bump version to 2.49.0 [skip ci] v2.49.0 2025-10-16 14:42:31 +00:00
Michele Dolfi 47b70d3c24 feat: python 3.14 compatibility (#403)
update deps for python 3.14 compatibility

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-16 15:58:29 +02:00
Maxim LysakandMaksym Lysak 2b519f2c59 chore: Silencing info logging on "delete_items" by moving it to debug level (#400)
moving log of deleting elements from info to debug

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-10-07 14:37:33 +02:00
Erik abfa81f065 feat: rendering of checkboxes in markdown (#381)
* added support for rendering checkboxes

* DCO Remediation Commit for Erik <erik@apption.com>

Signed-off-by: Erik <erik@apption.com>

* DCO Remediation Commit for Erik <erik@apption.com>

I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 7af470451b

Signed-off-by: Erik <erik@apption.com>

* added code suggestions for better checkboxes

* updated test data

* fixed formatting

* DCO Remediation Commit for Erik <erik@apption.com>

I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: fc7a57e1f7
I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 7ea737ecd4
I, Erik <erik@apption.com>, hereby add my Signed-off-by to this commit: 5c431b52c8

Signed-off-by: Erik <erik@apption.com>

---------

Signed-off-by: Erik <erik@apption.com>
2025-10-01 11:16:31 +02:00
github-actions[bot] c52605dded chore: bump version to 2.48.4 [skip ci] v2.48.4 2025-10-01 09:09:37 +00:00
Panos Vagenas 3e8d628eee fix: switch to safe YAML loader (#396)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-01 09:43:42 +02:00
github-actions[bot] ec0978652f chore: bump version to 2.48.3 [skip ci] v2.48.3 2025-09-29 08:25:51 +00:00
Michele Dolfi 26d21e3b52 fix: pin larger range of typer (#394)
pin larger range of typer and update lock

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-09-24 05:05:06 +02:00
github-actions[bot] e234480b58 chore: bump version to 2.48.2 [skip ci] v2.48.2 2025-09-22 08:38:56 +00:00
Cesar Berrospi Ramis dd6ebc3440 fix: expose escape_html param to DoclingDocument md serialization (#388)
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2025-09-22 10:14:52 +02:00
github-actions[bot] 12292d7317 chore: bump version to 2.48.1 [skip ci] v2.48.1 2025-09-11 12:32:45 +00:00
Panos Vagenas 9df72083c0 fix(markdown): fix single-row table serialization (#385)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-10 16:11:34 +02:00
github-actions[bot] cf73d256ef chore: bump version to 2.48.0 [skip ci] v2.48.0 2025-09-09 14:39:15 +00:00
Maxim LysakandMaksym Lysak b13267f18b feat: Introduction of fillable TableCell (#384)
* New properties for TableCell, corner_header and fillable

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Updated docs

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* bumped docling document version to 1.7.0

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-09-09 15:48:42 +02:00
Viroshaan Uthayamoorthy (KLP) 0512f4473d fix: add doc param to all export_to_dataframe() calls (#380)
* fix: add 'doc' to all export_to_dataframe calls

* DCO Remediation Commit for Viroshaan <viroshaan.uthayamoorthy@klp.no>

I, Viroshaan <viroshaan.uthayamoorthy@klp.no>, hereby add my Signed-off-by to this commit: 10317f8db7

Signed-off-by: Viroshaan <viroshaan.uthayamoorthy@klp.no>

---------

Signed-off-by: Viroshaan <viroshaan.uthayamoorthy@klp.no>
2025-09-09 14:42:57 +02:00
Panos Vagenas 2dc57c1769 fix: fix handling of generic groups in rich table cells (#383)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-09 09:55:55 +02:00
Panos Vagenas b60ac192ef feat: add support for heading with inline in HTML & DocTags (#379)
* test: refactor serialization tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* further refactor serialization tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* rename inline test file for markdown

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify Markdown handling of inline in heading / list item

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix input test data & make it more interesting

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML & DocTags support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize and refactor HTML ser case

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* generalize MD serializer too

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-03 16:56:58 +02:00
github-actions[bot] c0b0c4cfd8 chore: bump version to 2.47.0 [skip ci] v2.47.0 2025-09-02 14:11:03 +00:00
Panos Vagenas 9dc526de53 feat: add page filtering to DoclingDocument (#378)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-09-02 15:56:35 +02:00
github-actions[bot] d12621d3de chore: bump version to 2.46.0 [skip ci] v2.46.0 2025-09-01 09:30:18 +00:00
ta0a2000tandPanos Vagenas 3f25267d56 fix: fix text dir attribute (#281)
fix: fix text dir attribute

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-08-29 10:13:25 +02:00