Commit Graph
27 Commits
Author SHA1 Message Date
Cesar Berrospi Ramisandhardik-kansal 7775df1176 fix(chunker): preserve table structure across chunks in markdown (#678)
* fix(chunker): preserve markdown table structure when splitting across chunks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: extract shared fixtures and serializer providers in test_hybrid_chunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: replace inline comments with docstrings in segment and get_header_and_body_lines

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs(chunker): fix misleading docstrings in LineBasedTokenChunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: normalize import calls from DoclingDocument

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): correctly identify table preamble when caption starts with '|'

Co-Authored-By: hardik-kansal <hkhardik4321@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: hardik-kansal <hkhardik4321@gmail.com>
2026-07-14 10:50:03 +02:00
Peter W. J. Staar 014948b0e8 feat: updated the HTML serialization (#609)
* updated the HTML serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-05-12 16:49:08 +02:00
Cesar Berrospi Ramis 6cbdee9626 fix: prevent numeric precision loss in Markdown table serialization (#588)
* fix: preserve numeric precision in markdown serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: apply style conventions to test_serialization.py module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-04-15 17:16:57 +02:00
f2a61868d4 feat: DocChunk expansion (#549)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* chunk expansion + test

Signed-off-by: odelliab <odelliab@il.ibm.com>

* small fixes

Signed-off-by: odelliab <odelliab@il.ibm.com>

* small fixes

Signed-off-by: odelliab <odelliab@il.ibm.com>

* remove unnecessary code

Signed-off-by: odelliab <odelliab@il.ibm.com>

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <91875866+odelliab@users.noreply.github.com>, hereby add my Signed-off-by to this commit: 6aa0019fe0

Signed-off-by: odelliab <odelliab@il.ibm.com>

* change names

Signed-off-by: odelliab <odelliab@il.ibm.com>

* remove some tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* consolidate tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* fix failing test

Signed-off-by: odelliab <odelliab@il.ibm.com>

* test: regenerate ground truth for hybrid chunker test

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: extract chunk expansion into TreeChunkExpander and PageChunkExpander

- Create chunk_expander.py with TreeChunkExpander and PageChunkExpander classes
- Remove DocChunk expansion methods
- Improve docstrings
- Refactor tests: test_doc_chunk_expansion.py → test_chunk_expander.py

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-04-15 17:16:28 +02:00
b435090fdf feat: split html table to headers and body (#532)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* remove unnecessary lines

Signed-off-by: odelliab <odelliab@il.ibm.com>

* split html tables

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add bs4 dependency

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* typo

Signed-off-by: odelliab <odelliab@il.ibm.com>

* add beautifulsoup tests

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor(serializer): drop dependency 'beautifulsoup4' in HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(serializer): refactor tests for HTML serializer

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-13 10:40:05 +01:00
Michele Dolfi 4eb0d20d04 feat: Loosen dependency version constraints (#534)
* update deps

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix code to support all versions and regen tests

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-03-09 05:27:48 +01:00
ea359bcc63 feat: table aware chunking (#527)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-04 16:25:57 +01:00
Matvei SmirnovandCesar Berrospi Ramis c566268e0a fix: rich table triplet serialization (#425)
* fix: rich table triplet serialization

Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: remove kwargs from 'export_to_dataframe' signature

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-02-26 11:39:34 +01:00
Panos Vagenas 8b082e9b9d feat: enable heading-only chunks for empty-section headings (#461)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-12-18 14:35:15 +01:00
3097645198 feat: add code chunking functionality (#398)
* initial code chunking for docling-core

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* include language detections, add code chunking into hierarchical chunker

* add serializer, internal marking of chunkers, typing

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* Update docling_core/transforms/chunker/hierarchical_chunker.py

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* run all pre-commit less pytest

* update test files for code ID

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* update uv.lock

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* revert to stricter treesitter versioning due to compatibility

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove language detection (to be run by client, i.e. docling)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align new dependency specs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address backticks, ABC, and supported languages feedback

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove Language class and reuse CodeLanguageLabel

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* refactoring and improvements

- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 10:28:54 +01:00
Panos Vagenas 2ee3cacdd6 feat: add metadata model hierarchy (#408)
* feat: add metadata model hierarchy

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add deprecation, add first migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend annotations migration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update with feedback

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expose main prediction

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* ideas on enforcing separation between standard and custom fields

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add custom field setter method

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update Markdown serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert description, add include_non_meta, showcase custom serializer for summaries

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* simplify customization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix reference exclusion

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* eliminate serialization dupliation between meta & (legacy) annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* remove old file

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix item used in get_parts for meta ser

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* serialize GroupItem meta prior to content, DocItem meta after content

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* restore ser order for all nodeitems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* move meta serialization into DocSerializer.serialize() to maintain seamless chunking integration

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add allow- & block-lists for meta names, add std field name enum

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add HTML serializer, document meta field names, rename SMILES field

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump DoclingDocument version

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make TabularChartMetaField.title optional, expose new classes through __init__.py, add MetaUtils

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add DocTags serialization, revert smiles to smi to prevent confusion with plural

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-10-30 11:15:14 +01:00
Panos Vagenas 5c99722b38 fix(HybridChunker): improve long heading handling (#333)
* fix(HybridChunker): improve long heading handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix warning wording

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-06-17 12:21:51 +02:00
Panos Vagenas 87b72d6537 fix(HybridChunker): refine max_tokens auto-detection (#306)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-27 15:38:49 +02:00
Panos VagenasandMichele Dolfi f067c51c48 feat: add annotations in MD & HTML serialization (#295)
* feat: include annotations in MD & HTML serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* (HTML) move annotations into figcaptions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add explicit beginning/end markers, fix case of excluded refs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve annotation marking, extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* wrap captions (#305)

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* revert temp test changes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2025-05-27 10:49:13 +02:00
GuillermoandPanos Vagenas 763e1364ff feat: Add tiktoken tokenizers support to HybridChunker (#240)
* feat: Add tiktoken tokenizers support to HybridChunker

Signed-off-by: ruizguille <guillermo@codeawake.com>

* separate OpenAI tokenizer

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: ruizguille <guillermo@codeawake.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-18 07:43:20 +02:00
Panos Vagenas 5e4c0fd51f feat: integrate serialization API into chunkers (#221)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-08 14:04:40 +02:00
Christoph AuerandPanos Vagenas 786f0c6833 feat: Add ContentLayer attribute to designate items to body or furniture (#148)
* feat: Add ContentLayer attribute to designate items to body or furniture

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce safer data gen mechanism, update chunking test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Do not make test rely on order in yaml

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore: format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: legacy_to_docling_doc must use content_layer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add content_layer in iterate_items

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Bump format version, add model_validator for old page_header,page_footer in body

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: Change to before model_validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Address review comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-02-10 10:58:41 +01:00
Panos Vagenas b741eeaab4 fix: fix hybrid chunker token constraint (#131)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-01-17 17:22:26 +01:00
Panos Vagenas b546c0a50d fix: set origin when merging chunks (#109)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-13 09:05:56 +01:00
Panos Vagenas 2591c70c66 feat: add default tokenizer to HybridChunker (#107)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-12 16:51:08 +01:00
628ab679cb feat: add hybrid chunker (#68)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Bill Murdock <bmurdock@redhat.com>
Co-authored-by: Ben Rood <bash99@gmail.com>
2024-12-06 14:56:10 +01:00
Panos Vagenas bfeb2db24b fix: include titles to chunk heading metadata (#62)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-11-01 13:41:16 +01:00
Panos Vagenas d09fe7ed44 feat: extend chunk meta with schema, version, origin (#49)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-22 11:08:04 +02:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Panos Vagenas b5592ad747 fix: align chunk ref format with one used in Document (#37)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-07 09:07:08 +02:00
Panos Vagenas 4bde51528d feat: support heading as chunk metadata (#36)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-30 11:08:09 +02:00
Panos Vagenas 9698d30288 feat: add hierarchical chunker (#18)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-11 12:57:56 +02:00