Cesar Berrospi Ramis and hardik-kansal
7775df1176
fix(chunker): preserve table structure across chunks in markdown ( #678 )
...
* fix(chunker): preserve markdown table structure when splitting across chunks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* test: extract shared fixtures and serializer providers in test_hybrid_chunker
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* docs: replace inline comments with docstrings in segment and get_header_and_body_lines
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* docs(chunker): fix misleading docstrings in LineBasedTokenChunker
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* chore: normalize import calls from DoclingDocument
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* fix(chunker): correctly identify table preamble when caption starts with '|'
Co-Authored-By: hardik-kansal <hkhardik4321@gmail.com >
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
Co-authored-by: hardik-kansal <hkhardik4321@gmail.com >
2026-07-14 10:50:03 +02:00
Peter W. J. Staar
014948b0e8
feat: updated the HTML serialization ( #609 )
...
* updated the HTML serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
2026-05-12 16:49:08 +02:00
Cesar Berrospi Ramis
6cbdee9626
fix: prevent numeric precision loss in Markdown table serialization ( #588 )
...
* fix: preserve numeric precision in markdown serialization
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* test: apply style conventions to test_serialization.py module
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
2026-04-15 17:16:57 +02:00
f2a61868d4
feat: DocChunk expansion ( #549 )
...
* line_chunker
* split table to header and body
* duplicat table headers
* Revert "duplicat table headers"
This reverts commit 5d17bdacf2 .
* Revert "split table to header and body"
This reverts commit 91b43f97e4 .
* Revert "line_chunker"
This reverts commit 5cc61d93fb .
* chunk expansion + test
Signed-off-by: odelliab <odelliab@il.ibm.com >
* small fixes
Signed-off-by: odelliab <odelliab@il.ibm.com >
* small fixes
Signed-off-by: odelliab <odelliab@il.ibm.com >
* remove unnecessary code
Signed-off-by: odelliab <odelliab@il.ibm.com >
* DCO Remediation Commit for odelliab <odelliab@il.ibm.com >
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <91875866+odelliab@users.noreply.github.com >, hereby add my Signed-off-by to this commit: 6aa0019fe0
Signed-off-by: odelliab <odelliab@il.ibm.com >
* change names
Signed-off-by: odelliab <odelliab@il.ibm.com >
* remove some tests
Signed-off-by: odelliab <odelliab@il.ibm.com >
* address review comments
Signed-off-by: odelliab <odelliab@il.ibm.com >
* Apply suggestions from code review
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
* consolidate tests
Signed-off-by: odelliab <odelliab@il.ibm.com >
* fix failing test
Signed-off-by: odelliab <odelliab@il.ibm.com >
* test: regenerate ground truth for hybrid chunker test
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* refactor: extract chunk expansion into TreeChunkExpander and PageChunkExpander
- Create chunk_expander.py with TreeChunkExpander and PageChunkExpander classes
- Remove DocChunk expansion methods
- Improve docstrings
- Refactor tests: test_doc_chunk_expansion.py → test_chunk_expander.py
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
---------
Signed-off-by: odelliab <odelliab@il.ibm.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
2026-04-15 17:16:28 +02:00
b435090fdf
feat: split html table to headers and body ( #532 )
...
* line_chunker
* split table to header and body
* duplicat table headers
* Revert "duplicat table headers"
This reverts commit 5d17bdacf2 .
* Revert "split table to header and body"
This reverts commit 91b43f97e4 .
* Revert "line_chunker"
This reverts commit 5cc61d93fb .
* line chunker
* split table to header and body
* duplicate table headers
* DCO Remediation Commit for odelliab <odelliab@il.ibm.com >
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 0642a0701e
Signed-off-by: odelliab <odelliab@il.ibm.com >
* style changes
* pre-commit fixes
* expected output name change
* DCO Remediation Commit for odelliab <odelliab@il.ibm.com >
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 9d393df546
Signed-off-by: odelliab <odelliab@il.ibm.com >
* Apply suggestion from @ceberam
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
* Apply suggestions from code review
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
* address review comments
Signed-off-by: odelliab <odelliab@il.ibm.com >
* refactor: move get_default_tokenizer to huggingface module
Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* remove unnecessary lines
Signed-off-by: odelliab <odelliab@il.ibm.com >
* split html tables
Signed-off-by: odelliab <odelliab@il.ibm.com >
* add bs4 dependency
Signed-off-by: odelliab <odelliab@il.ibm.com >
* Apply suggestions from code review
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
* address review comments
Signed-off-by: odelliab <odelliab@il.ibm.com >
* typo
Signed-off-by: odelliab <odelliab@il.ibm.com >
* add beautifulsoup tests
Signed-off-by: odelliab <odelliab@il.ibm.com >
* refactor(serializer): drop dependency 'beautifulsoup4' in HTML serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* test(serializer): refactor tests for HTML serializer
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
---------
Signed-off-by: odelliab <odelliab@il.ibm.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
2026-03-13 10:40:05 +01:00
Michele Dolfi
4eb0d20d04
feat: Loosen dependency version constraints ( #534 )
...
* update deps
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* fix code to support all versions and regen tests
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
---------
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
2026-03-09 05:27:48 +01:00
ea359bcc63
feat: table aware chunking ( #527 )
...
* line_chunker
* split table to header and body
* duplicat table headers
* Revert "duplicat table headers"
This reverts commit 5d17bdacf2 .
* Revert "split table to header and body"
This reverts commit 91b43f97e4 .
* Revert "line_chunker"
This reverts commit 5cc61d93fb .
* line chunker
* split table to header and body
* duplicate table headers
* DCO Remediation Commit for odelliab <odelliab@il.ibm.com >
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 0642a0701e
Signed-off-by: odelliab <odelliab@il.ibm.com >
* style changes
* pre-commit fixes
* expected output name change
* DCO Remediation Commit for odelliab <odelliab@il.ibm.com >
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com >, hereby add my Signed-off-by to this commit: 9d393df546
Signed-off-by: odelliab <odelliab@il.ibm.com >
* Apply suggestion from @ceberam
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
* Apply suggestions from code review
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
* address review comments
Signed-off-by: odelliab <odelliab@il.ibm.com >
* refactor: move get_default_tokenizer to huggingface module
Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
---------
Signed-off-by: odelliab <odelliab@il.ibm.com >
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com >
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
2026-03-04 16:25:57 +01:00
Matvei Smirnov and Cesar Berrospi Ramis
c566268e0a
fix: rich table triplet serialization ( #425 )
...
* fix: rich table triplet serialization
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com >
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
* refactor: remove kwargs from 'export_to_dataframe' signature
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
---------
Signed-off-by: Vdaleke <vdalekesmirnov@gmail.com >
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com >
2026-02-26 11:39:34 +01:00
Panos Vagenas
8b082e9b9d
feat: enable heading-only chunks for empty-section headings ( #461 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-12-18 14:35:15 +01:00
3097645198
feat: add code chunking functionality ( #398 )
...
* initial code chunking for docling-core
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com >
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* include language detections, add code chunking into hierarchical chunker
* add serializer, internal marking of chunkers, typing
* Update pyproject.toml
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Signed-off-by: Bridget <bridget.mcginn@ibm.com >
* Update docling_core/transforms/chunker/hierarchical_chunker.py
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Signed-off-by: Bridget <bridget.mcginn@ibm.com >
* run all pre-commit less pytest
* update test files for code ID
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com >
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* update uv.lock
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* revert to stricter treesitter versioning due to compatibility
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com >
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* remove language detection (to be run by client, i.e. docling)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* align new dependency specs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* address backticks, ABC, and supported languages feedback
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* remove Language class and reuse CodeLanguageLabel
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com >
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com >, hereby add my Signed-off-by to this commit: 025aea31ab
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
* refactoring and improvements
- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com >
Signed-off-by: Bridget <bridget.mcginn@ibm.com >
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com >
2025-11-12 10:28:54 +01:00
Panos Vagenas
2ee3cacdd6
feat: add metadata model hierarchy ( #408 )
...
* feat: add metadata model hierarchy
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add deprecation, add first migration
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* extend annotations migration
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* update with feedback
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* expose main prediction
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* ideas on enforcing separation between standard and custom fields
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add custom field setter method
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* update Markdown serialization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* revert description, add include_non_meta, showcase custom serializer for summaries
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* simplify customization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* fix reference exclusion
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* eliminate serialization dupliation between meta & (legacy) annotations
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* remove old file
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* fix item used in get_parts for meta ser
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* serialize GroupItem meta prior to content, DocItem meta after content
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* restore ser order for all nodeitems
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* move meta serialization into DocSerializer.serialize() to maintain seamless chunking integration
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add allow- & block-lists for meta names, add std field name enum
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add HTML serializer, document meta field names, rename SMILES field
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* bump DoclingDocument version
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* make TabularChartMetaField.title optional, expose new classes through __init__.py, add MetaUtils
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add DocTags serialization, revert smiles to smi to prevent confusion with plural
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-10-30 11:15:14 +01:00
Panos Vagenas
5c99722b38
fix(HybridChunker): improve long heading handling ( #333 )
...
* fix(HybridChunker): improve long heading handling
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* fix warning wording
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-06-17 12:21:51 +02:00
Panos Vagenas
87b72d6537
fix(HybridChunker): refine max_tokens auto-detection ( #306 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-05-27 15:38:49 +02:00
Panos Vagenas and Michele Dolfi
f067c51c48
feat: add annotations in MD & HTML serialization ( #295 )
...
* feat: include annotations in MD & HTML serialization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* (HTML) move annotations into figcaptions
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add explicit beginning/end markers, fix case of excluded refs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* improve annotation marking, extend tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* wrap captions (#305 )
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* revert temp test changes
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com >
2025-05-27 10:49:13 +02:00
Guillermo and Panos Vagenas
763e1364ff
feat: Add tiktoken tokenizers support to HybridChunker ( #240 )
...
* feat: Add tiktoken tokenizers support to HybridChunker
Signed-off-by: ruizguille <guillermo@codeawake.com >
* separate OpenAI tokenizer
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: ruizguille <guillermo@codeawake.com >
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-18 07:43:20 +02:00
Panos Vagenas
5e4c0fd51f
feat: integrate serialization API into chunkers ( #221 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-08 14:04:40 +02:00
Christoph Auer and Panos Vagenas
786f0c6833
feat: Add ContentLayer attribute to designate items to body or furniture ( #148 )
...
* feat: Add ContentLayer attribute to designate items to body or furniture
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* introduce safer data gen mechanism, update chunking test data
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* Do not make test rely on order in yaml
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* chore: format fixes
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* fix: legacy_to_docling_doc must use content_layer
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Add content_layer in iterate_items
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Bump format version, add model_validator for old page_header,page_footer in body
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* fix: Change to before model_validator
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Update tests
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Address review comments
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2025-02-10 10:58:41 +01:00
Panos Vagenas
b741eeaab4
fix: fix hybrid chunker token constraint ( #131 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2025-01-17 17:22:26 +01:00
Panos Vagenas
b546c0a50d
fix: set origin when merging chunks ( #109 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-12-13 09:05:56 +01:00
Panos Vagenas
2591c70c66
feat: add default tokenizer to HybridChunker ( #107 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-12-12 16:51:08 +01:00
628ab679cb
feat: add hybrid chunker ( #68 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Bill Murdock <bmurdock@redhat.com >
Co-authored-by: Ben Rood <bash99@gmail.com >
2024-12-06 14:56:10 +01:00
Panos Vagenas
bfeb2db24b
fix: include titles to chunk heading metadata ( #62 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-11-01 13:41:16 +01:00
Panos Vagenas
d09fe7ed44
feat: extend chunk meta with schema, version, origin ( #49 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-22 11:08:04 +02:00
03df97fa73
feat!: Expose DoclingDocument as main type, move old typing to legacy ( #41 )
...
* Fix area method of BoundingBox
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* add image placeholder
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* enable picture label
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* refactor captions and markdown
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* add logic to skip repeated caption
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* use DocItemLabel
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Extend default export labels, add convenience mehtods
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Introduce ListItem API, with marker and enumerated properties
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* add classification and description in PictureData
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* add molecule picture data
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Fixes for DoclingDocument and aligned methods on legacy doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* add advanced picture data content
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Many markdown export fixes, renaming BaseTableData
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Rename module paths doc->legacy_doc, experimental->doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* feat: imageref with pil_image
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Small fixes
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* docs: remove documentation in markdown to support python 3.13 (#43 )
Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
* Fix TableCell model validator
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* store list of classes in classification
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Fixes for DocumentOrigin mimetype validation
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* introduce picturedata as list of annotations
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* feat: adapt hierarchical chunker to v2 DoclingDocument
[skip-ci]
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* feat: add table support in chunker, incl. captions
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* use Field constraints instead of conlist, refactor chunking types
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* revert unnecessary doc module change
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* align test data with upstream changes
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* Update __init__.py on docling_core.types.doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Remove DescriptionItem
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com >
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-16 15:53:44 +02:00
Panos Vagenas
b5592ad747
fix: align chunk ref format with one used in Document ( #37 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-07 09:07:08 +02:00
Panos Vagenas
4bde51528d
feat: support heading as chunk metadata ( #36 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-09-30 11:08:09 +02:00
Panos Vagenas
9698d30288
feat: add hierarchical chunker ( #18 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-09-11 12:57:56 +02:00