Commit Graph
5 Commits
Author SHA1 Message Date
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
Cesar Berrospi Ramis afa5bd9e3a chore: document-level metadata serialization via body field (#551)
* feat: add 'meta' field to DoclingDocument at root level

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(metadata): reuse DoclingDocument objects with fixtures

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore(serializer): add document-level meta to the outline serialization

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: add helper function to compare or regenerate ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: regenerate (upgrade) ground truth files

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): set default indent to 2 spaces in outline serializer

To be aligned with other indented text serializations, the outline serializer takes 2 spaces
as default indent for serialization.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor(serializer): add line break after ref in outline md serialization

To render a line break after the reference element in the outline markdown serializer, add 2 trailing spaces.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: rollback adding 'meta' field to DoclingDocument

Revert the changes in commit 7e17c52 since DoclingDocument meta-information can
alreadby be achieved by leveraging the 'body' field, which is of type 'NodeItem'
and thus it has a 'meta' field.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-17 14:55:52 +01:00
Anish Raghavendra e00125c477 feat: handle wide table outliers with LineBasedTokenChunker (#536)
* feat: handle large prefixes by splitting into multiple chunks in LineBasedTokenChunker

Signed-off-by: z404 <anishr890@gmail.com>

* feat: add option to omit table headers on row overflow

Signed-off-by: z404 <anishr890@gmail.com>

* chore: add warning when prefix is omitted on overflow

Signed-off-by: z404 <anishr890@gmail.com>

* refactor: simplify prefix chunking logic by removing redundant method

Signed-off-by: z404 <anishr890@gmail.com>

* fix: ensure prefix appears as standalone chunk when all lines overflow

Signed-off-by: z404 <anishr890@gmail.com>

* docs: add usage examples for LineBasedTokenChunker with prefix handling

Signed-off-by: z404 <anishr890@gmail.com>

* docs: remove examples and describe parameters for LineBasedTokenChunker

Signed-off-by: z404 <anishr890@gmail.com>

* docs: simplify class docstring by removing parameter details

Signed-off-by: z404 <anishr890@gmail.com>

---------

Signed-off-by: z404 <anishr890@gmail.com>
2026-03-13 07:29:46 +01:00
Cesar Berrospi Ramis a661bb10cb fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens (#533)
* fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests: add a pytest fixture to test LineBasedTokenChunker efficiently

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: apply style conventions to test_line_chunker.py test module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-06 12:52:30 +01:00
ea359bcc63 feat: table aware chunking (#527)
* line_chunker

* split table to header and body

* duplicat table headers

* Revert "duplicat table headers"

This reverts commit 5d17bdacf2.

* Revert "split table to header and body"

This reverts commit 91b43f97e4.

* Revert "line_chunker"

This reverts commit 5cc61d93fb.

* line chunker

* split table to header and body

* duplicate table headers

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5cc61d93fb
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 91b43f97e4
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 5d17bdacf2
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: a50392e53c
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: e5894290d5
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 30c72a99be
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 510e949692
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 6c3a8f726e
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 0642a0701e

Signed-off-by: odelliab <odelliab@il.ibm.com>

* style changes

* pre-commit fixes

* expected output name change

* DCO Remediation Commit for odelliab <odelliab@il.ibm.com>

I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9b9ef09007
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: b3699e32cc
I, odelliab <odelliab@il.ibm.com>, hereby add my Signed-off-by to this commit: 9d393df546

Signed-off-by: odelliab <odelliab@il.ibm.com>

* Apply suggestion from @ceberam

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>

* address review comments

Signed-off-by: odelliab <odelliab@il.ibm.com>

* refactor: move get_default_tokenizer to huggingface module

Move the method get_default_tokenizer to module huggingface.py to avoid circular dependencies
and avoid an unconventional import in a method

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: odelliab <odelliab@il.ibm.com>
Signed-off-by: odelliab <91875866+odelliab@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-04 16:25:57 +01:00