 Panos Vagenas
|
87b72d6537
|
fix(HybridChunker): refine max_tokens auto-detection (#306)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
|
2025-05-27 15:38:49 +02:00 |
|
 Panos VagenasandMichele Dolfi
|
f067c51c48
|
feat: add annotations in MD & HTML serialization (#295)
* feat: include annotations in MD & HTML serialization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* (HTML) move annotations into figcaptions
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* add explicit beginning/end markers, fix case of excluded refs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* improve annotation marking, extend tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* wrap captions (#305)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* revert temp test changes
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
|
2025-05-27 10:49:13 +02:00 |
|
  GuillermoandPanos Vagenas
|
763e1364ff
|
feat: Add tiktoken tokenizers support to HybridChunker (#240)
* feat: Add tiktoken tokenizers support to HybridChunker
Signed-off-by: ruizguille <guillermo@codeawake.com>
* separate OpenAI tokenizer
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: ruizguille <guillermo@codeawake.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
|
2025-04-18 07:43:20 +02:00 |
|
|
Panos Vagenas
|
5e4c0fd51f
|
feat: integrate serialization API into chunkers (#221)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
|
2025-04-08 14:04:40 +02:00 |
|
 Christoph AuerandPanos Vagenas
|
786f0c6833
|
feat: Add ContentLayer attribute to designate items to body or furniture (#148)
* feat: Add ContentLayer attribute to designate items to body or furniture
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* introduce safer data gen mechanism, update chunking test data
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
* Do not make test rely on order in yaml
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* chore: format fixes
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* fix: legacy_to_docling_doc must use content_layer
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Add content_layer in iterate_items
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Bump format version, add model_validator for old page_header,page_footer in body
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* fix: Change to before model_validator
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Update tests
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Address review comments
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2025-02-10 10:58:41 +01:00 |
|
|
Panos Vagenas
|
b741eeaab4
|
fix: fix hybrid chunker token constraint (#131)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2025-01-17 17:22:26 +01:00 |
|
|
Panos Vagenas
|
b546c0a50d
|
fix: set origin when merging chunks (#109)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-12-13 09:05:56 +01:00 |
|
|
Panos Vagenas
|
2591c70c66
|
feat: add default tokenizer to HybridChunker (#107)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-12-12 16:51:08 +01:00 |
|
 
|
628ab679cb
|
feat: add hybrid chunker (#68)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Bill Murdock <bmurdock@redhat.com>
Co-authored-by: Ben Rood <bash99@gmail.com>
|
2024-12-06 14:56:10 +01:00 |
|
|
Panos Vagenas
|
bfeb2db24b
|
fix: include titles to chunk heading metadata (#62)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-11-01 13:41:16 +01:00 |
|
|
Panos Vagenas
|
d09fe7ed44
|
feat: extend chunk meta with schema, version, origin (#49)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-10-22 11:08:04 +02:00 |
|
|
03df97fa73
|
feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* add image placeholder
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* enable picture label
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* refactor captions and markdown
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* add logic to skip repeated caption
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* use DocItemLabel
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* Extend default export labels, add convenience mehtods
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Introduce ListItem API, with marker and enumerated properties
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* add classification and description in PictureData
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* add molecule picture data
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* Fixes for DoclingDocument and aligned methods on legacy doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* add advanced picture data content
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* Many markdown export fixes, renaming BaseTableData
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Rename module paths doc->legacy_doc, experimental->doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* feat: imageref with pil_image
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* Small fixes
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* docs: remove documentation in markdown to support python 3.13 (#43)
Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
* Fix TableCell model validator
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* store list of classes in classification
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* Fixes for DocumentOrigin mimetype validation
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* introduce picturedata as list of annotations
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
* feat: adapt hierarchical chunker to v2 DoclingDocument
[skip-ci]
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
* feat: add table support in chunker, incl. captions
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
* use Field constraints instead of conlist, refactor chunking types
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
* revert unnecessary doc module change
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
* align test data with upstream changes
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
* Update __init__.py on docling_core.types.doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Remove DescriptionItem
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-10-16 15:53:44 +02:00 |
|
|
Panos Vagenas
|
b5592ad747
|
fix: align chunk ref format with one used in Document (#37)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-10-07 09:07:08 +02:00 |
|
|
Panos Vagenas
|
4bde51528d
|
feat: support heading as chunk metadata (#36)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-09-30 11:08:09 +02:00 |
|
|
Panos Vagenas
|
9698d30288
|
feat: add hierarchical chunker (#18)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
|
2024-09-11 12:57:56 +02:00 |
|