Panos Vagenas and GitHub
d8a5256b2c
feat: add table annotations ( #304 )
...
* feat: add table annotations
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* refactor annotation types
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* expand to HTML
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* introduce annotation serializer
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* Update dummy_doc.yaml
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2025-06-05 15:38:46 +02:00
Panos Vagenas and GitHub
87b72d6537
fix(HybridChunker): refine max_tokens auto-detection ( #306 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-05-27 15:38:49 +02:00
f067c51c48
feat: add annotations in MD & HTML serialization ( #295 )
...
* feat: include annotations in MD & HTML serialization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* (HTML) move annotations into figcaptions
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add explicit beginning/end markers, fix case of excluded refs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* improve annotation marking, extend tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* wrap captions (#305 )
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* revert temp test changes
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com >
2025-05-27 10:49:13 +02:00
Cesar Berrospi Ramis and GitHub
4a174b5679
chore: fix deprecation warnings ( #303 )
...
* chore: fix deprecation warnings
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
* chore: disregard deprecated captions from hierarchical chunker in hybrid chunker
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
* chore: update poetry lock file
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
---------
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
2025-05-26 15:58:08 +02:00
Panos Vagenas and GitHub
ebc356a787
fix: fix hybrid chunker legacy patching ( #300 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-05-22 13:34:50 +02:00
Panos Vagenas and GitHub
c49a50e76b
fix(markdown): fix case of empty page break string ( #298 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-05-20 10:51:24 +02:00
Panos Vagenas and GitHub
6a7eb537eb
feat: provide visualizer option in HTML split view ( #294 )
...
* feat: provide visualizer option in HTML split view
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* loosen test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-05-16 11:22:12 +02:00
Peter W. J. Staar and GitHub
7f83f1ce84
feat: add image group serialization in html ( #284 )
...
* feat: add-image group serialization in html
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* reformatted the code
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
2025-05-06 13:59:47 +02:00
Panos Vagenas and GitHub
8677d6e9c6
fix: fix multi-provenance item visualization ( #277 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-30 15:55:27 +02:00
Panos Vagenas and GitHub
d05fe08546
feat: promote serializers to stable API ( #276 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-30 11:26:39 +02:00
Peter W. J. Staar and GitHub
a947440545
fix: visualization of document pages without items ( #271 )
...
* fix: visualization of document pages without items
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* reformatted the code
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* fixed MyPy
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* fixed MyPy (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* fixed MyPy (3)
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* fixed flake
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* fixed flake (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
2025-04-25 11:41:08 +02:00
Eugene and GitHub
c30ada64d6
chore: Types fixes ( #267 )
...
* docs: Recommend installing all groups for mypy to work properly
Signed-off-by: Eugene <fogaprod@gmail.com >
* chore: Fix some incomplete type defs
Signed-off-by: Eugene <fogaprod@gmail.com >
---------
Signed-off-by: Eugene <fogaprod@gmail.com >
2025-04-22 10:29:20 +02:00
763e1364ff
feat: Add tiktoken tokenizers support to HybridChunker ( #240 )
...
* feat: Add tiktoken tokenizers support to HybridChunker
Signed-off-by: ruizguille <guillermo@codeawake.com >
* separate OpenAI tokenizer
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: ruizguille <guillermo@codeawake.com >
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-18 07:43:20 +02:00
Panos Vagenas and GitHub
a258d525e1
feat: add visualizers ( #263 )
...
* feat: add visualizers
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* make visualizers composable
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* use BoundingRectangle instead of BoundingBox
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* enforce top-left coordinates
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* narrow down test data to first 3 pages
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
* add file deletions
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-17 11:43:45 +02:00
Panos Vagenas and GitHub
23036e17fa
feat: add serializer provider to chunkers ( #239 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-09 12:45:44 +02:00
Panos Vagenas and GitHub
5e4c0fd51f
feat: integrate serialization API into chunkers ( #221 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-04-08 14:04:40 +02:00
Panos Vagenas and GitHub
419252c39b
fix(HybridChunker): remove max_length from tokenization ( #178 )
...
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com >
2025-02-28 13:45:10 +01:00
916323fb55
feat: Redefine CodeItem as floating object with captions ( #160 )
...
* updated the CodeItem with captions
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
* use FloatItem and add captions to add_code interface
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com >
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com >
2025-02-14 11:34:40 +01:00
f10b228cdb
chore: annotate kwargs across BaseChunker class hierarchy ( #116 )
...
* chore: add missing type hint for HybridChunker.chunk
Signed-off-by: Eugene <fogaprod@gmail.com >
* chore: apply typing to whole hierarchy
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
---------
Signed-off-by: Eugene <fogaprod@gmail.com >
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2025-01-21 09:47:55 +01:00
Panos Vagenas and GitHub
b741eeaab4
fix: fix hybrid chunker token constraint ( #131 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2025-01-17 17:22:26 +01:00
Panos Vagenas and GitHub
b546c0a50d
fix: set origin when merging chunks ( #109 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-12-13 09:05:56 +01:00
Panos Vagenas and GitHub
2591c70c66
feat: add default tokenizer to HybridChunker ( #107 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-12-12 16:51:08 +01:00
628ab679cb
feat: add hybrid chunker ( #68 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Bill Murdock <bmurdock@redhat.com >
Co-authored-by: Ben Rood <bash99@gmail.com >
2024-12-06 14:56:10 +01:00
Panos Vagenas and GitHub
bfeb2db24b
fix: include titles to chunk heading metadata ( #62 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-11-01 13:41:16 +01:00
Panos Vagenas and GitHub
b5d07b2fa5
fix: fix non-string table cell handling in chunker ( #58 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-26 13:01:21 +02:00
Panos Vagenas and GitHub
d09fe7ed44
feat: extend chunk meta with schema, version, origin ( #49 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-22 11:08:04 +02:00
03df97fa73
feat!: Expose DoclingDocument as main type, move old typing to legacy ( #41 )
...
* Fix area method of BoundingBox
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* add image placeholder
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* enable picture label
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* refactor captions and markdown
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* add logic to skip repeated caption
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* use DocItemLabel
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Extend default export labels, add convenience mehtods
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Introduce ListItem API, with marker and enumerated properties
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* add classification and description in PictureData
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* add molecule picture data
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Fixes for DoclingDocument and aligned methods on legacy doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* add advanced picture data content
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Many markdown export fixes, renaming BaseTableData
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Rename module paths doc->legacy_doc, experimental->doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* feat: imageref with pil_image
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Small fixes
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* docs: remove documentation in markdown to support python 3.13 (#43 )
Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
* Fix TableCell model validator
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* store list of classes in classification
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* Fixes for DocumentOrigin mimetype validation
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* introduce picturedata as list of annotations
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
* feat: adapt hierarchical chunker to v2 DoclingDocument
[skip-ci]
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* feat: add table support in chunker, incl. captions
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* use Field constraints instead of conlist, refactor chunking types
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* revert unnecessary doc module change
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* align test data with upstream changes
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* Update __init__.py on docling_core.types.doc
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
* Remove DescriptionItem
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com >
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com >
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com >
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com >
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-16 15:53:44 +02:00
Panos Vagenas and GitHub
aec1a41402
fix: loosen pandas version ( #40 )
...
* fix: loosen pandas version
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* add pandas stubs
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
* explicitly convert to str
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
---------
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-09 13:46:57 +02:00
Panos Vagenas and GitHub
39ae6cd253
chore: minor refactor incl. explicit typing ( #39 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-07 13:59:40 +02:00
Panos Vagenas and GitHub
246627f4f6
fix: make doc metadata keys pure strings ( #38 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-07 09:38:47 +02:00
Panos Vagenas and GitHub
b5592ad747
fix: align chunk ref format with one used in Document ( #37 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-10-07 09:07:08 +02:00
Panos Vagenas and GitHub
b76780c3b2
feat: add doc metadata extractor and ID generator classes ( #34 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-09-30 15:27:35 +02:00
Panos Vagenas and GitHub
4bde51528d
feat: support heading as chunk metadata ( #36 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-09-30 11:08:09 +02:00
Panos Vagenas and GitHub
9698d30288
feat: add hierarchical chunker ( #18 )
...
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com >
2024-09-11 12:57:56 +02:00
Panos VagenasandGitHub
Cesar Berrospi RamisandGitHub
Peter W. J. StaarandGitHub
EugeneandGitHub
