Commit Graph
71 Commits
Author SHA1 Message Date
Peter Staar 29fe809cf0 refactored the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-06-05 16:54:37 +02:00
Panos VagenasandGitHub d8a5256b2c feat: add table annotations (#304)
* feat: add table annotations

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* refactor annotation types

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* expand to HTML

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* introduce annotation serializer

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* Update dummy_doc.yaml

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-06-05 15:38:46 +02:00
Panos VagenasandGitHub 87b72d6537 fix(HybridChunker): refine max_tokens auto-detection (#306)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-27 15:38:49 +02:00
f067c51c48 feat: add annotations in MD & HTML serialization (#295)
* feat: include annotations in MD & HTML serialization

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* (HTML) move annotations into figcaptions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add explicit beginning/end markers, fix case of excluded refs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* improve annotation marking, extend tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* wrap captions (#305)

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* revert temp test changes

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2025-05-27 10:49:13 +02:00
Said GürbüzandGitHub ae1cdcedad test: update tests for load_from_doctags (#299)
* update tests for load_from_doctags

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update tests to fix the flakiness

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

---------

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>
2025-05-20 15:06:25 +02:00
Panos VagenasandGitHub c49a50e76b fix(markdown): fix case of empty page break string (#298)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-20 10:51:24 +02:00
Panos VagenasandGitHub 7eb9fa96e6 fix: include captions regardless of traverse_pictures flag (#278)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-05 10:11:18 +02:00
763e1364ff feat: Add tiktoken tokenizers support to HybridChunker (#240)
* feat: Add tiktoken tokenizers support to HybridChunker

Signed-off-by: ruizguille <guillermo@codeawake.com>

* separate OpenAI tokenizer

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: ruizguille <guillermo@codeawake.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-18 07:43:20 +02:00
Panos VagenasandGitHub a258d525e1 feat: add visualizers (#263)
* feat: add visualizers

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* make visualizers composable

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* use BoundingRectangle instead of BoundingBox

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* enforce top-left coordinates

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* narrow down test data to first 3 pages

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add file deletions

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-17 11:43:45 +02:00
d0a49da6de fix: HTML serialization for single image documents (#261)
* fix for HTML serialization for single image documents

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* minor refactor, add test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-04-16 16:40:27 +02:00
caa8aeefae feat: Chart tabular data serialization for HTML serializer (#258)
* Add chart tabular data serialization to HTML serializer

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixing pre-commit issues

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixed table serialization for tabular chart data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* test for loading doctags with chart data

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Improved loading doctags test with chart example, added tests for chart serialization into html and md

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-04-15 13:31:50 +02:00
Panos VagenasandGitHub 928e5c5046 fix: fix page breaking in case page starts with group (#253)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-14 18:12:47 +02:00
Panos VagenasandGitHub 15d2f2c1d5 fix: fix code handling in HTML serialization (#251)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-14 07:10:14 +02:00
Panos VagenasandGitHub 98c60bb65b perf: fix serialization performance (#249)
* perf: fix serialization performance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add missing leading HTML tag

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-11 15:37:52 +02:00
5d406008f2 feat: add HTML serializer (#232)
* iunitial attempt at HTML serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first version, to be tested thoroughly

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the new test

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rewrote carefully the export-to-html into new framework

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the inline list-items

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the inline code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* migrated the table html code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the picture HTML serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the html for Form and KeyValue

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first version of KeyValue serialisation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the key-value and form-region and added the GraphData serializer

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* need to do some mypy work now

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* passed the mypy

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the get_excluded_refs function to obtain proper serialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* enabled the captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed empty lists

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added initial split view and customised styles

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* cleaned up, now waiting for page-indices propagationg

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the styles and parameters with split_page

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* propagated the parameter split_page_view

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first fully working version

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the prints

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests for html export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed dead code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test output

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rename parameter tag to class_name

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added serializers to table and picture

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed dead code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* various HTML serialization improvements (#242)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* added enum for different output styles

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-10 10:59:07 +02:00
Panos VagenasandGitHub 5e4c0fd51f feat: integrate serialization API into chunkers (#221)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-08 14:04:40 +02:00
e9259a5f87 feat: Support of DocTags charts (serialization and deserialization) (#229)
* Doctags charts deserialization into Docling Document, added PictureTabularChartData

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Correction for chart label mapping

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* simplified label mapping

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Improved docummentation comment

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Added stacked_bar_chart to labels and tokens, as well as adjusted doctags_load

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* doctags serializer for charts

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* - Fixed DocTags table deserialization
- Updated tests

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* cleaning

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Allowing DocItemLabel.CHART as a label for PictureItem

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-04-07 12:27:48 +02:00
f2fe1c1bc6 feat: added initial delete and insert methods in DoclingDocument (#220)
* added intial delete and merge

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* work-in-progress for the delete option

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the delete_document_item

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* verified the model-validation

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the delete item

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reproducible error in tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* before rebase with main

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the tests (2) and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the iterate_items to reduce code duplication

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* mapped path method to _split_ref_to_path

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* made get_parent_ref private method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated with new public functions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on the comments of the PR

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* factored out the internal tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the code and updated the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* update return types, extend unit test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* add replace_item test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* revert unnecessary comments

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-04-04 14:34:25 +02:00
a055e1a9f0 fix: DocTags import location fix for tables, pictures, captions (#227)
* Fix for the table with captions bboxes

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* cleaned up

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Correcting caption bbox logic for pictures

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Added test, small fix with adding caption to the picture when image not available

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* test cases (pictures + png)

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* fixed comment

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Corrected caption spans for tables and pictures, added check of captions in the tests

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2025-04-02 09:52:54 +02:00
Panos VagenasandGitHub 2efb71a0ca fix: allow captions without holding item (#215)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-25 11:13:59 +01:00
Panos VagenasandGitHub ff13a9385d feat: expose MD page break & DocTags minification (#213)
* feat: expose MD page break & DocTags minification

Also:
- aligned DocTags method naming
- fixed mutable default arguments

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* cover newly exposed params in testing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* drop DocTags-specific post-processing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-25 10:52:09 +01:00
db119f4665 feat: Add document tokens from key value items (#170)
* add export_to_document_tokens method for KeyValueItem

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* fix export_to_document_tokens kv-item

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update key-link representations in document tokens

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* fix key-value cell location for doctags

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* set default page_no to 1

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* fix get_location call with to_top_left_origin

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* integrate export_to_document_tokens to doctag serializer

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* Add DocTags serializer dispatching, deprecate new_line param (#212)

* updates for key value region

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* deprecate "new_line" parameter

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-03-24 12:24:40 +01:00
Panos VagenasandGitHub 1f4d57ed18 feat: add DocTags serializers (#192)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-21 14:49:28 +01:00
SaidgurbuzandGitHub 2371c11b8f feat: add kv_item support for doctag to docling_document (#188)
* add kv_item support for doctag to docling_document

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* use resize_by_scale to save locations

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add kv region to tag_to_doclabel

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add test for doctags_load_for_kv_region

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update the naming to .dt for consistency

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

---------

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>
2025-03-19 14:07:13 +01:00
Panos VagenasandGitHub 7ed4d225b6 fix(markdown): fix ordered list numbering (#200)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-18 17:20:07 +01:00
Panos VagenasandGitHub 65a82a158c chore: update text label for test doc creation (#195)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-17 14:14:03 +01:00
Michele DolfiandGitHub 11eb1dc5aa fix: new favicon url (#191)
fix favicon url

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-03-17 13:16:42 +01:00
Panos VagenasandGitHub a7cdc87411 feat: add serializers, text formatting, update Markdown export (#182)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-13 17:54:32 +01:00
c065c4c7c6 feat: Add DoclingDocument.load_from_doctags method and DocTags data models (#187)
* Adding load_from_document_tokens method to Docling document

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Added skeleton for the unit test

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Processing ordered and unordered lists

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Define new types for DocTags containers

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Implement load_from_doctags with container types

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* update test data

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Cleanup commented lines

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix for proper list groups

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Check for number of page doctags must be equal to page images

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2025-03-12 16:11:56 +01:00
Lucas MorinandGitHub 32398b8ac0 feat: add document tokens for SMILES (#176)
Signed-off-by: lum <lum@zurich.ibm.com>
2025-03-10 11:04:21 +01:00
Panos VagenasandGitHub 53f6d099b0 fix(doctags): fix code export (#181)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-04 09:41:32 +01:00
Panos VagenasandGitHub 834db4bc66 fix(markdown): fix escaping in case of nesting (#180)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-03 14:51:59 -05:00
Panos VagenasandGitHub 349f7daa0c fix(markdown): fix handling of ordered lists (#175)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-02-27 16:08:05 +01:00
Panos VagenasandGitHub c77c59bec0 fix(markdown): fix case of leading list (#174)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-02-27 11:10:42 +01:00
Panos VagenasandGitHub 2abaf9b537 feat: add inline groups, revamp Markdown export incl. list groups (#156)
* feat: add inline groups, update markdown export

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* rename parameter for clarity

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* drop redundant `traverse_inline`, use only `visited`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update API usage after rebase

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* minor refactor

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* extend nesting logic to list groups

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* (MD export) add inline formulas, placeholders for `KeyValueItem`, `FormItem`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* include 3rd-level list item in test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update test file with more recent conversion result

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* update list construction pattern so that sublists are added to actual ListItems

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-02-25 09:48:10 +01:00
d622800750 feat: Introduce Key-Value and Forms items (#158)
* Draft KeyValueItem content

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* added add_key_value_item method

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add KeyValueLink

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update an add_key_value_item argument

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* remove KeyOrValueCellType and KeyValueLinkType

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update tests for KeyValueItem

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add union method to create bbox that covers all the given bboxes

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* added the , and rewrote it to make it more general with a GraphItem, as well as having cell- and link-labels

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the code due to MyPy and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the reference docs with form-key

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the circle

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the figure, testing now ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added square in image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed square in image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* tests should go through

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* update poetry.lock versions

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* fix import issue

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add validator for links in GraphData

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added field_validator

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rename add_key_values and add_form

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* rename union to enclosingbbox

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
2025-02-19 09:50:45 +01:00
916323fb55 feat: Redefine CodeItem as floating object with captions (#160)
* updated the CodeItem with captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* use FloatItem and add captions to add_code interface

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2025-02-14 11:34:40 +01:00
f751b45b62 feat: Implementation of doc tags (#138)
* Migration to new DocTags, gt missing

Signed-off-by: Ahmed Nassar <ahn@zurich.ibm.com>

* fixed the tests

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* overloaded export to markdown in CodeItem to include code language

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* minor change

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* implemented page break token instead of page number

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added figure classification in doc tags

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed table repr. to otsl

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed table repr. to otsl fixed new line

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fixed new line bug

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fixed new line bug

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* removed print in test

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* new normalization to 500 and top left coord

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* export of FOOTER AND HEADER

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* export of FOOTER AND HEADER

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fix code bug

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added export of standalon captions

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added export of standalon captions

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed generate to False

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added export of furnitures and fixed bug in iterate_items function

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

---------

Signed-off-by: Ahmed Nassar <ahn@zurich.ibm.com>
Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Co-authored-by: Ahmed Nassar <ahn@zurich.ibm.com>
Co-authored-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
2025-02-13 13:26:28 +00:00
786f0c6833 feat: Add ContentLayer attribute to designate items to body or furniture (#148)
* feat: Add ContentLayer attribute to designate items to body or furniture

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce safer data gen mechanism, update chunking test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Do not make test rely on order in yaml

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore: format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: legacy_to_docling_doc must use content_layer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add content_layer in iterate_items

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Bump format version, add model_validator for old page_header,page_footer in body

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: Change to before model_validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Address review comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-02-10 10:58:41 +01:00
Michele DolfiandGitHub 23477f7674 feat(HTML): Fallback showing formulas as images (#146)
* remove un-needed logic

the labels allowlist is checked before

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* textitem cannot have label code

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* display formulas with mathml in exported html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* expose argument in save_as_html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* rename sanitize in prepare and add \n

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix mypy parsing

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove unused/impossible elif

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove strip()

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add display none for latex annotation

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fallback showing equations as image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add html placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* markdown placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-02-03 09:29:30 +01:00
Michele DolfiandGitHub ed36437346 feat(HTML): Export formulas with mathml (#144)
* remove un-needed logic

the labels allowlist is checked before

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* textitem cannot have label code

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* display formulas with mathml in exported html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* expose argument in save_as_html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* rename sanitize in prepare and add \n

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix mypy parsing

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove unused/impossible elif

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove strip()

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add display none for latex annotation

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* make mathml the default

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* revert wrong commit

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-01-31 15:43:43 +01:00
Michele DolfiandGitHub c6590e83e2 fix: add html escape in md export and fix formula escapes (#143)
* add html escape in md export and fix formula escapes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* escape also html export

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-01-31 11:31:35 +01:00
Panos VagenasandGitHub b741eeaab4 fix: fix hybrid chunker token constraint (#131)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-01-17 17:22:26 +01:00
c940aa5ca9 feat: Add CodeItem as pydantic type, update export methods and APIs (#129)
* added code item

* added code item

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* add constraints to allow numpy > 2.1.0 on python3.13 and others

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Add CodeItem to ContentItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* added CodeItem in ContentItem tagged union.

* added enum for programming languages

* removed double CodeItem in ContentItem Union

* fixed type of code_language in CodeItem class

* fixed sorting of programming languages, not sorted anymore by value of string but variable name

---------

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2025-01-17 16:59:33 +01:00
Michele DolfiandGitHub b31e0a3d05 feat: add legacy to DoclingDocument utility (#108)
* add legacy_to_docling_document

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix for python < 3.12

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add test

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add docs with not-preserved components

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-12-13 13:28:41 +01:00
Panos VagenasandGitHub 047a1960af fix: improve doc item typing (#105)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-13 12:51:53 +01:00
Panos VagenasandGitHub b546c0a50d fix: set origin when merging chunks (#109)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-13 09:05:56 +01:00
Panos VagenasandGitHub 2591c70c66 feat: add default tokenizer to HybridChunker (#107)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-12 16:51:08 +01:00
437c498f77 feat: utilities converting document formats (#91)
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2024-12-09 14:14:01 +01:00
BoyceandGitHub fd7529f409 fix(markdown): preserve underscores in image URLs during markdown export (#98)
Signed-off-by: Boyce <aboyskyi@gmail.com>
2024-12-09 11:38:23 +01:00