Commit Graph
247 Commits
Author SHA1 Message Date
Yusik Kim 4708f933a7 fix: properly handle missing page image case for export_to_html (#166)
* fix: properly handle missing image case

Signed-off-by: Yusik Kim <kmyusk@gmail.com>

* chore: pre-commit changes

Signed-off-by: Yusik Kim <kmyusk@gmail.com>

* fix: simplify test and address broken test

Signed-off-by: Yusik Kim <kmyusk@gmail.com>

---------

Signed-off-by: Yusik Kim <kmyusk@gmail.com>
2025-02-25 09:43:39 +01:00
d622800750 feat: Introduce Key-Value and Forms items (#158)
* Draft KeyValueItem content

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* added add_key_value_item method

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add KeyValueLink

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update an add_key_value_item argument

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* remove KeyOrValueCellType and KeyValueLinkType

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* update tests for KeyValueItem

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add union method to create bbox that covers all the given bboxes

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* added the , and rewrote it to make it more general with a GraphItem, as well as having cell- and link-labels

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the code due to MyPy and reformatted

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the reference docs with form-key

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the circle

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the figure, testing now ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added square in image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed square in image

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* tests should go through

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* update poetry.lock versions

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* fix import issue

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* add validator for links in GraphData

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added field_validator

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* rename add_key_values and add_form

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

* rename union to enclosingbbox

Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Saidgurbuz <said.gurbuz@epfl.ch>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
2025-02-19 09:50:45 +01:00
Peter W. J. StaarandMichele Dolfi 916323fb55 feat: Redefine CodeItem as floating object with captions (#160)
* updated the CodeItem with captions

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* use FloatItem and add captions to add_code interface

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2025-02-14 11:34:40 +01:00
f751b45b62 feat: Implementation of doc tags (#138)
* Migration to new DocTags, gt missing

Signed-off-by: Ahmed Nassar <ahn@zurich.ibm.com>

* fixed the tests

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* overloaded export to markdown in CodeItem to include code language

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* minor change

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* implemented page break token instead of page number

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added figure classification in doc tags

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed table repr. to otsl

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed table repr. to otsl fixed new line

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fixed new line bug

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fixed new line bug

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* removed print in test

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* new normalization to 500 and top left coord

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* export of FOOTER AND HEADER

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* export of FOOTER AND HEADER

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* fix code bug

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added export of standalon captions

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added export of standalon captions

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* changed generate to False

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added export of furnitures and fixed bug in iterate_items function

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

---------

Signed-off-by: Ahmed Nassar <ahn@zurich.ibm.com>
Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Co-authored-by: Ahmed Nassar <ahn@zurich.ibm.com>
Co-authored-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
2025-02-13 13:26:28 +00:00
Christoph AuerandPanos Vagenas 786f0c6833 feat: Add ContentLayer attribute to designate items to body or furniture (#148)
* feat: Add ContentLayer attribute to designate items to body or furniture

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce safer data gen mechanism, update chunking test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Do not make test rely on order in yaml

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore: format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: legacy_to_docling_doc must use content_layer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add content_layer in iterate_items

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Bump format version, add model_validator for old page_header,page_footer in body

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: Change to before model_validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Address review comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-02-10 10:58:41 +01:00
Michele Dolfi 23477f7674 feat(HTML): Fallback showing formulas as images (#146)
* remove un-needed logic

the labels allowlist is checked before

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* textitem cannot have label code

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* display formulas with mathml in exported html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* expose argument in save_as_html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* rename sanitize in prepare and add \n

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix mypy parsing

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove unused/impossible elif

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove strip()

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add display none for latex annotation

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fallback showing equations as image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add html placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* markdown placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-02-03 09:29:30 +01:00
Michele Dolfi ed36437346 feat(HTML): Export formulas with mathml (#144)
* remove un-needed logic

the labels allowlist is checked before

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* textitem cannot have label code

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* display formulas with mathml in exported html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* expose argument in save_as_html

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* rename sanitize in prepare and add \n

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix mypy parsing

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove unused/impossible elif

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* remove strip()

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add display none for latex annotation

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* make mathml the default

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* revert wrong commit

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-01-31 15:43:43 +01:00
Michele Dolfi c6590e83e2 fix: add html escape in md export and fix formula escapes (#143)
* add html escape in md export and fix formula escapes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* escape also html export

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-01-31 11:31:35 +01:00
Peter W. J. Staar f02bbaea47 feat: added the geometric operations to BoundingBox (#136)
* added the geometric operations

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the geometric operations and added deprecation warnings to scaled and normalize

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* finalised the geometric operations

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated tests and functionality

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-01-29 11:11:46 +01:00
Panos Vagenas b741eeaab4 fix: fix hybrid chunker token constraint (#131)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2025-01-17 17:22:26 +01:00
c940aa5ca9 feat: Add CodeItem as pydantic type, update export methods and APIs (#129)
* added code item

* added code item

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* added code item

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>

* add constraints to allow numpy > 2.1.0 on python3.13 and others

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Add CodeItem to ContentItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* added CodeItem in ContentItem tagged union.

* added enum for programming languages

* removed double CodeItem in ContentItem Union

* fixed type of code_language in CodeItem class

* fixed sorting of programming languages, not sorted anymore by value of string but variable name

---------

Signed-off-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Matteo-Omenetti <Matteo.Omenetti1@ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2025-01-17 16:59:33 +01:00
Michele Dolfi 268c294cc9 fix: always write with utf8 encoding (#111)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-12-17 09:37:01 +01:00
Michele Dolfi b31e0a3d05 feat: add legacy to DoclingDocument utility (#108)
* add legacy_to_docling_document

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix for python < 3.12

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add test

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add docs with not-preserved components

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-12-13 13:28:41 +01:00
Panos Vagenas 047a1960af fix: improve doc item typing (#105)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-13 12:51:53 +01:00
Panos Vagenas b546c0a50d fix: set origin when merging chunks (#109)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-13 09:05:56 +01:00
Panos Vagenas 2591c70c66 feat: add default tokenizer to HybridChunker (#107)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-12 16:51:08 +01:00
Michele DolfiandChristoph Auer 437c498f77 feat: utilities converting document formats (#91)
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2024-12-09 14:14:01 +01:00
Eric Musa e8f775569b Fix: solved embedded image serialization/deserialization with Pydantic. (#95)
Signed-off-by: Eric Musa <eric.musa17@gmail.com>
2024-12-09 14:12:59 +01:00
Boyce fd7529f409 fix(markdown): preserve underscores in image URLs during markdown export (#98)
Signed-off-by: Boyce <aboyskyi@gmail.com>
2024-12-09 11:38:23 +01:00
628ab679cb feat: add hybrid chunker (#68)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Bill Murdock <bmurdock@redhat.com>
Co-authored-by: Ben Rood <bash99@gmail.com>
2024-12-06 14:56:10 +01:00
Cesar Berrospi Ramis 266e7fc603 build: set pydantic version to 2.10.3 (#93)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-12-04 14:26:22 +01:00
Maxim LysakandMaksym Lysak 180e294aad feat: Export to OTSL method for docling doc tables (#86)
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2024-12-03 16:31:14 +01:00
Michele DolfiandPanos Vagenas 9a74d13fd6 feat: extend source resolution with streams and workdir (#79)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-02 14:12:55 +01:00
Maxim LysakandMaksym Lysak 8422ad4fbf fix: Even better style for HTML export (#78)
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2024-11-28 16:54:01 +01:00
Maxim LysakandMaksym Lysak a94ba273b1 chore: Improving styling for HTML export (#77)
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2024-11-27 20:19:43 +01:00
ef49fd3f34 feat: adding HTML export to DoclingDocument, adding export of images in png with links to Markdown & HTML (#69)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2024-11-27 05:27:23 +01:00
Shubham GuptaandShubham Gupta 9d7e831fb2 feat: add get_image for all DocItem (#67)
* Moved image attribute from FloatingItem to DocItem

Signed-off-by: Shubham Gupta <26436285+sh-gupta@users.noreply.github.com>

* Revert "Moved image attribute from FloatingItem to DocItem"

This reverts commit e48cd47d3743a25ce17bbaf31b3152ed53365fba.

Signed-off-by: Shubham Gupta <26436285+sh-gupta@users.noreply.github.com>

* Added get_image to DocItem and FloatingItem

Signed-off-by: Shubham Gupta <26436285+sh-gupta@users.noreply.github.com>

* Added tests for get_image in DocItem and FloatingItem

Signed-off-by: Shubham Gupta <26436285+sh-gupta@users.noreply.github.com>

* Updated get_image docstring in DocItem and FloatingItem

Signed-off-by: Shubham Gupta <26436285+sh-gupta@users.noreply.github.com>

---------

Signed-off-by: Shubham Gupta <26436285+sh-gupta@users.noreply.github.com>
Co-authored-by: Shubham Gupta <Shubham.Gupta1@ibm.com>
2024-11-18 13:34:16 +01:00
Panos Vagenas bfeb2db24b fix: include titles to chunk heading metadata (#62)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-11-01 13:41:16 +01:00
Panos Vagenas 2c88e56f8c fix: fix resolution in case of URL without path (#55)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-24 13:07:59 +02:00
Panos Vagenas d09fe7ed44 feat: extend chunk meta with schema, version, origin (#49)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-22 11:08:04 +02:00
Panos Vagenas e12d6a70c3 fix: fix legacy doc ref (#48)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-18 15:21:01 +02:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Panos Vagenas 39ae6cd253 chore: minor refactor incl. explicit typing (#39)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-07 13:59:40 +02:00
Panos Vagenas b5592ad747 fix: align chunk ref format with one used in Document (#37)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-07 09:07:08 +02:00
688789ea75 feat: (experimental) introduce new document format (#21)
* Draft new docling document format, pydantic model and tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix tests to have unique document_hashes per test

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Manual update from main

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Move new-format to experimental path

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Updates for document construction API and format

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add BaseTableData and table cell typing

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Tree element iterator, several API fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Turn captions into list field

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add export methods to DoclingDocument and types

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Change DoclingDocument.iterate_elements and add print tree function

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce label enum types, apply everywhere

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce provenance info, use enum labels

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update formatting

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Docstrings and linter fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Lockfile rollback, since updating breaks tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Several improvements and cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Big redesign for usage of hashes, several other fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix flake8 config

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove hash, renamings

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* updating the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added some unit tests for DocItem

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Add tree validation API and test case

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add extra=Forbid to NodeItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: set DoclingDocument version as SemanticVersion with default

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Simpler literal enforcement

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix static document version

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rollback changes to allow for semver<3

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* build: rollback changes to include python-semantic-release as dev dependency

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* feat: set version field as string with pattern and check compatibility

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* add JSON Pointer validation to refs, fix test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-01 09:41:16 +02:00
Panos Vagenas 4bde51528d feat: support heading as chunk metadata (#36)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-30 11:08:09 +02:00
Cesar Berrospi Ramis a4ddd142ee fix: change order of JSON Schema to search mapper transformations (#32)
* fix: change order of JSON Schema to search mapper transformations

In the JSON Schema to search mapper, the suppression flags need to be addressed first,
otherwise the jsonref.replace_refs function may remove them.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* build: update dependencies

Updating pydantic from 2.8.2 to 2.9.2 triggers a change in JSON Schema from models:
the  lists with only 1 element get flatten.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: improve verbose in JSON Schema to search mapper test

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-09-26 13:49:18 +02:00
Michele Dolfi a334b9fc72 fix: remove duplicate captions in markdown (#31)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-24 14:33:56 +02:00
Michele Dolfi b843ae6688 feat: add figures in markdown export (#27)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-23 20:12:06 +02:00
Peter W. J. Staar 891530f595 feat: add export to doctags for document components (#25)
* feat: refactoring doc-tokens in new file and add new export function to table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted and fixed bugs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on table exporting in document tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated code and tests, need to decide what to commit as test-cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the test-files

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (boundingbox to figure and table)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (making BaseCell the base-class of BaseText)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (re-use of function in BaseCell)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed issues from review (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-20 14:16:33 +02:00
Panos Vagenas 9698d30288 feat: add hierarchical chunker (#18)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-11 12:57:56 +02:00
Peter W. J. Staar acdf816081 feat: added the XML export (#16)
* added the XML export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted all

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the DocumentTokens class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the strict-text in the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added page-tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the location/page tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-09 17:41:58 +02:00
Michele Dolfi f450c8cbfd fix: use same base type for all components (#10)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-08-28 11:21:05 +02:00
Cesar Berrospi Ramis faf472c168 fix: set type to optional (#7)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-23 14:07:42 +02:00
Panos Vagenas d0ffc85e0c feat: add document Markdown export (#4)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-18 10:12:43 +02:00
Cesar Berrospi Ramis 656f56380f fix: fix definition issues in record type (#2)
* fix(test): set static typing compatible to python 3.10

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* style: enforce some pre-commit hooks on tests

Enforce pre-commit hooks black, isort, autoflake, and mypy on test modules.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* fix(rec): fix definition issues in attribute, predicate, subject

Remove duplicate generic types across base and predicate modules.
Create an identifier class for subject names.
Remove unnecessary type variables in attribute model.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* docs: refer to Docling data objects

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-17 17:24:31 +02:00
Cesar Berrospi Ramis 9134ffee93 Initial commit
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-15 12:55:20 +02:00