Commit Graph
172 Commits
Author SHA1 Message Date
Boyce fd7529f409 fix(markdown): preserve underscores in image URLs during markdown export (#98)
Signed-off-by: Boyce <aboyskyi@gmail.com>
2024-12-09 11:38:23 +01:00
628ab679cb feat: add hybrid chunker (#68)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Bill Murdock <bmurdock@redhat.com>
Co-authored-by: Ben Rood <bash99@gmail.com>
2024-12-06 14:56:10 +01:00
Cesar Berrospi Ramis 266e7fc603 build: set pydantic version to 2.10.3 (#93)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-12-04 14:26:22 +01:00
Maxim LysakandMaksym Lysak 8422ad4fbf fix: Even better style for HTML export (#78)
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2024-11-28 16:54:01 +01:00
Maxim LysakandMaksym Lysak a94ba273b1 chore: Improving styling for HTML export (#77)
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
2024-11-27 20:19:43 +01:00
ef49fd3f34 feat: adding HTML export to DoclingDocument, adding export of images in png with links to Markdown & HTML (#69)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
2024-11-27 05:27:23 +01:00
Panos Vagenas bfeb2db24b fix: include titles to chunk heading metadata (#62)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-11-01 13:41:16 +01:00
Panos Vagenas d09fe7ed44 feat: extend chunk meta with schema, version, origin (#49)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-22 11:08:04 +02:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Panos Vagenas b5592ad747 fix: align chunk ref format with one used in Document (#37)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-07 09:07:08 +02:00
688789ea75 feat: (experimental) introduce new document format (#21)
* Draft new docling document format, pydantic model and tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix tests to have unique document_hashes per test

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Manual update from main

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Move new-format to experimental path

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Updates for document construction API and format

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add BaseTableData and table cell typing

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Tree element iterator, several API fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Turn captions into list field

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add export methods to DoclingDocument and types

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Change DoclingDocument.iterate_elements and add print tree function

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce label enum types, apply everywhere

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce provenance info, use enum labels

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update formatting

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Docstrings and linter fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Lockfile rollback, since updating breaks tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Several improvements and cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Big redesign for usage of hashes, several other fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix flake8 config

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove hash, renamings

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* updating the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added some unit tests for DocItem

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Add tree validation API and test case

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add extra=Forbid to NodeItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: set DoclingDocument version as SemanticVersion with default

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Simpler literal enforcement

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix static document version

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rollback changes to allow for semver<3

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* build: rollback changes to include python-semantic-release as dev dependency

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* feat: set version field as string with pattern and check compatibility

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* add JSON Pointer validation to refs, fix test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-01 09:41:16 +02:00
Panos Vagenas 4bde51528d feat: support heading as chunk metadata (#36)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-30 11:08:09 +02:00
Michele Dolfi a334b9fc72 fix: remove duplicate captions in markdown (#31)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-24 14:33:56 +02:00
Michele Dolfi b843ae6688 feat: add figures in markdown export (#27)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-23 20:12:06 +02:00
Peter W. J. Staar 891530f595 feat: add export to doctags for document components (#25)
* feat: refactoring doc-tokens in new file and add new export function to table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted and fixed bugs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on table exporting in document tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated code and tests, need to decide what to commit as test-cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the test-files

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (boundingbox to figure and table)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (making BaseCell the base-class of BaseText)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (re-use of function in BaseCell)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed issues from review (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-20 14:16:33 +02:00
Panos Vagenas 9698d30288 feat: add hierarchical chunker (#18)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-11 12:57:56 +02:00
Peter W. J. Staar acdf816081 feat: added the XML export (#16)
* added the XML export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted all

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the DocumentTokens class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the strict-text in the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added page-tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the location/page tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-09 17:41:58 +02:00
Michele Dolfi f450c8cbfd fix: use same base type for all components (#10)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-08-28 11:21:05 +02:00
Cesar Berrospi Ramis faf472c168 fix: set type to optional (#7)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-23 14:07:42 +02:00
Panos Vagenas d0ffc85e0c feat: add document Markdown export (#4)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-18 10:12:43 +02:00
Cesar Berrospi Ramis 656f56380f fix: fix definition issues in record type (#2)
* fix(test): set static typing compatible to python 3.10

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* style: enforce some pre-commit hooks on tests

Enforce pre-commit hooks black, isort, autoflake, and mypy on test modules.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* fix(rec): fix definition issues in attribute, predicate, subject

Remove duplicate generic types across base and predicate modules.
Create an identifier class for subject names.
Remove unnecessary type variables in attribute model.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* docs: refer to Docling data objects

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-17 17:24:31 +02:00
Cesar Berrospi Ramis 9134ffee93 Initial commit
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-15 12:55:20 +02:00