Commit Graph
596 Commits
Author SHA1 Message Date
github-actions[bot] 4496b448f8 chore: bump version to 1.7.0 [skip ci] v1.7.0 2024-10-01 07:56:08 +00:00
688789ea75 feat: (experimental) introduce new document format (#21)
* Draft new docling document format, pydantic model and tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix tests to have unique document_hashes per test

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Manual update from main

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Move new-format to experimental path

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Updates for document construction API and format

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add BaseTableData and table cell typing

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Tree element iterator, several API fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Turn captions into list field

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add export methods to DoclingDocument and types

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Change DoclingDocument.iterate_elements and add print tree function

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce label enum types, apply everywhere

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce provenance info, use enum labels

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update formatting

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Docstrings and linter fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Lockfile rollback, since updating breaks tests

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Several improvements and cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Format fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Big redesign for usage of hashes, several other fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix flake8 config

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove hash, renamings

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* updating the tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added some unit tests for DocItem

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Add tree validation API and test case

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add extra=Forbid to NodeItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: set DoclingDocument version as SemanticVersion with default

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Simpler literal enforcement

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix static document version

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rollback changes to allow for semver<3

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* build: rollback changes to include python-semantic-release as dev dependency

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* feat: set version field as string with pattern and check compatibility

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* add JSON Pointer validation to refs, fix test data

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-01 09:41:16 +02:00
Panos Vagenas b76780c3b2 feat: add doc metadata extractor and ID generator classes (#34)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-30 15:27:35 +02:00
Panos Vagenas 4bde51528d feat: support heading as chunk metadata (#36)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-30 11:08:09 +02:00
github-actions[bot] 17af1fc659 chore: bump version to 1.6.3 [skip ci] v1.6.3 2024-09-26 11:53:10 +00:00
Cesar Berrospi Ramis a4ddd142ee fix: change order of JSON Schema to search mapper transformations (#32)
* fix: change order of JSON Schema to search mapper transformations

In the JSON Schema to search mapper, the suppression flags need to be addressed first,
otherwise the jsonref.replace_refs function may remove them.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* build: update dependencies

Updating pydantic from 2.8.2 to 2.9.2 triggers a change in JSON Schema from models:
the  lists with only 1 element get flatten.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: improve verbose in JSON Schema to search mapper test

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-09-26 13:49:18 +02:00
github-actions[bot] b49e93e4a9 chore: bump version to 1.6.2 [skip ci] v1.6.2 2024-09-24 12:35:54 +00:00
Michele Dolfi a334b9fc72 fix: remove duplicate captions in markdown (#31)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-24 14:33:56 +02:00
github-actions[bot] bdda0ee1e8 chore: bump version to 1.6.1 [skip ci] v1.6.1 2024-09-24 08:45:59 +00:00
Panos Vagenas e706d686db fix: remove unnecessary package dependency (#30)
Triggering a release retrospectively for #29.

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-24 10:43:56 +02:00
Panos Vagenas b53690feb1 chore: remove odd lib (#29)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-24 10:23:11 +02:00
github-actions[bot] 1ae27f05a8 chore: bump version to 1.6.0 [skip ci] v1.6.0 2024-09-23 18:14:55 +00:00
Michele Dolfi b843ae6688 feat: add figures in markdown export (#27)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-23 20:12:06 +02:00
github-actions[bot] ded530aa14 chore: bump version to 1.5.0 [skip ci] v1.5.0 2024-09-20 14:48:33 +00:00
Michele Dolfi 1a4440a771 chore: switch to gh apps user (#26)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-20 16:40:30 +02:00
Peter W. J. Staar 891530f595 feat: add export to doctags for document components (#25)
* feat: refactoring doc-tokens in new file and add new export function to table

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted and fixed bugs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* working on table exporting in document tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated code and tests, need to decide what to commit as test-cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the test cases

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored the test-files

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (boundingbox to figure and table)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (making BaseCell the base-class of BaseText)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* refactored duplicate code (re-use of function in BaseCell)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed issues from review (2)

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-20 14:16:33 +02:00
Panos Vagenas 752cbc3e89 feat: add file source resolution utility (#22)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-20 06:06:43 +02:00
Panos Vagenas 9b8e97b658 chore: update cryptography locked version (#23)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-18 14:23:42 +02:00
github-actions[bot] 2dfc775812 chore: bump version to 1.4.1 [skip ci] v1.4.1 2024-09-18 08:11:51 +00:00
Peter W. J. Staar 9bc256e5bb fix: add export to xml and html (#17)
* added the XML export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted all

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the DocumentTokens class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the strict-text in the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added page-tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the location/page tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* small fix to have correct special document-tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-18 09:23:15 +02:00
github-actions[bot] 2f55d92732 chore: bump version to 1.4.0 [skip ci] v1.4.0 2024-09-18 05:47:16 +00:00
Michele Dolfi 2cc2429e27 feat: add table exporters (#20)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-17 16:04:05 +02:00
github-actions[bot] 1ed846c942 chore: bump version to 1.3.0 [skip ci] v1.3.0 2024-09-11 11:02:08 +00:00
Panos Vagenas 9698d30288 feat: add hierarchical chunker (#18)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-09-11 12:57:56 +02:00
github-actions[bot] c4826105b2 chore: bump version to 1.2.0 [skip ci] v1.2.0 2024-09-10 11:36:33 +00:00
Peter W. J. Staar acdf816081 feat: added the XML export (#16)
* added the XML export

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted all

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed tests

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the DocumentTokens class

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updating the to-xml method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added the strict-text in the to-md method

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* added page-tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the location/page tokens

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2024-09-09 17:41:58 +02:00
github-actions[bot] 25af125530 chore: bump version to 1.1.4 [skip ci] v1.1.4 2024-09-06 15:09:33 +00:00
Michele Dolfi 235b2cd10b fix: validate_model() could be called with other types rather than dict (#14)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-06 16:53:31 +02:00
Christoph Auer aaf17fe0f6 docs: Update link to Docling (#12)
* docs: Update link to Docling

Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>

* Update citation

Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>

---------

Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
2024-09-02 12:47:51 +02:00
Panos Vagenas 05aab2745c chore: clean up dependencies (#11)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-08-29 15:12:37 +02:00
github-actions[bot] 5aa41fcc96 chore: bump version to 1.1.3 [skip ci] v1.1.3 2024-08-28 09:29:34 +00:00
Michele Dolfi f450c8cbfd fix: use same base type for all components (#10)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-08-28 11:21:05 +02:00
github-actions[bot] 3f81690a87 chore: bump version to 1.1.2 [skip ci] v1.1.2 2024-07-31 11:42:54 +00:00
Panos Vagenas ec3cff97e5 fix: make page number strictly positive (#8)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-31 13:36:49 +02:00
github-actions[bot] 0cdcc9bf0b chore: bump version to 1.1.1 [skip ci] v1.1.1 2024-07-23 12:13:53 +00:00
Cesar Berrospi Ramis faf472c168 fix: set type to optional (#7)
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-23 14:07:42 +02:00
Panos VagenasandMichele Dolfi 3f77b2e92c docs: revamp installation instructions (#6)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
2024-07-23 14:03:40 +02:00
Cesar Berrospi Ramis 7b9e81f979 chore: set default name field in Package to docling-core (#5)
For consistency, set the default value for name field in search.package.Package to doocling-core,
since the field version is set to docling-core version by default.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-23 13:55:47 +02:00
github-actions[bot] f276691afd chore: bump version to 1.1.0 [skip ci] v1.1.0 2024-07-18 08:17:10 +00:00
Panos Vagenas d0ffc85e0c feat: add document Markdown export (#4)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-18 10:12:43 +02:00
github-actions[bot] 09d357613c chore: bump version to 1.0.0 [skip ci] v1.0.0 2024-07-17 16:01:31 +00:00
Michele Dolfi daece4ceae feat!: trigger v1.0.0 release (#3)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-07-17 17:59:54 +02:00
github-actions[bot] 3ff5f3e1cd chore: bump version to 0.0.1 [skip ci] v0.0.1 2024-07-17 15:48:18 +00:00
Michele DolfiandPanos Vagenas 4e5dbdf703 ci: add cicd (#1)
* single source of version

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add cicd

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply pre-commit validators

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* update lock

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-17 17:43:02 +02:00
Cesar Berrospi Ramis 656f56380f fix: fix definition issues in record type (#2)
* fix(test): set static typing compatible to python 3.10

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* style: enforce some pre-commit hooks on tests

Enforce pre-commit hooks black, isort, autoflake, and mypy on test modules.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* fix(rec): fix definition issues in attribute, predicate, subject

Remove duplicate generic types across base and predicate modules.
Create an identifier class for subject names.
Remove unnecessary type variables in attribute model.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* docs: refer to Docling data objects

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-17 17:24:31 +02:00
Cesar Berrospi Ramis 9134ffee93 Initial commit
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-15 12:55:20 +02:00