Commit Graph
18 Commits
Author SHA1 Message Date
Peter W. J. Staar 8951e6f202 chore: removing all legacy / deepsearch code (#644)
* What was removed

  The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.

  Deleted packages/modules
  - docling_core/types/legacy_doc/ (whole folder)
  - docling_core/types/rec/ (whole folder)
  - docling_core/utils/legacy.py (legacy conversion functions)
  - docling_core/utils/validate.py (RAW/ANN/OCR CLI)

  Edited production code
  - types/__init__.py — dropped the Record export
  - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
  validate_unique_list/validate_datetime
  - utils/generate_docs.py — dropped Record from MODELS

  Removed resources/artifacts
  - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json

  Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
  replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.

  Verification

  - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
  - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
  - Full project suite: 496 passed.
  - generate_docs.py now emits only DoclingDocument.json and Generic.json.

  The full plan is saved at _plans/remove-legacy-doc-and-rec.md.

  Two notes worth flagging:
  - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
  legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
  bump downstream.
  - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
  approval flow.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the checks.yaml

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the minimal_document_schema_flat.json

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/resources

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the files and updated the imports according to Cesar's suggestion

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the ruff linter

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-18 17:00:52 +02:00
Michele Dolfi 01373c4aab fix: drop python 3.9 and pin built tree-sitter versions (#487)
* drop python 3.9

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply linter for python 3.10 changes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-01-22 19:40:35 +01:00
Michele Dolfi 1a023dad6f chore: fix discord (#406)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-17 17:08:41 +02:00
Michele Dolfi 9fa02d08e3 chore: publish release notes on Discord (#405)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-17 16:31:40 +02:00
Michele Dolfi 47b70d3c24 feat: python 3.14 compatibility (#403)
update deps for python 3.14 compatibility

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-16 15:58:29 +02:00
Michele Dolfi 73113f6a43 ci: add dco-advisor (#331)
* add dco-advisor

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix yaml syntax

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* avoid js template syntax

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* search comment more robust

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* this commit is not signed-off on purpose

* update the logic for sha filtering

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* this commit is without sign-off

* add debug output

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* skip itself as it contains dco in name

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* restore pull_request on

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use pr checks

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix trigger

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* new status check

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* chore: DCO remediation for failing commits

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* DCO Remediation Commit for Michele Dolfi <dol@zurich.ibm.com>

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* more logs

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add output text to logs

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use logic scanning commits

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* print html_url

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix map function

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add username

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix body arg

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add link to dco details

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* DCO Remediation Commit for Michele Dolfi <dol@zurich.ibm.com>

I, Michele Dolfi <dol@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 5c1ef004ff
I, Michele Dolfi <dol@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 761f62fc3f

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use correct event

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-06-17 08:37:41 +02:00
1b0b39b0b4 refactor: use uv as dependencies management and packaging (#307)
* use new pyproject.toml format with uv

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* update ci/cd scripts

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* update MD files

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* build without pre-commit cache

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* run pre-commit from uv

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* small changes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add uv package install

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* docs: update README

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-28 13:06:06 +02:00
Cesar Berrospi Ramis 1af07218e1 fix(codecov): fix codecov argument and yaml file (#260)
* fix(codecov): fix codecov argument and yaml file

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* ci: set the codecov status to success even if the CI fails

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-04-15 16:53:33 +02:00
Michele Dolfi 64bafa1afe ci: add tests in ci with coverage upload (#257)
* add coverage of tests

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use coverage from pre-commit

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-04-15 10:08:07 +02:00
Michele Dolfi 8b5dd171a5 ci: fix new pypi build (#255)
fix new pypi build

Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
2025-04-15 06:24:33 +02:00
Michele Dolfi 37d545a1c8 ci: use trusted publishers and sign releases (#254)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-04-14 19:15:39 +02:00
Michele Dolfi 5739033408 chore: move to docling-project gh org (#190)
* chore: move to docling-project gh org

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* retrigger ci with DCO

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-03-14 09:28:17 +01:00
Panos Vagenas ee2d67b2ec chore: use GH cache for HF models (#183)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-04 13:12:44 +01:00
Panos Vagenas 0cd8b94ca6 chore: switch release pipeline to manual triggering (#56)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-24 13:02:39 +02:00
Michele Dolfi 5cc6ba73f7 chore: allow forks with synchronize event (#46)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-10-18 12:56:26 +02:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Michele Dolfi 1a4440a771 chore: switch to gh apps user (#26)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-20 16:40:30 +02:00
Michele DolfiandPanos Vagenas 4e5dbdf703 ci: add cicd (#1)
* single source of version

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add cicd

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply pre-commit validators

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* update lock

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-17 17:43:02 +02:00