Commit Graph
24 Commits
Author SHA1 Message Date
Cesar Berrospi Ramis a59c38c1da chore: rename test folder to tests (#739)
test: rename 'test' folder to 'tests'

To align with other Docling repositories, use 'tests' for test modules.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-09-04 10:49:52 +02:00
Christoph Auer cf2ac70632 feat: dclq tool (#696)
* feat: add initial dlgrep implementation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* refactor: migrate CLI to Typer

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: expand README

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: improve README examples

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: streamline dlgrep output

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: make text context output grep-compatible

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Improve XPath and inline context output

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* dlgrep support extensions

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Use git sources for doclang and docling-core

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: emit native JSON records

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove search class filter

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Make section output addressable

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Improve README examples

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: build and publish dlgrep from this repository

dlgrep becomes a uv workspace member under packages/dlgrep, published as
its own wheel and released in lockstep with docling-core: same version,
exact `docling-core==` pin, both wheels built from the same commit. This
removes the git-branch dependency dlgrep needed to track unreleased
docling-core changes.

- workspace wiring; ruff, mypy and pytest config inherited from the root,
  dlgrep in the dev group so `uv sync` covers it
- release.sh writes both versions and the pin from one TARGET_VERSION
- build-packages.sh builds into per-package dist dirs; pypi.yml publishes
  docling-core first, then dlgrep
- check_lockstep.py guards against version drift, which uv cannot catch on
  its own: the workspace source replaces the pin with a local path, so a
  wrong pin resolves and locks cleanly
- checks.yml and pypi.yml install both wheels together, which makes a
  mismatched pin fail before anything is published
- declare click and lxml, previously only transitive
- read __version__ from package metadata instead of a hardcoded literal
- flag dlgrep as experimental in its help output, since the shared version
  number would otherwise imply a maturity it does not have

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* ci: simplify lockstep release tooling

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore(dlgrep): drop handoff notes, keep only the CLI plan

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* refactor(doclang): isolate source mapping recorder

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* build(dlgrep): move dependencies to core extra

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(dlq): rename CLI and add list command

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(dclq): rename tool and consolidate plan

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(dclq): bound archive document reads

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-07-31 13:38:43 +02:00
Peter W. J. Staar 8951e6f202 chore: removing all legacy / deepsearch code (#644)
* What was removed

  The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.

  Deleted packages/modules
  - docling_core/types/legacy_doc/ (whole folder)
  - docling_core/types/rec/ (whole folder)
  - docling_core/utils/legacy.py (legacy conversion functions)
  - docling_core/utils/validate.py (RAW/ANN/OCR CLI)

  Edited production code
  - types/__init__.py — dropped the Record export
  - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
  validate_unique_list/validate_datetime
  - utils/generate_docs.py — dropped Record from MODELS

  Removed resources/artifacts
  - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json

  Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
  replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.

  Verification

  - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
  - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
  - Full project suite: 496 passed.
  - generate_docs.py now emits only DoclingDocument.json and Generic.json.

  The full plan is saved at _plans/remove-legacy-doc-and-rec.md.

  Two notes worth flagging:
  - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
  legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
  bump downstream.
  - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
  approval flow.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the checks.yaml

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the minimal_document_schema_flat.json

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/resources

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the files and updated the imports according to Cesar's suggestion

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the ruff linter

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-18 17:00:52 +02:00
Michele Dolfi 6a6512ebe4 docs(security): Document security processes (#606)
add SECURITY.md file

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-05-08 10:00:26 +02:00
mergify[bot] aaa1fbb031 ci(mergify): upgrade configuration to current format (#576)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-06 09:19:25 +02:00
Michele Dolfi 01373c4aab fix: drop python 3.9 and pin built tree-sitter versions (#487)
* drop python 3.9

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply linter for python 3.10 changes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2026-01-22 19:40:35 +01:00
Michele Dolfi 1a023dad6f chore: fix discord (#406)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-17 17:08:41 +02:00
Michele Dolfi 9fa02d08e3 chore: publish release notes on Discord (#405)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-17 16:31:40 +02:00
Michele Dolfi 47b70d3c24 feat: python 3.14 compatibility (#403)
update deps for python 3.14 compatibility

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-10-16 15:58:29 +02:00
Michele Dolfi 73113f6a43 ci: add dco-advisor (#331)
* add dco-advisor

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix yaml syntax

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* avoid js template syntax

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* search comment more robust

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* this commit is not signed-off on purpose

* update the logic for sha filtering

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* this commit is without sign-off

* add debug output

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* skip itself as it contains dco in name

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* restore pull_request on

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use pr checks

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix trigger

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* new status check

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* chore: DCO remediation for failing commits

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* DCO Remediation Commit for Michele Dolfi <dol@zurich.ibm.com>

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* more logs

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add output text to logs

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use logic scanning commits

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* print html_url

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix map function

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add username

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix body arg

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add link to dco details

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* DCO Remediation Commit for Michele Dolfi <dol@zurich.ibm.com>

I, Michele Dolfi <dol@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 5c1ef004ff
I, Michele Dolfi <dol@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 761f62fc3f

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use correct event

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-06-17 08:37:41 +02:00
Michele Dolfi d13e103237 chore: add dco settings (#329)
add dco settings

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-06-16 11:32:11 +02:00
1b0b39b0b4 refactor: use uv as dependencies management and packaging (#307)
* use new pyproject.toml format with uv

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* update ci/cd scripts

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* update MD files

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* build without pre-commit cache

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* run pre-commit from uv

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* small changes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add uv package install

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Apply suggestions from code review

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* docs: update README

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-05-28 13:06:06 +02:00
Cesar Berrospi Ramis 1af07218e1 fix(codecov): fix codecov argument and yaml file (#260)
* fix(codecov): fix codecov argument and yaml file

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* ci: set the codecov status to success even if the CI fails

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2025-04-15 16:53:33 +02:00
Michele Dolfi 64bafa1afe ci: add tests in ci with coverage upload (#257)
* add coverage of tests

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use coverage from pre-commit

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-04-15 10:08:07 +02:00
Michele Dolfi 8b5dd171a5 ci: fix new pypi build (#255)
fix new pypi build

Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
2025-04-15 06:24:33 +02:00
Michele Dolfi 37d545a1c8 ci: use trusted publishers and sign releases (#254)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-04-14 19:15:39 +02:00
Michele Dolfi 5739033408 chore: move to docling-project gh org (#190)
* chore: move to docling-project gh org

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* retrigger ci with DCO

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2025-03-14 09:28:17 +01:00
Panos Vagenas ee2d67b2ec chore: use GH cache for HF models (#183)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2025-03-04 13:12:44 +01:00
Panos Vagenas 4f653c2fb6 chore: add mergify config (#101)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-12-10 22:26:36 +01:00
Panos Vagenas 0cd8b94ca6 chore: switch release pipeline to manual triggering (#56)
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-24 13:02:39 +02:00
Michele Dolfi 5cc6ba73f7 chore: allow forks with synchronize event (#46)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-10-18 12:56:26 +02:00
03df97fa73 feat!: Expose DoclingDocument as main type, move old typing to legacy (#41)
* Fix area method of BoundingBox

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add image placeholder

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* enable picture label

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* refactor captions and markdown

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add logic to skip repeated caption

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* use DocItemLabel

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Extend default export labels, add convenience mehtods

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Introduce ListItem API, with marker and enumerated properties

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add classification and description in PictureData

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add molecule picture data

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DoclingDocument and aligned methods on legacy doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* add advanced picture data content

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Many markdown export fixes, renaming BaseTableData

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Rename module paths doc->legacy_doc, experimental->doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: imageref with pil_image

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Small fixes

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: remove documentation in markdown to support python 3.13 (#43)

Since json-schema-for-humans dependency does not support python 3.13,
remove the generation of documentation in markdown of main docling types.
Remove 'ds' prefix from documentation scripts.
Update README.
Add python 3.13 in CI/CD workflow checks.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* Fix TableCell model validator

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* store list of classes in classification

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Fixes for DocumentOrigin mimetype validation

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* introduce picturedata as list of annotations

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: adapt hierarchical chunker to v2 DoclingDocument

[skip-ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* feat: add table support in chunker, incl. captions

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* use Field constraints instead of conlist, refactor chunking types

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* revert unnecessary doc module change

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* align test data with upstream changes

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Update __init__.py on docling_core.types.doc

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove DescriptionItem

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-10-16 15:53:44 +02:00
Michele Dolfi 1a4440a771 chore: switch to gh apps user (#26)
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
2024-09-20 16:40:30 +02:00
Michele DolfiandPanos Vagenas 4e5dbdf703 ci: add cicd (#1)
* single source of version

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add cicd

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply pre-commit validators

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* update lock

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-17 17:43:02 +02:00