Commit Graph
7 Commits
Author SHA1 Message Date
Peter W. J. Staar 8951e6f202 chore: removing all legacy / deepsearch code (#644)
* What was removed

  The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.

  Deleted packages/modules
  - docling_core/types/legacy_doc/ (whole folder)
  - docling_core/types/rec/ (whole folder)
  - docling_core/utils/legacy.py (legacy conversion functions)
  - docling_core/utils/validate.py (RAW/ANN/OCR CLI)

  Edited production code
  - types/__init__.py — dropped the Record export
  - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
  validate_unique_list/validate_datetime
  - utils/generate_docs.py — dropped Record from MODELS

  Removed resources/artifacts
  - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json

  Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
  replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.

  Verification

  - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
  - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
  - Full project suite: 496 passed.
  - generate_docs.py now emits only DoclingDocument.json and Generic.json.

  The full plan is saved at _plans/remove-legacy-doc-and-rec.md.

  Two notes worth flagging:
  - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
  legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
  bump downstream.
  - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
  approval flow.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the checks.yaml

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the minimal_document_schema_flat.json

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/resources

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the files and updated the imports according to Cesar's suggestion

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the ruff linter

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-18 17:00:52 +02:00
c73904e68e style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added ruff settings to pyproject.toml as in docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Cleanup uf pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Copied settings for ruff pre-commit hooks from docling

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Excluded test/data/** from ruff formatting / linting

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff format

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* ruff check --fix

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Ignored some more rules

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Fixed the rest of the errors that would only concern 1 - 3 files

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added another ignore related to df for DataFrame names

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Modified CONTRIBUTING.md such that black / isort are replaced by ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Added UP045 to ignore list such that Optional[...] does not raise

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved .flake8 configs to pyproject.toml

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved autoflake to be used with ruff

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Removed flake8 from .pre-commit hooks

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Adding PD901 to ignore list such that pre-commit hooks run fully again

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* Replaced dtype | None syntax by Optional[dtype] in remaining places

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>

* chore: fix 'test' ref in pyproject

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove typing List, Set, Tuple, Dict

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP015 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove UP034 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: normalize dashes in comments and docstrings

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove PD901 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove C403, C413, C416 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: remove E203, F811 check from ignore list

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-01-13 17:03:10 +01:00
Peter W. J. Staar 928aaed9ab chore: remove unnecesary copyright headers (#440)
* chore: remove unnecesary copyright headers

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted, still need to fix a test ...

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2025-11-27 11:17:23 +01:00
Cesar Berrospi Ramis a4ddd142ee fix: change order of JSON Schema to search mapper transformations (#32)
* fix: change order of JSON Schema to search mapper transformations

In the JSON Schema to search mapper, the suppression flags need to be addressed first,
otherwise the jsonref.replace_refs function may remove them.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* build: update dependencies

Updating pydantic from 2.8.2 to 2.9.2 triggers a change in JSON Schema from models:
the  lists with only 1 element get flatten.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

* chore: improve verbose in JSON Schema to search mapper test

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>

---------

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-09-26 13:49:18 +02:00
Cesar Berrospi Ramis 7b9e81f979 chore: set default name field in Package to docling-core (#5)
For consistency, set the default value for name field in search.package.Package to doocling-core,
since the field version is set to docling-core version by default.

Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-23 13:55:47 +02:00
Michele DolfiandPanos Vagenas 4e5dbdf703 ci: add cicd (#1)
* single source of version

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* add cicd

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* apply pre-commit validators

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* update lock

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
2024-07-17 17:43:02 +02:00
Cesar Berrospi Ramis 9134ffee93 Initial commit
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
2024-07-15 12:55:20 +02:00