mirror of
https://github.com/docling-project/docling-core.git
synced 2026-11-18 20:27:00 +00:00
dev/entity-data-point
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8951e6f202 |
chore: removing all legacy / deepsearch code (#644)
* What was removed The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them. Deleted packages/modules - docling_core/types/legacy_doc/ (whole folder) - docling_core/types/rec/ (whole folder) - docling_core/utils/legacy.py (legacy conversion functions) - docling_core/utils/validate.py (RAW/ANN/OCR CLI) Edited production code - types/__init__.py — dropped the Record export - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept validate_unique_list/validate_datetime - utils/generate_docs.py — dropped Record from MODELS Removed resources/artifacts - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts; replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/. Verification - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched). - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError. - Full project suite: 496 passed. - generate_docs.py now emits only DoclingDocument.json and Generic.json. The full plan is saved at _plans/remove-legacy-doc-and-rec.md. Two notes worth flagging: - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated bump downstream. - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit approval flow. Signed-off-by: Peter Staar <taa@zurich.ibm.com> * updated the checks.yaml Signed-off-by: Peter Staar <taa@zurich.ibm.com> * removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec Signed-off-by: Peter Staar <taa@zurich.ibm.com> * removed the minimal_document_schema_flat.json Signed-off-by: Peter Staar <taa@zurich.ibm.com> * removed docling_core/resources Signed-off-by: Peter Staar <taa@zurich.ibm.com> * removed the files and updated the imports according to Cesar's suggestion Signed-off-by: Peter Staar <taa@zurich.ibm.com> * fixed the ruff linter Signed-off-by: Peter Staar <taa@zurich.ibm.com> --------- Signed-off-by: Peter Staar <taa@zurich.ibm.com> |
||
|
|
2a78701b3f |
chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * docs: update CONTRIBUTING after pre-commit hook changes Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> --------- Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> |
||
|
|
c73904e68e |
style: replace black, isort, flake8 and autoflake with ruff (#456)
* Added ruff to dev dependencies Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added ruff settings to pyproject.toml as in docling Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Cleanup uf pyproject.toml Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Copied settings for ruff pre-commit hooks from docling Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Excluded test/data/** from ruff formatting / linting Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * ruff format Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added some ignore statements to pyproject.toml such that ruff check raises fewer issues Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * ruff check --fix Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Ignored some more rules Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Fixed the rest of the errors that would only concern 1 - 3 files Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added another ignore related to df for DataFrame names Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Modified CONTRIBUTING.md such that black / isort are replaced by ruff Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Added UP045 to ignore list such that Optional[...] does not raise Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Moved .flake8 configs to pyproject.toml Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Moved autoflake to be used with ruff Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Removed flake8 from .pre-commit hooks Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Adding PD901 to ignore list such that pre-commit hooks run fully again Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * Replaced dtype | None syntax by Optional[dtype] in remaining places Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> * chore: fix 'test' ref in pyproject Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove typing List, Set, Tuple, Dict Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove UP015 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove UP034 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: normalize dashes in comments and docstrings Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove PD901 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove C403 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove C403, C413, C416 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> * style: remove E203, F811 check from ignore list Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> --------- Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com> Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com> Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com> |
||
|
|
8c330ab81a |
fix(deps): switch to tree-sitter-java-orchard and expand typer compatibility (#473)
* switch to tree-sitter-java-orchard and work with newer typer Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> * update lock for typer Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> * use new lib Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> * ignore 3.9 issues Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> * lazy import tree-splitter languages Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> --------- Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> |
||
|
|
3097645198 |
feat: add code chunking functionality (#398)
* initial code chunking for docling-core * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62 Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * include language detections, add code chunking into hierarchical chunker * add serializer, internal marking of chunkers, typing * Update pyproject.toml Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com> Signed-off-by: Bridget <bridget.mcginn@ibm.com> * Update docling_core/transforms/chunker/hierarchical_chunker.py Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com> Signed-off-by: Bridget <bridget.mcginn@ibm.com> * run all pre-commit less pytest * update test files for code ID * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768 Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * update uv.lock Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * revert to stricter treesitter versioning due to compatibility Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18 I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86 I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36 I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2 Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * remove language detection (to be run by client, i.e. docling) Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * align new dependency specs Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * address backticks, ABC, and supported languages feedback Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * remove Language class and reuse CodeLanguageLabel Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: |