DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.
The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.
Fixes#685
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* fix: lazy load transformers in HybridChunker
Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(chunker): guard check at point of use
Guard import check at the point of use in hybrid chunker.
Follow the identical pattern for lazy load as in docling.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(chunker): add tests for lazy-loading with HybridChunker
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: normalize import calls from DoclingDocument
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* docs(page): add Field descriptions to page primitives
Adds Pydantic `Field(description=...)` to `TextCell`, `PdfPageGeometry`
and `SegmentedPage`, matching the `Annotated[..., Field(...)]` style
already used in `document.py`. Metadata only; no behavior change.
Closes#411
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* docs(page): clarify boundary_type scope, bitmap wording, has_* flag semantics
Address review feedback on the field descriptions:
- boundary_type: scope the computed-from-crop_bbox note to PdfPageGeometry,
since PageGeometry reads width/height/origin from self.rect.
- bitmap_resources: drop "embedded", which only covers one of the three
ImageRefMode states.
- has_chars/has_words/has_lines: state the process-vs-content distinction
explicitly, since the flag names read as a content claim.
Signed-off-by: Max Hsu <maxmilian@gmail.com>
---------
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* feat: forward remaining export_to_markdown kwargs through save_as_markdown
save_as_markdown already forwards most of export_to_markdown's parameters
but was missing five: enable_chart_tables, mark_annotations,
traverse_pictures, allowed_meta_names and blocked_meta_names. Add them as
keyword-only arguments with the same defaults as export_to_markdown and
forward them through, so save_as_markdown can drive the same output options
without dropping silently to defaults.
Defaults are unchanged, so existing behavior and tests are unaffected.
Fixes#619
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* refactor: drop mark_annotations and meta-name kwargs per review
Per review feedback, only expose enable_chart_tables and traverse_pictures
on save_as_markdown. mark_annotations is deprecated in MarkdownParams, and
allowed_meta_names/blocked_meta_names refer to old implementations that
should not be further promoted in the API.
Signed-off-by: Max Hsu <maxmilian@gmail.com>
---------
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* fix(doclang): enforce image size budgets for archive assets and ImageRef
- Cap `.dclx` `pages/` and `assets/` rasters with `max_image_decoded_size` before Pillow opens them
- Apply the same limit to Path and `file://` ImageRef loads
- Add tests covering oversize archive images and path-based ImageRefs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* bump doclang to 0.7.3 and clarify ImageRef remote-fetch guidance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- Cap DocLang XML input bytes before `parseString`
- Bound nesting depth and element count after DOM construction
- Expose limits via settings / `deserialize_str` kwargs and cover with unit tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- Cap per-member and total uncompressed size during safe archive extraction
- Apply limits to bytes actually read, not only declared ZipInfo sizes
- Expose the limits on `load_from_doclang_archive` and cover them with unit tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* fix(mypy): constrain numpy to '<2.5.0' to avoid syntax error
numpy 2.5.0 ships stubs that require Python 3.12 syntax, it can be installed into a 3.12 environment
that is being type-checked against a 3.10 language target.
Constraint numpy to '<2.5.0' for mypy compatibility checks.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: upgrade dependencies in uv.lock
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(dev): add doclang schematron-saxon extra to dev dependencies
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
DocumentOrigin rejects application/vnd.box.boxnote because it isn't in
Python's mimetypes registry, so the Box Note backend in docling
(docling-project/docling#3722) falls back to application/json. This
registers it in _extra_mimetypes so the backend can use the correct mime.
The list is also sorted alphabetically now.
Signed-off-by: pablopupo <145598901+pablopupo@users.noreply.github.com>
* feat: add universal-pathlib support for document to_*/from_* methods
Enable cloud storage backends (S3, GCS, Azure, etc.) for document
serialization methods via fsspec/universal-pathlib integration.
Signed-off-by: Nintorac <Nintorac@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(fsspec): remove universal-pathlib check
Remove the import check for universal-pathlib library and the decorators
'@pytest.mark.skipif' since it is unnecessary and adds complexity.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: remove potentially misleading comment
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(types): convert image URI path to AnyUrl for type safety
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(upath): add error handling for remote paths in 'relative_path()'
Catch error in case 'relative_path()' is used with remote paths and 'resolve' fails.
Consolidate the tests in test_upath_support.py module.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: misplaced block, annotations, tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor: deduplicate code in '_with_pictures_refs'
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Nintorac <Nintorac@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat(doclang): add .dclx archive save and load support
- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add support for `<description>` and `<summary>`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat: enable OpenDocument formats in DoclingDocument and allow the serialization of charts
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the labels and comments
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* What was removed
The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.
Deleted packages/modules
- docling_core/types/legacy_doc/ (whole folder)
- docling_core/types/rec/ (whole folder)
- docling_core/utils/legacy.py (legacy conversion functions)
- docling_core/utils/validate.py (RAW/ANN/OCR CLI)
Edited production code
- types/__init__.py — dropped the Record export
- utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
validate_unique_list/validate_datetime
- utils/generate_docs.py — dropped Record from MODELS
Removed resources/artifacts
- resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json
Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.
Verification
- No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
- from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
- Full project suite: 496 passed.
- generate_docs.py now emits only DoclingDocument.json and Generic.json.
The full plan is saved at _plans/remove-legacy-doc-and-rec.md.
Two notes worth flagging:
- Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
bump downstream.
- I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
approval flow.
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the checks.yaml
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the minimal_document_schema_flat.json
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/resources
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the files and updated the imports according to Cesar's suggestion
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the ruff linter
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- when merging documents, successive identical source names are deduplicated in the resulting document name (e.g. `a + a + b` → `a + b`)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
The regex in extract_inner_text used a lazy '.*?' with re.DOTALL, so a '<'
in plain text would match forward to the next '>' anywhere in the string,
silently dropping the span in between. This bit common scientific notation
such as 'P < 0.05 ... P > 0.05', where the whole comparison was deleted.
Require the tag to start with a letter or '/' and use '[^>]*' so a match can
never cross a '>' boundary. '<_..._>' tags remain preserved.
Fixes#618
Signed-off-by: ANIS <119749586+assinscreedFC@users.noreply.github.com>
* feat(doclang): align with DocLang v0.6 picture model
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): enhance picture metadata handling with custom fields
- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): deserialize field regions and add round-trip fixtures
- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* Early-abort HTTP source fetch when it is too large
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* Address review comments
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
---------
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
* test(doclang): add simple multi-page roundtrip test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add container-level threading
- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with v0.5 threading and heads
- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): switch to suppressing namespace and version by default
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align with DocLang 0.4
- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add basic virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] add virtual `<text>` support
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): extend virtual `<text>` support, unify el. head parsing
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): align code/picture labels to DocLang, add LabelMode
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): emit bare `<value>` for read-only field values
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] align with DocLang v0.5
- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage
- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* chore(doclang): [WIP] drop legacy vocabulary
- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>