Commit Graph
591 Commits
Author SHA1 Message Date
Cesar Berrospi Ramis 0bd5c1fd4f test(meta): add examples with entities
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-27 22:30:15 +02:00
Cesar Berrospi Ramis f705bd41dd refactor(meta): 'value' in DataPointMention is now required
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-27 15:30:18 +02:00
Cesar Berrospi Ramis e0ff8526dd feat(meta): add DataPointMention for quantitative data points mentioned in text
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-27 15:06:45 +02:00
github-actions[bot] c357d5c6c4 chore: bump version to 2.88.0 [skip ci] v2.88.0 2026-07-27 11:53:54 +00:00
Max Hsu b869c67cf8 fix(doclang): clamp deep section headers instead of raising (#688)
DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.

The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.

Fixes #685

Signed-off-by: Max Hsu <maxmilian@gmail.com>
2026-07-27 13:48:31 +02:00
Akhil Trivedi 26a50ab140 fix(doclang): sanitize XML-illegal text characters (#687)
Signed-off-by: AkhilTrivediX <118957648+AkhilTrivediX@users.noreply.github.com>
2026-07-27 13:48:15 +02:00
Peter W. J. Staar 129811fe49 fix: DocLang deserialization of XML-sensitive OTSL cell content (#689)
fix: doclang deserialization

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-07-24 16:23:43 +02:00
Hamza Badreddine b7a4298b4f fix: accept application/epub+zip mimetype in DocumentOrigin (#692)
Signed-off-by: manticore999 <hamzabadreddine852@gmail.com>
2026-07-24 07:35:33 +02:00
Rehan KhanandCesar Berrospi Ramis 9ceb26665f fix: lazy load transformers in HybridChunker (#468)
* fix: lazy load transformers in HybridChunker

Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): guard check at point of use

Guard import check at the point of use in hybrid chunker.
Follow the identical pattern for lazy load as in docling.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(chunker): add tests for lazy-loading with HybridChunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: normalize import calls from DoclingDocument

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-22 11:01:30 +02:00
Max Hsu 2718adefc2 docs(page): add Field descriptions to page primitives (#681)
* docs(page): add Field descriptions to page primitives

Adds Pydantic `Field(description=...)` to `TextCell`, `PdfPageGeometry`
and `SegmentedPage`, matching the `Annotated[..., Field(...)]` style
already used in `document.py`. Metadata only; no behavior change.

Closes #411

Signed-off-by: Max Hsu <maxmilian@gmail.com>

* docs(page): clarify boundary_type scope, bitmap wording, has_* flag semantics

Address review feedback on the field descriptions:

- boundary_type: scope the computed-from-crop_bbox note to PdfPageGeometry,
  since PageGeometry reads width/height/origin from self.rect.
- bitmap_resources: drop "embedded", which only covers one of the three
  ImageRefMode states.
- has_chars/has_words/has_lines: state the process-vs-content distinction
  explicitly, since the flag names read as a content claim.

Signed-off-by: Max Hsu <maxmilian@gmail.com>

---------

Signed-off-by: Max Hsu <maxmilian@gmail.com>
2026-07-20 16:37:21 +02:00
Max Hsu fb9ffbd437 feat: add enable_chart_tables and traverse_pictures to save_as_markdown (#662)
* feat: forward remaining export_to_markdown kwargs through save_as_markdown

save_as_markdown already forwards most of export_to_markdown's parameters
but was missing five: enable_chart_tables, mark_annotations,
traverse_pictures, allowed_meta_names and blocked_meta_names. Add them as
keyword-only arguments with the same defaults as export_to_markdown and
forward them through, so save_as_markdown can drive the same output options
without dropping silently to defaults.

Defaults are unchanged, so existing behavior and tests are unaffected.

Fixes #619

Signed-off-by: Max Hsu <maxmilian@gmail.com>

* refactor: drop mark_annotations and meta-name kwargs per review

Per review feedback, only expose enable_chart_tables and traverse_pictures
on save_as_markdown. mark_annotations is deprecated in MarkdownParams, and
allowed_meta_names/blocked_meta_names refer to old implementations that
should not be further promoted in the API.

Signed-off-by: Max Hsu <maxmilian@gmail.com>

---------

Signed-off-by: Max Hsu <maxmilian@gmail.com>
2026-07-20 15:05:43 +02:00
github-actions[bot] 0215808e9b chore: bump version to 2.87.1 [skip ci] v2.87.1 2026-07-15 09:10:55 +00:00
Panos Vagenas 224eeeab60 fix(doclang): enforce image size budgets for archive assets and ImageRef (#684)
* fix(doclang): enforce image size budgets for archive assets and ImageRef

- Cap `.dclx` `pages/` and `assets/` rasters with `max_image_decoded_size` before Pillow opens them
- Apply the same limit to Path and `file://` ImageRef loads
- Add tests covering oversize archive images and path-based ImageRefs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* bump doclang to 0.7.3 and clarify ImageRef remote-fetch guidance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-07-15 10:47:13 +02:00
Panos Vagenas aee0674a28 fix(doclang): add deserialize size, depth, and element budgets (#683)
- Cap DocLang XML input bytes before `parseString`
- Bound nesting depth and element count after DOM construction
- Expose limits via settings / `deserialize_str` kwargs and cover with unit tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-07-14 15:49:17 +02:00
Cesar Berrospi Ramisandhardik-kansal 7775df1176 fix(chunker): preserve table structure across chunks in markdown (#678)
* fix(chunker): preserve markdown table structure when splitting across chunks

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: extract shared fixtures and serializer providers in test_hybrid_chunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: replace inline comments with docstrings in segment and get_header_and_body_lines

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs(chunker): fix misleading docstrings in LineBasedTokenChunker

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: normalize import calls from DoclingDocument

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(chunker): correctly identify table preamble when caption starts with '|'

Co-Authored-By: hardik-kansal <hkhardik4321@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: hardik-kansal <hkhardik4321@gmail.com>
2026-07-14 10:50:03 +02:00
github-actions[bot] 9e9c28f309 chore: bump version to 2.87.0 [skip ci] v2.87.0 2026-07-11 11:36:02 +00:00
Panos Vagenas 994881b8d4 fix(doclang): enforce size limits when extracting .dclx archives (#682)
- Cap per-member and total uncompressed size during safe archive extraction
- Apply limits to bytes actually read, not only declared ZipInfo sizes
- Expose the limits on `load_from_doclang_archive` and cover them with unit tests

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-07-10 17:47:22 +02:00
Peter W. J. Staar f8a36e9a10 feat: split the monolithic document.py into focused modules (#664)
refactored the docling-core for the document.py

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-07-09 14:39:46 +02:00
maxandmaxtaran2010 500a6e7d12 docs: fix typos in docstrings and comments (#677)
Corrects several misspellings in docstrings and inline comments:
- explicity -> explicit (ProvenanceItem docstring + generated schema)
- horisontal -> horizontal
- ommitted -> omitted
- to serializes -> to serialize
- succesive -> successive
- questionaires -> questionnaires
- optionaly -> optionally

No functional changes; identifiers and enum values are untouched.

Signed-off-by: maxtaran2010 <ocotifuzo727@gmail.com>
Co-authored-by: maxtaran2010 <ocotifuzo727@gmail.com>
2026-07-09 08:49:20 +02:00
Cesar Berrospi Ramis a6b60238df test(utils): mock network calls to make URL-related tests offline-safe (#671)
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-07 13:35:10 +02:00
Cesar Berrospi Ramis e71da6bf05 fix(deps): constrain numpy <2.5.0 and add doclang schematron-saxon to dev (#669)
* fix(mypy): constrain numpy to '<2.5.0' to avoid syntax error

numpy 2.5.0 ships stubs that require Python 3.12 syntax, it can be installed into a 3.12 environment
that is being type-checked against a 3.10 language target.
Constraint numpy to '<2.5.0' for mypy compatibility checks.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: upgrade dependencies in uv.lock

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(dev): add doclang schematron-saxon extra to dev dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-03 17:36:00 +02:00
github-actions[bot] c3177448b3 chore: bump version to 2.86.0 [skip ci] v2.86.0 2026-07-03 11:20:50 +00:00
Pablo Pupo 50710f16b3 chore(document): register the Box Note mime type (#668)
DocumentOrigin rejects application/vnd.box.boxnote because it isn't in
Python's mimetypes registry, so the Box Note backend in docling
(docling-project/docling#3722) falls back to application/json. This
registers it in _extra_mimetypes so the backend can use the correct mime.

The list is also sorted alphabetically now.

Signed-off-by: pablopupo <145598901+pablopupo@users.noreply.github.com>
2026-07-03 10:38:34 +02:00
NintoracandCesar Berrospi Ramis be42fcea2e feat: add fsspec support (#444)
* feat: add universal-pathlib support for document to_*/from_* methods

Enable cloud storage backends (S3, GCS, Azure, etc.) for document
serialization methods via fsspec/universal-pathlib integration.

Signed-off-by: Nintorac <Nintorac@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(fsspec): remove universal-pathlib check

Remove the import check for universal-pathlib library and the decorators
'@pytest.mark.skipif' since it is unnecessary and adds complexity.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* chore: remove potentially misleading comment

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix(types): convert image URI path to AnyUrl for type safety

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test(upath): add error handling for remote paths in 'relative_path()'

Catch error in case 'relative_path()' is used with remote paths and 'resolve' fails.
Consolidate the tests in test_upath_support.py module.

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* fix: misplaced block, annotations, tests

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* refactor: deduplicate code in '_with_pictures_refs'

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Nintorac <Nintorac@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-07-02 16:36:19 +02:00
github-actions[bot] cf8648b6f1 chore: bump version to 2.85.0 [skip ci] v2.85.0 2026-06-25 16:15:48 +00:00
Panos Vagenas 3704874312 feat(doclang): add support for .dclx archive, <description>, <summary> (#658)
* feat(doclang): add .dclx archive save and load support

- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): add support for `<description>` and `<summary>`

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-25 18:06:10 +02:00
Cesar Berrospi Ramis ecb3e84220 fix(document): preserve FloatingItem descendants in _hierarchize() (#646)
* fix(document): preserve FloatingItem descendants in _hierarchize()

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* test: reduce verbosity in test_hierarchize_all_docs

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-24 10:18:49 +02:00
Michele-Zhu 9ba55c2456 perf: string join instead of concatenation (#655)
The join method is more efficient than the string concatenator

Signed-off-by: Michele-Zhu <71699139+Michele-Zhu@users.noreply.github.com>
2026-06-23 17:01:02 +02:00
github-actions[bot] dff585385a chore: bump version to 2.84.0 [skip ci] v2.84.0 2026-06-23 09:44:00 +00:00
Peter W. J. Staar dc0fc7d020 feat: enable OpenDocument formats in DoclingDocument and allow the serialization of charts (#654)
* feat: enable OpenDocument formats in DoclingDocument and allow the serialization of charts

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the labels and comments

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-23 11:39:29 +02:00
github-actions[bot] ef5091295e chore: bump version to 2.83.1 [skip ci] v2.83.1 2026-06-19 05:45:12 +00:00
Christoph Auer 246f131aff fix: Add legacy stubs to unbreak old docling version imports (#649)
Add legacy shims to unbreak old docling version imports

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-06-19 07:41:21 +02:00
github-actions[bot] ce0523ea84 chore: bump version to 2.83.0 [skip ci] v2.83.0 2026-06-18 15:33:58 +00:00
Peter W. J. Staar 8951e6f202 chore: removing all legacy / deepsearch code (#644)
* What was removed

  The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.

  Deleted packages/modules
  - docling_core/types/legacy_doc/ (whole folder)
  - docling_core/types/rec/ (whole folder)
  - docling_core/utils/legacy.py (legacy conversion functions)
  - docling_core/utils/validate.py (RAW/ANN/OCR CLI)

  Edited production code
  - types/__init__.py — dropped the Record export
  - utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
  validate_unique_list/validate_datetime
  - utils/generate_docs.py — dropped Record from MODELS

  Removed resources/artifacts
  - resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json

  Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
  replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.

  Verification

  - No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
  - from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
  - Full project suite: 496 passed.
  - generate_docs.py now emits only DoclingDocument.json and Generic.json.

  The full plan is saved at _plans/remove-legacy-doc-and-rec.md.

  Two notes worth flagging:
  - Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
  legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
  bump downstream.
  - I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
  approval flow.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the checks.yaml

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the minimal_document_schema_flat.json

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed docling_core/resources

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* removed the files and updated the imports according to Cesar's suggestion

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the ruff linter

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
2026-06-18 17:00:52 +02:00
sriharanandsriharan2005@Tamil-- 8ad52a3596 fix: handle page-less documents in table doctags export (#632)
* fix: handle page-less documents in table doctags export

* DCO Remediation Commit for sriharan2005@Tamil-- <sriharan0804@users.noreply.github.com>

I, sriharan2005@Tamil-- <sriharan0804@users.noreply.github.com>, hereby add my Signed-off-by to this commit: d9808022a5

Signed-off-by: sriharan2005@Tamil-- <sriharan0804@users.noreply.github.com>

* style: fix import ordering

Signed-off-by: sriharan2005@Tamil-- <sriharan0804@users.noreply.github.com>

---------

Signed-off-by: sriharan2005@Tamil-- <sriharan0804@users.noreply.github.com>
Co-authored-by: sriharan2005@Tamil-- <sriharan0804@users.noreply.github.com>
2026-06-18 14:28:35 +02:00
Panos Vagenas 9dddcf7137 fix: clamp provenance bboxes and DocLang locations to valid page bounds (#643)
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:44:20 +02:00
Panos Vagenas 6fd2c1a247 fix: collapse consecutive names in concatenate output (#645)
- when merging documents, successive identical source names are deduplicated in the resulting document name (e.g. `a + a + b` → `a + b`)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:25:37 +02:00
Panos Vagenas 34bd98eb3d feat(doclang): default include_version to True (#642)
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-18 10:20:20 +02:00
Max SwainandChristoph Auer ab3666c081 perf: Use OpenCV over PIL for PNG encoding in ImageRef.from_pil (#562)
* perf: Use OpenCV over PIL for PNG encoding

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>

* refactor: Make opencv optional and add fallback to PIL

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>

* refactor: Update opencv dependency to use dev group

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>

* Fix RGBA vs RGB handling in openCV

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update test GT

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* empty commit to trigger DCO

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Max Swain <89113255+maxdswain@users.noreply.github.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
2026-06-17 16:44:14 +02:00
ANIS 7e4a2729e3 fix(doctags): stop extract_inner_text deleting text between < and > (#627)
The regex in extract_inner_text used a lazy '.*?' with re.DOTALL, so a '<'
in plain text would match forward to the next '>' anywhere in the string,
silently dropping the span in between. This bit common scientific notation
such as 'P < 0.05 ... P > 0.05', where the whole comparison was deleted.

Require the tag to start with a letter or '/' and use '[^>]*' so a match can
never cross a '>' boundary. '<_..._>' tags remain preserved.

Fixes #618

Signed-off-by: ANIS <119749586+assinscreedFC@users.noreply.github.com>
2026-06-17 14:09:05 +02:00
Cesar Berrospi Ramis 2a78701b3f chore: update pre-commit hooks and development dependencies (#638)
* chore: update pre-commit hooks and development dependencies

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* docs: update CONTRIBUTING after pre-commit hook changes

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-06-15 17:07:34 +02:00
github-actions[bot] e3360a0562 chore: bump version to 2.82.0 [skip ci] v2.82.0 2026-06-12 08:57:42 +00:00
Panos Vagenas 0b7690509e feat(doclang): align with DocLang v0.6, add field region deser. (#636)
* feat(doclang): align with DocLang v0.6 picture model

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): enhance picture metadata handling with custom fields

- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* feat(doclang): deserialize field regions and add round-trip fixtures

- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-11 12:30:09 +02:00
github-actions[bot] d3633168ef chore: bump version to 2.81.0 [skip ci] v2.81.0 2026-06-10 14:25:19 +00:00
Christoph Auer d585a13775 feat: Make HTTP downloads respect file-size limits and abort early (#635)
* Early-abort HTTP source fetch when it is too large

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Address review comments

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-06-10 13:38:06 +02:00
Christoph Auer 7b47b5a506 fix: externalize page images in REFERENCED mode for JSON/YAML export (#634)
On JSON export, control inclusion of page images for ImageRefMode.REFERENCED

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
2026-06-10 10:32:40 +02:00
github-actions[bot] f90b48f8f3 chore: bump version to 2.80.0 [skip ci] v2.80.0 2026-06-09 13:38:14 +00:00
Panos Vagenas 5d740c260a feat(doclang): promote serializer and deserializer (#633)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 15:23:21 +02:00
Panos Vagenas ba110e8671 chore(doclang): add v0.5 container-level threading (#631)
* test(doclang): add simple multi-page roundtrip test

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add container-level threading

- Emit threaded fragments with page breaks for multi-prov text, list groups, and tables
- Merge continuation fragments on deserialize using a (thread_id, host) registry
- Allocate document-scoped auto-increment thread IDs to satisfy spec uniqueness rules
- Add serialization and round-trip fixtures for paragraph, list, and table threading
- Suppress duplicate document-level page breaks when list/table serializers embed their own

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with v0.5 threading and heads

- Emit fragment-scoped OTSL for multi-prov tables using continuation tokens (`lcel`/`ucel`/`xcel`) instead of duplicating full grids, and merge fragments on deserialize with correct offsets and `TableData` dimensions.
- Fix element-head order to match the XSD (`label` → `thread` → `href` → `layer` → `location` → `caption` → `custom`) and add coverage for href/layer ordering.
- Drop legacy `unknown` label handling in favor of the v0.5 `undefined` default, adjusting both serializer and deserializer behavior plus goldens.
- Update cross-page/column table factories and fixtures to use proper 2×2 layouts and the new slice-based threading semantics.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-09 11:17:14 +02:00
Panos Vagenas 28342dcc6c chore(doclang): align with DocLang v0.5 foundation (#630)
* chore(doclang): switch to suppressing namespace and version by default

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align with DocLang 0.4

- Align DocLang serialization/deserialization with the doclang 0.4 schema (`<label value>`, `<layer value>`, `<href uri>`, `<src uri>`, element-head property order).
- Auto-wrap orphan key/value markup in `<field_item>` / `<field_region>` when the document tree is missing those containers.
- Add optional `doclang-validation` extra and `verify_doclang()` so golden tests also pass the reference `doclang` validator.
- Refresh `.dclg.xml` fixtures and add coverage for hyperlinks, field-region wrapping, and known incompat output under `test/data/doc/incompat/`.
- Pin emitted DocLang version to 0.4 and defer `<layer>`-related expectations until doclang 0.5.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add basic virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] add virtual `<text>` support

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): extend virtual `<text>` support, unify el. head parsing

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): align code/picture labels to DocLang, add LabelMode

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): emit bare `<value>` for read-only field values

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] align with DocLang v0.5

- Bump doclang validation to 0.5 and enable <layer> emission.
- Align open-tag shapes with the recommended vocabulary: bare <heading>/<field_heading> at level 1, bare <list> for unordered lists.
- Add field_heading deserializer support and remove deferred layer xfails.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): add `<rtl>`/`<corn>`/`<xcel>` coverage

- Emit <rtl> for RTL script text via get_text_direction()
- Emit <corn> for row/column header intersection cells
- Fix OTSL span tokens so <xcel> only appears at interior continuations
- Add rtl/corn/xcel golden tests; defer multi-prov threading test to P2

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(doclang): [WIP] drop legacy vocabulary

- Remove non-spec tokens (`section`, `watermark`, `time_break`, `facets`, `meta`, `uri`) and `GROUP@type` / `TABLE@class` from vocabulary
- Reject `class` on `<table>` at deserialize time; resolve label from `<table>` vs `<index>` only
- Drop legacy `class="data"` / `class="index"` table parsing paths and update deserializer tests accordingly
- Tighten `WRAP_WHEN_NEEDED` comment to include embedded newlines

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
2026-06-08 16:42:35 +02:00