DocLangDocSerializer raised "level must be in [1, 6]" for any
SectionHeaderItem with level 6. That is reachable in normal use, not a
synthetic edge case: SectionHeaderItem.level accepts up to 100, and
docling's heading-hierarchy inference emits level-6 headings by default,
so a PDF with nested procedure lists crashes the serializer.
The +1 shift (level 1 is reserved for TitleItem) pushed level 6 to 7,
past the heading token's allowed range, with nothing clamping it back.
html.py already clamps the identical item.level + 1 computation with
min(..., 6); this applies the same treatment, which is the smallest fix
and keeps the two serializers consistent.
Fixes#685
Signed-off-by: Max Hsu <maxmilian@gmail.com>
* fix: lazy load transformers in HybridChunker
Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(chunker): guard check at point of use
Guard import check at the point of use in hybrid chunker.
Follow the identical pattern for lazy load as in docling.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(chunker): add tests for lazy-loading with HybridChunker
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore: normalize import calls from DoclingDocument
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: ryyhan <dayel.rehan@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(doclang): enforce image size budgets for archive assets and ImageRef
- Cap `.dclx` `pages/` and `assets/` rasters with `max_image_decoded_size` before Pillow opens them
- Apply the same limit to Path and `file://` ImageRef loads
- Add tests covering oversize archive images and path-based ImageRefs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* bump doclang to 0.7.3 and clarify ImageRef remote-fetch guidance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- Cap DocLang XML input bytes before `parseString`
- Bound nesting depth and element count after DOM construction
- Expose limits via settings / `deserialize_str` kwargs and cover with unit tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add .dclx archive save and load support
- Add `DoclingDocument.save_as_doclang_archive()` and `load_from_doclang_archive()` for DocLang OPC archives (`.dclx`)
- Store markup in `document.xml` with page rasters under `pages/` and picture assets under `assets/`
- Resolve relative `<src uri="..."/>` paths when loading from an extracted archive
- Bump DocLang serialization to version `0.7` and update namespace/version golden fixtures
- Restore picture images from archive-relative URIs via `DocLangDocDeserializer` (`media_root`)
- Add safe zip extraction and archive path helpers in `docling_core.types.doc.utils`
- Promote `doclang` to a runtime dependency (`>=0.7,<0.8`) on PyPI 0.7.0
- Remove `doclang` from dev dependency group and refresh `uv.lock`
- Add `test/test_doclang_archive.py` with save, load, and roundtrip coverage
- Add two-page archive fixtures under `test/data/doc/doclang_archive/`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): add support for `<description>` and `<summary>`
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* What was removed
The legacy_doc document model and the rec (Record) types, plus all code/resources/tests that existed only to serve them.
Deleted packages/modules
- docling_core/types/legacy_doc/ (whole folder)
- docling_core/types/rec/ (whole folder)
- docling_core/utils/legacy.py (legacy conversion functions)
- docling_core/utils/validate.py (RAW/ANN/OCR CLI)
Edited production code
- types/__init__.py — dropped the Record export
- utils/validators.py — removed validate_schema/validate_raw_schema/validate_ann_schema/validate_ocr_schema and their now-unused imports; kept
validate_unique_list/validate_datetime
- utils/generate_docs.py — dropped Record from MODELS
Removed resources/artifacts
- resources/schemas/doc/ (legacy RAW/ANN/OCR/DOC), resources/schemas/generated/ccs_document_schema.json, docs/Record.json
Tests — deleted the 5 legacy/rec-only suites + 2 stale gold files; trimmed test_collection.py, test_base.py, test_doc_base.py to keep their non-legacy parts;
replaced the _create_hash import in test_code_chunker.py with a local helper. Removed test/data/legacy_doc/ and test/data/rec/.
Verification
- No dangling code references remain (only string literals and the vendored _reference/chunker_repo scratch data, intentionally untouched).
- from docling_core.types import DoclingDocument, Generic works; Record now correctly raises ImportError.
- Full project suite: 496 passed.
- generate_docs.py now emits only DoclingDocument.json and Generic.json.
The full plan is saved at _plans/remove-legacy-doc-and-rec.md.
Two notes worth flagging:
- Breaking public API: docling_core.types.Record and the docling_core.utils.legacy conversion functions were public — downstream docling uses
legacy_to_docling_document/docling_document_to_legacy. The branch name (dev/remove-legacy-document) suggests this is intended, but it'll need a coordinated
bump downstream.
- I left _reference/ and _plans/ alone (untracked scratch dirs). Nothing was committed — all changes are staged/working-tree only, per your edit-by-edit
approval flow.
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the checks.yaml
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/types/gen/, docling_core/types/nlp, docling_core/types/legacy_doc, docling_core/types/rec
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the minimal_document_schema_flat.json
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed docling_core/resources
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the files and updated the imports according to Cesar's suggestion
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the ruff linter
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
- clamp bbox coordinates during document validation and quantize location
tokens at page edges without warning
- emit warnings only when values
exceed a 1% page-dimension tolerance
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
- emit the DocLang version in serialized output by default
- pass `include_version=False` in tests that compare version-less fixtures
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): align with DocLang v0.6 picture model
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): enhance picture metadata handling with custom fields
- Introduced support for custom metadata fields in the picture model, including summary, description, and SMILES.
- Updated serializer and deserializer to handle new custom vocabulary tags.
- Added tests for round-trip serialization and deserialization of picture metadata, ensuring correct handling of custom fields.
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat(doclang): deserialize field regions and add round-trip fixtures
- Parse `<field_region>`, `<field_item>`, `<key>`, `<value>`, `<hint>`, and checkbox markup into `FieldRegionItem` / field text items, including inline key/value bodies
- Add `test_field_region_doclang_roundtrip` and `test_kv_annot_doclang_roundtrip` with materialized `input.json`, `deserialized.json`, and `reserialized.dclg.xml` goldens under `test/data/doc/field_region_kv*/` and `test/data/doc/kv/*/`
- Assert lossless XML round-trip where supported today (`_KV_ANNOT_XML_LOSSLESS`); partial cases remain golden-locked pending list `<group>` and in-region `<heading>` handling
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* feat: add keywords and topics meta fields
Introduce two simple, additive meta fields:
- KeywordsMetaField — unique list of keyword strings
- TopicsMetaField — unique list of topic / subject strings
Both attach to BaseMeta (node-level) and to the new DocumentMeta
(document-level), which is wired into DoclingDocument.meta. HTML,
Markdown, and Doclang serializers render the new fields as comma-
separated values. Schema bump is intentionally avoided since the
fields are optional and backward-compatible.
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* test: cover keywords and topics meta fields
Add round-trip, dedup, has_content, HTML escape, and Markdown
rendering tests for KeywordsMetaField, TopicsMetaField, and
DocumentMeta. Document-level meta absent-by-default is also covered.
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* style: satisfy ruff UP038 and import order
Convert tuple-form isinstance checks to PEP 604 unions and
reorder a misplaced import flagged by ruff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* fix: align with HTML attribute format from #613
Post-rebase onto main, the HTML metadata serializer now emits
`data-meta-name="<field>"` instead of valueless `data-meta-<field>`
and applies a single trailing html.escape pass. Drop the per-type
escape from the Keywords/Topics branch (was producing double
escaping after rebase) and update the two test assertions to the
new attribute shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* revert: remove DocLang keywords/topics serialization
Per discussion, DocLang is a standard whose extension is out of
scope for this PR. Drop the KeywordsMetaField/TopicsMetaField
branches from DoclangMetaSerializer and their imports; HTML and
Markdown serialization remain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
* refactor: address review feedback on keywords/topics meta fields
- Replace MetaFieldName inline comments with Attributes docstring (TableFormerMode pattern)
- Swap list[str]+manual dedup → UniqueList[str] in KeywordsMetaField and TopicsMetaField
- Add Field(description, examples) to all BaseMeta fields distinguishing entities/keywords/topics
- Remove DocumentMeta class and DoclingDocument.meta field; document-level meta goes via body
- Consolidate keywords/topics tests into test_semantic_base_meta_fields_roundtrip_and_html_rendering
- Regenerate docs/DoclingDocument.json
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(meta): reuse validator between keywords and topics
Revert the deduplication as part of a validator.
Share the validator between keywords and topics meta classes.
Adopt the standard Google style for enum classes.
Adapt the regression tests to the changes above.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Ana Daniele <ana.daniele@ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Ana Daniele <ana.daniele@ibm.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(html-serializer): emit XHTML-compatible metadata attributes
HTMLMetaSerializer previously rendered each metadata field as a valueless
HTML5 attribute (e.g. `<div data-meta-classification>`). Lenient HTML
parsers accept this, but the output is not valid XHTML/XML, so downstream
consumers that parse exported HTML as XML — including TEDS table
evaluation — fail with `xml.etree.ElementTree.ParseError: not well-formed
(invalid token)`.
Switch the attribute shape to a standard name=value pair
(`data-meta-name="<field>"`), HTML-escape both the field name and the
text body (`TabularChartMetaField` is left unescaped because its value is
an intentionally embedded HTML table), and update the embedded CSS
selectors and serialization ground-truth fixtures accordingly. Adds a
regression test that round-trips the output of `export_to_html` through
`xml.etree.ElementTree.fromstring`.
Fixes#598.
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
* fix(html-serializer): rebase changes to 'main' and add tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(html-serializer): remove redundant CSS rule
Remove duplicate CSS rules in 'html_styles.py' and ground-truth files.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Brian Wang <20699847+BrianWang1990@users.noreply.github.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
When a `TableItem` produced empty serialized text, `TripletTableSerializer`
was still populating the shared `visited` set via `RichTableCell._get_text()`.
Downstream items were then treated as already chunked and the chunker
emitted 0 chunks (e.g. on DOCX layout/header-only tables).
This change:
- Isolates `visited` while exporting the table dataframe and only merges
the local copy back into the shared set when non-empty text is produced.
- Promotes the header row into a data row when the dataframe has 0 data
rows but non-empty column names, so header-only tables still emit text.
- Handles single-row single-column tables by emitting the cell text
directly instead of an empty triplet list.
- Adds `_flatten_table_text` as a fallback for tables whose triplet form
collapses to empty text.
- Guards against appending empty text to serialization parts.
Adds regression tests covering header-only tables, single-row
single-column tables, end-to-end `HierarchicalChunker`/`HybridChunker`
runs on header-only / single-cell rich tables, and verifies that empty
tables do not consume their children's refs.
Closesdocling-project/docling#3335
Signed-off-by: Manohar Mallipudi <manohar.m4475@yahoo.com>
* feat: extending BaseMeta with language, entities, etc for docling-agent
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ignore serialization of empty Meta in HTML
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* addressing the good comments from Cesar
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat(serializer): add MsExcelMarkdownDocSerializer for sheet-name headings
Add `MsExcelMarkdownFallbackSerializer` and `MsExcelMarkdownDocSerializer`
to the serializer package so that `GroupLabel.SHEET` groups are rendered as
level-2 Markdown headings when exporting Excel-sourced documents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* DCO Remediation Commit for Smeet23 <smeetagrawal2003@gmail.com>
I, Smeet23 <smeetagrawal2003@gmail.com>, hereby add my Signed-off-by to this commit: 2a3808e5dc
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
* style: apply ruff formatter to markdown_excel.py
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
---------
Signed-off-by: Smeet23 <smeetagrawal2003@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: profile a document or collection
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(profiler): add deciles and histograms
Add deciles and histograms to the Docling collection statistics.
Add an example script to plot histograms.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(profiler): add option to plot log frequencies in histogram
Add the option to plot the histogram frequencies in logarithmic scale.
Extend README with documentation on the document profiler.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(profiler): cover missing lines in doc_profiler with tests
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests: add a pytest fixture to test LineBasedTokenChunker efficiently
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: apply style conventions to test_line_chunker.py test module
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: prevent hang in export_to_markdown() on nested RichTableCells
When a table cell contains another table (via RichTableCell.ref pointing
to a nested table item), MarkdownTableSerializer recursively invoked
itself through doc_serializer.serialize(). Each level of nesting caused
the outer tabulate() call to receive cell strings that were already
formatted markdown tables, which then got |-encoded and re-processed.
On documents with many such cells (e.g. Wikipedia pages with taxobox or
classification tables containing hyperlink-rich cells), this caused
exponential string growth, eventually exhausting memory or hanging.
Fix: add two module-level helpers:
_cell_content_has_table(item, doc) — walks the item subtree and
returns True if any node is a TableItem.
_mark_subtree_visited(item, doc, visited) — adds all nodes in the
subtree to the shared 'visited' set that the document serializer uses
to prevent double-emitting items.
In MarkdownTableSerializer.serialize(), when a RichTableCell's content
contains a nested table, call _mark_subtree_visited() on it and then use
col.text (the precomputed plain-text fallback) instead of calling
doc_serializer.serialize(). This avoids the recursive MarkdownTableSerializer
call entirely, and keeps the visited set consistent so the nested table is
not emitted again as a separate top-level item.
For RichTableCells whose content does NOT contain a nested table (e.g.
italic text, lists, hyperlinks), doc_serializer.serialize() is called as
before, so inline Markdown formatting is preserved.
Bug introduced in docling-core v2.46.0 (commit 1d04154) when RichTableCell
markdown serialization was first added.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* style: apply ruff formatting to markdown serializer
Keep the branch green in local and CI pre-commit runs by applying Ruff's required formatting in the nested rich table serializer path.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* test(markdown): cover descendant table detection in rich-cell helper
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): preserve nested table content as flattened text
Instead of falling back to the opaque col.text placeholder when a
RichTableCell contains a nested table, walk the subtree and collect
text from all grid cells and doc items. This preserves the actual
inner table content in the markdown output while still avoiding the
recursive doc_serializer.serialize() call that caused exponential
string growth and the OOM hang.
Signed-off-by: Ivan Traus <ivan@liminary.io>
* fix(markdown): use TextItem instead of DocItem in subtree text collector
Signed-off-by: Ivan Traus <ivan@liminary.io>
* refactor(markdown): flatten only nested tables, preserve sibling rich formatting
Address review feedback: replace the all-or-nothing _cell_content_has_table
branch with a _nested_in_table flag propagated through kwargs, so only
MarkdownTableSerializer flattens when nested — sibling rich elements (italic,
lists, etc.) keep their markdown formatting. Also replace getattr with
isinstance(NodeItem) + dot notation and remove string annotations.
Signed-off-by: Ivan Traus <ivan@liminary.io>
---------
Signed-off-by: Ivan Traus <ivan@liminary.io>
* feat(vtt): export and save to WebVTT format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(vtt): omit empty blocks in parsing
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(vtt): add tests for exporting and saving to WebVTT
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(chunker): propagate 'traverse_pictures' parameter from serializer to chunker
Propagate 'traverse_pictures' parameter from the 'serializer_provider' to the HierarchicalChunker, to ensure
that chunks include any TextItem children from PictureItem.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(chunker): simplify imports
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(serializer): add 'traverse_pictures' to CommonParams
Expose 'traverse_pictures' parameter to serialier common parameters to control
whether to traverse into PictureItem objects to serialize their children.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(serializer): add test for 'traverse_pictures' parameter
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(markdown): add an option to compact table serialization
Add an option in MarkdownTableSerializer to remove padding in tables in markdown serialization.
Propagate the option to MarkdownDocSerializer and DoclingDocument.
Remove unnecessary use of 'mode' argument in 'open' function when mode is 'r'.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(markdown): keep alignment marks in '_compact_table'
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor: move WebVTT data model from docling
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(webvtt): deal with HTML entities in cue text spans
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): support more WebVTT models
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(DoclingDocument): create a new provenance model for media file types
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): make WebVTTTimestamp public
Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): set languages to a list of strings in ProvenanceTrack
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests(webvtt): add test for ProvenanceTrack
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): make all WebVTT classes public for reuse
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): preserve newlines as WebVTTLineTerminator
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): set ProvenanceTrack time fields as float
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): ensure start time offsets are in sequence
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): improve regex to remove note,region,style blocks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): parse the WebVTT file title
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): rebase to latest changes in idoctags
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat(webvtt): add WebVTT serializer
Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(webvtt): add 'text/vtt' as extra mimetype
Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests(webvtt): fix test with STYLE and NOTE blocks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style(webvtt): apply X | Y annotation instead of Optional, Union
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): simplify TrackProvenance model with tags
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): align class and field names to new 'source' type
Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(DoclingDocument): drop the validation on field assignment
Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
refactor(webvtt): drop cue span classes, 'lang' and 'c' tags
Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat(IDocTags): add content wrapping for handling whitespace (#489)
* feat(IDocTags): add content wrapping for handling whitespace
Also:
- fix caption serialiazation
- consolidate indendation via `pretty_indentation` param
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* fix content tag value
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: add `minimal` parameter to get_row_bounding_boxes and get_column_bounding_boxes
Add option to return uniform bounding boxes instead of minimal ones:
- When minimal=False, rows have uniform horizontal extent (same x0/x1)
- When minimal=False, columns have uniform vertical extent (same y0/y1)
Also propagate the parameter to TableVisualizer via new params:
- minimal_row_bboxes
- minimal_col_bboxes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated the color of the row-section
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Added ruff to dev dependencies
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added ruff settings to pyproject.toml as in docling
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Cleanup uf pyproject.toml
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Copied settings for ruff pre-commit hooks from docling
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Excluded test/data/** from ruff formatting / linting
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* ruff format
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added some ignore statements to pyproject.toml such that ruff check raises fewer issues
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* ruff check --fix
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Ignored some more rules
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Fixed the rest of the errors that would only concern 1 - 3 files
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added another ignore related to df for DataFrame names
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Modified CONTRIBUTING.md such that black / isort are replaced by ruff
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Added UP045 to ignore list such that Optional[...] does not raise
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved .flake8 configs to pyproject.toml
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved autoflake to be used with ruff
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Moved all .flake8 settings to pyproject.toml to be compatible with ruff (i.e. no separate [tool.flake8] section
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Removed flake8 from .pre-commit hooks
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Applied ruff format (again); formatted some files as the line-length = 120 equals now what was set for the .flake8 settings
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Set max-complexity to 30 (as was originally) in the pyproject.toml as one linting check would fail
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Adding PD901 to ignore list such that pre-commit hooks run fully again
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* Replaced dtype | None syntax by Optional[dtype] in remaining places
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
* chore: fix 'test' ref in pyproject
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove typing List, Set, Tuple, Dict
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove UP015 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove UP034 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: normalize dashes in comments and docstrings
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove PD901 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove C403 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove C403, C413, C416 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style: remove E203, F811 check from ignore list
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Florian Schwarb <florian.schwarb@gmail.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Co-authored-by: Florian Schwarb <florian.schwarb@gmail.com>
Co-authored-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>