Files
docling-core/tests/data/doc/webvtt_example_02.json
738abeee00 feat: make inline serializers concatenate runs faithfully (#693)
* fix(doclang): preserve leading text before a lone formatting tag

`_get_children_simple_text_block` guarded against a second *text* fragment
but the Element branch just overwrote `result`, so
`<text>2<superscript>nd</superscript></text>` returned "nd" and the leading
`2` was silently discarded. Guard both branches so the element is routed to
`_parse_inline_group`, which already handles mixed children.

* feat: inline runs carry their own whitespace; serializers concatenate faithfully

The Markdown, HTML, LaTeX and DocLang inline serializers joined runs with a
hard `" "`, inventing spaces the model never asked for: `2nd` -> `2 nd`,
`O2.` -> `O 2 .`, `italic text.` -> `italic text .`. Those artifacts can never
be removed while the separator lives in the serializer.

Serializers:
- join inline parts with `""`; DocLang keeps its delimiter between the element
  head and the body only, where `<content>` is the whitespace channel
- hoist edge whitespace around the *complete* decoration stack (formatting and
  hyperlink together) for Markdown, HTML and LaTeX. Decorating it verbatim
  emits `**bold ** tail`, which CommonMark renders as literal asterisks;
  hoisting inside one formatter would let the hyperlink recapture it. DocLang
  opts out: it keeps whitespace inside the markup, in `<content>`.

DocLang deserializer:
- `_get_text` stripped every fragment and joined with `""`, collapsing
  `Advanced <bold>Topics</bold>` into `AdvancedTopics`. It now trims the outer
  block boundary only, and never trims whitespace delivered by `<content>`.
- bare text nodes use the standard XML heuristic: a whitespace run containing a
  newline is pretty-print indentation, one without it is content. This is what
  hand-written and VLM-emitted DocLang needs, since it will not use `<content>`.
- element-head children (`<caption>`, `<description>`, `<summary>`, `<custom>`)
  no longer bleed into body text, where they were also double-modelled in
  `item.meta`.

Data at rest: schema minor bumped to 1.11.0 and a `mode="before"` validator
re-injects the legacy separator for documents stamped below it, preserving their
previous *visible* output including its extra-space bugs (`H 2 O` stays
`H 2 O`). It must run before `check_version_is_compatible`, which rewrites
`version` and destroys the evidence. Versionless input is left alone. The
validator works on a copy, so the caller's dict is untouched and re-validating
the same raw value cannot compound separators.

A separator can never land inside a delimiter. Code and formula runs are wrapped
in `` ` ``/`$` by the serializers, so the space goes on the plain neighbour; when
both neighbours are delimited it becomes a plain run of its own, because
`` `a``b` `` and `$a$$b$` are both corrupt.

Chunk text changes too, since ChunkingDocSerializer inherits the Markdown join
-- anyone with a persisted vector index gets silent retrieval drift, so that
needs its own line in the release notes.

DocTags code is untouched. Its golden gains one empty `<text></text>` for the
legacy fixture because that document now carries an extra whitespace run, which
DocTags renders empty by its own pre-existing strip-per-run behaviour.

Ships as a minor, not a major: `feat!:` would make python-semantic-release
(7.34.6, both repos) bump to 3.0, and a 3.0 is not affordable on either package.

* DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 64404b3709d9e3ba7caf08957cf2ef19df29459f
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 66040cb7b96f77699c9d74051500d645e7e758ed

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(compat): register 1.11->1.10 projector for the whitespace-contract bump

Integrates the 1.11.0 schema bump with the downgrade-projector framework
(#703) landed upstream. 1.11 and 1.10 are structurally identical; the only
difference is that 1.11 inline runs own their boundary whitespace, so the
projector just re-stamps the version (a 1.10 client re-applies its own join,
yielding the documented harmless double space).

- add @register_projector(from_minor=11, to_minor=10) + TestProjector_1_11_to_1_10
- add the auto-generated 1.11 JSON Schema snapshot
- re-format the 1.11 SCHEMA_VERSION_HISTORY entry (ruff)
- update the stale dclq CLI expectation (old join produced 'public  class';
  faithful join now yields 'public class')

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(dclq): use typer's vendored click; require typer>=0.25

typer >=0.25 vendors its own copy of click (typer._click) and raises those
exceptions instead of the external click package's. The dclq CLI referenced
the external click, so on typer >=0.25 bad input escaped the exit-code
handlers and the context-flag callback (-C/-A/-B/--context-scope) silently
no-op'd because the isinstance(param, click.Option) check no longer matched.

Import the exception and Parameter types from typer._click, duck-type the
option check, raise the [dclq] typer floor to >=0.25, and drop the now-unused
external click dependency.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(doclang): keep inherited formatting on <content> runs in inline groups

<content> is the explicit whitespace-preserving text wrapper, so inside an
inline group it is a run, not a nested item: it must inherit the formatting
accumulated while descending through <bold>/<italic>/... . Dispatching it
through _dispatch_element dropped that formatting, which surfaced after the
rebase as <bold><content>x </content></bold> coming back unformatted and
broke the doclang round-trip for whitespace_inside_bold / _link.

Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>

* chore(compat): point schema 1.11.0 at the next minor release

The whitespace-contract bump was first written against v2.86.0; main is now
at v2.104.0, so record v2.105.0 as the placeholder release that ships schema
1.11.0. Adjust to the actual version at release time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* test: rebaseline goldens for the inline whitespace contract

Regenerated with DOCLING_GEN_TEST_DATA=1. The only change is the schema
version stamp 1.10.0 -> 1.11.0 in goldens added or moved on main
(including the six test/ -> tests/ renames and misplaced_list_items.out.yaml).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: join inline runs faithfully in the plain-text reader

_collect_subtree_text (TextItem._get_plain_text, FloatingItem.caption_text,
nested rich table cells) still joined the runs of an InlineGroup with " ",
so mixed captions read "Table 1 : results". It now concatenates the runs of
an inline group; every other join, including a host text item's own text
and its children, keeps its space. Children of a text item are block
structure (producers put a section's paragraphs under its heading), so the
host text stays apart from its inline group, as on main; add_inline_group
documents this.

Tests that asserted the invented run spacing now expect the contract
output, fixtures that built runs without boundary whitespace now carry it,
and a new test pins the host-text boundary for headings and captions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs: note that load_from_doctags cannot restore inline run whitespace

DocTags stores inline runs trimmed, so the boundary whitespace the inline
whitespace contract expects is not in the source. Document the limitation
instead of guessing separators back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(md): keep the host separator before a list item's inline group

The Markdown list serializer glued an inline group onto the preceding
list item with no separator, so a list item "Intro" whose child group
starts with "more" exported "- Intromore". Captions, headings and the
plain-text reader keep a space between host text and its inline group;
the list serializer now does the same when the group is the child of a
text item with its own text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): keep migrated separators between code or formula runs in page filters

A legacy document with two adjacent code or formula runs gets a standalone
" " run between them on load. It had no provenance, so filter(page_nrs=...)
and export_to_markdown(page_no=...) dropped it and rendered `$a$$b$`.

The separator now takes the left run's location when both runs are on the
same page, so it survives a page filter exactly when both runs do. Runs on
different pages keep the unlocated separator, which matches the old join.
Inline group locations in DocLang and DocTags are a union of the runs, so
they do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): count empty inline hosts and nested inline groups as legacy runs

The legacy separator migration counted only direct text children with
non-empty text as runs. An empty-text item whose content is a child inline
group (a formula or footnote host) and an inline group nested directly in
another were skipped, so `left`, host(`x`), `right` loaded as
`left $x$right` instead of the old `left $x$ right`. The DocLang
deserializer writes both shapes for nested mixed-content elements.

A child now counts as a run when it or its subtree has text. Only a plain
run with its own text takes the separator; an empty host must keep its
empty text to render its group, so it is treated like a code or formula run
and the separator stays outside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* docs(doclang): note that pre-contract archives load without legacy separators

DocLang has no schema version, so a .dclx written before schema 1.11.0
cannot be told apart from a current one and is not migrated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* chore(dclq): move the vendored-click fix to its own PR

This reverts 6a8b8374; the change is unrelated to the whitespace contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): don't run the legacy inline migration in project_to

project_to validated the projected dict under its target stamp. For 1.10.0 that ran the load-time separator migration on runs that already carry their whitespace, so 'Normal ', 'bold', ' tail' came back as 'Normal  ', 'bold ', ' tail'. Validate under the source stamp and restore the target stamp afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): strip inline runs when projecting 1.11 to 1.10

A 1.10 client joins inline runs with " ", so runs that keep their boundary whitespace rendered with double spaces and with the space inside the markup (**bold **, which CommonMark does not render as bold). The projector now strips each inline-group child's text to the shape old producers emitted. Whitespace-only runs become "", which old serializers skip, so no refs move. orig and text outside inline groups are left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(compat): reuse the stripped separator run when migrating

Between two code or formula runs the migration inserts a plain " " run. The 1.11 -> 1.10 projector strips it to "", and the next migration skipped the empty run and inserted another, so every projection round trip added a run. Refill an empty plain run between the two neighbours instead. Old serializers skipped empty runs, so legacy output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 7600953bc0
I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: f0e26b6722

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat(doclang): read and write whitespace per DocLang 0.9

- Deserializer applies spec rules to the DOM before parsing: a
  whitespace sequence reads as one space, none at block edges,
  <content> is literal.
- Serializer keeps each text block body on one line; text with
  non-single-space whitespace goes in <content>, unwrapped where
  the rules read it the same bare.
- Emit version 0.9; validate against doclang 0.9.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix(doclang): keep inline code and formula in virtual text

In a list item or cell with text of its own, <code> and <formula> are
inline, so whitespace around them reads as a space under DocLang 0.9.

- Reader applies the whitespace rules across them instead of treating
  them as blocks.
- Pretty-printer writes such a virtual text on one line, so it no
  longer adds spaces next to inline code or formulas.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* chore(compat): point schema 1.11.0 at v2.106.0

v2.105.0 was released without the whitespace contract, so record the
next minor release as the one that ships schema 1.11.0.

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* docs(compat): name schema 1.11 as current in the projectors header

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* fix(doclang): harden the DocLang 0.9 whitespace reader and writer

- read list items with leading inline code or formula as virtual text
- keep whitespace-only content between inline code runs
- keep blank lines in content when pretty printing
- keep inline handwriting and rtl text as runs
- trim only XML whitespace when reading text
- keep content literal in headings, captions and list items
- read table cells with inline elements as virtual text
- keep text beside a whitespace-only formatted run
- read spaces around an empty content as one sequence
- read a checkbox as part of virtual text
- keep a whitespace-only inline code run

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix(doclang): preserve checkbox spaces and formatted whitespace runs

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: nanhanq1 <157583847+nanhanq1@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2026-11-03 10:38:54 +01:00

269 lines
5.2 KiB
JSON
Vendored

{
"schema_name": "DoclingDocument",
"version": "1.11.0",
"name": "webvtt_example_02",
"origin": {
"mimetype": "text/vtt",
"binary_hash": 8584853280299071027,
"filename": "webvtt_example_02.vtt"
},
"furniture": {
"self_ref": "#/furniture",
"children": [],
"content_layer": "furniture",
"name": "_root_",
"label": "unspecified"
},
"body": {
"self_ref": "#/body",
"children": [
{
"$ref": "#/texts/0"
},
{
"$ref": "#/texts/1"
},
{
"$ref": "#/groups/0"
},
{
"$ref": "#/texts/5"
},
{
"$ref": "#/groups/1"
}
],
"content_layer": "body",
"name": "_root_",
"label": "unspecified"
},
"groups": [
{
"self_ref": "#/groups/0",
"parent": {
"$ref": "#/body"
},
"children": [
{
"$ref": "#/texts/2"
},
{
"$ref": "#/texts/3"
},
{
"$ref": "#/texts/4"
}
],
"content_layer": "body",
"name": "WebVTT cue span",
"label": "inline"
},
{
"self_ref": "#/groups/1",
"parent": {
"$ref": "#/body"
},
"children": [
{
"$ref": "#/texts/6"
},
{
"$ref": "#/texts/7"
},
{
"$ref": "#/texts/8"
}
],
"content_layer": "body",
"name": "WebVTT cue span",
"label": "inline"
}
],
"texts": [
{
"self_ref": "#/texts/0",
"parent": {
"$ref": "#/body"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 0.0,
"end_time": 2.0,
"voice": "Esme"
}
],
"orig": "It\u2019s a blue apple tree!",
"text": "It\u2019s a blue apple tree!"
},
{
"self_ref": "#/texts/1",
"parent": {
"$ref": "#/body"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 2.0,
"end_time": 4.0,
"voice": "Mary"
}
],
"orig": "No way!",
"text": "No way!"
},
{
"self_ref": "#/texts/2",
"parent": {
"$ref": "#/groups/0"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 4.0,
"end_time": 6.0,
"voice": "Esme"
}
],
"orig": "Hee!",
"text": "Hee!"
},
{
"self_ref": "#/texts/3",
"parent": {
"$ref": "#/groups/0"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 4.0,
"end_time": 6.0
}
],
"orig": " ",
"text": " "
},
{
"self_ref": "#/texts/4",
"parent": {
"$ref": "#/groups/0"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 4.0,
"end_time": 6.0
}
],
"orig": "laughter",
"text": "laughter",
"formatting": {
"bold": false,
"italic": true,
"underline": false,
"strikethrough": false,
"script": "baseline"
}
},
{
"self_ref": "#/texts/5",
"parent": {
"$ref": "#/body"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 6.0,
"end_time": 8.0,
"voice": "Mary"
}
],
"orig": "That\u2019s awesome!",
"text": "That\u2019s awesome!"
},
{
"self_ref": "#/texts/6",
"parent": {
"$ref": "#/groups/1"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 8.0,
"end_time": 10.0
}
],
"orig": "Sur les ",
"text": "Sur les "
},
{
"self_ref": "#/texts/7",
"parent": {
"$ref": "#/groups/1"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 8.0,
"end_time": 10.0
}
],
"orig": "playground",
"text": "playground",
"formatting": {
"bold": false,
"italic": true,
"underline": false,
"strikethrough": false,
"script": "baseline"
}
},
{
"self_ref": "#/texts/8",
"parent": {
"$ref": "#/groups/1"
},
"children": [],
"content_layer": "body",
"label": "text",
"source": [
{
"kind": "track",
"start_time": 8.0,
"end_time": 10.0
}
],
"orig": ", ici \u00e0 Montpellier",
"text": ", ici \u00e0 Montpellier"
}
],
"pictures": [],
"tables": [],
"key_value_items": [],
"form_items": [],
"pages": {}
}