* feat(vtt): export and save to WebVTT format
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(vtt): omit empty blocks in parsing
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* test(vtt): add tests for exporting and saving to WebVTT
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor: move WebVTT data model from docling
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(webvtt): deal with HTML entities in cue text spans
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): support more WebVTT models
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(DoclingDocument): create a new provenance model for media file types
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): make WebVTTTimestamp public
Since WebVTTTimestamp is used in DoclingDocument, the class should be public.
Strengthen validation of cue language start tag annotation.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): set languages to a list of strings in ProvenanceTrack
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests(webvtt): add test for ProvenanceTrack
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): make all WebVTT classes public for reuse
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): preserve newlines as WebVTTLineTerminator
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): set ProvenanceTrack time fields as float
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): ensure start time offsets are in sequence
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): improve regex to remove note,region,style blocks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): parse the WebVTT file title
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(webvtt): rebase to latest changes in idoctags
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat(webvtt): add WebVTT serializer
Add a DoclingDocument serializer to WebVTT format.
Improve WebVTT data model.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(webvtt): add 'text/vtt' as extra mimetype
Add 'text/vtt' as extra MIME type to support WebVTT serialization, since it is not
supported by 'mimetypes' with python < 3.11
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): roll back DocItem.prov as list of ProvenanceItem
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* tests(webvtt): fix test with STYLE and NOTE blocks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* style(webvtt): apply X | Y annotation instead of Optional, Union
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): simplify TrackProvenance model with tags
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* refactor(webvtt): align class and field names to new 'source' type
Classes and fields that are related to the new source type should aign with their names.
The term 'provenance' will identify the legacy implementation.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* chore(DoclingDocument): drop the validation on field assignment
Drop the validation on field assignment in NodeItem objects.
Add the 'source' argument in the convenient function 'add_text' to create TextItem with track source data.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
refactor(webvtt): drop cue span classes, 'lang' and 'c' tags
Drop WebVTT formatting features not covered by Docling across formats.
Only 'u', 'b', 'i', and 'v' are supported and without classes.
Make 'v' tag explicit as 'voice' feature in SourceTrack class.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* feat(DocItem): Add comments field for linking annotations to document items
Implements support for linking comments (from Word/PPT documents) to their
annotated content using the established FloatingItem/RefItem pattern.
Changes:
- Add `comments: List[RefItem]` field to DocItem class
- Update `_update_breadth_first_with_lookup()` to handle comment references on deletion
- Bump CURRENT_VERSION to 1.9.0
- Fix version comparison bug (string vs integer for minor version)
- Add 4 new tests for comments functionality
- Update test data files for new schema
Closes: docling-project/docling#464
Related: docling-project/docling#2834
Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
* improve comment Pydantic serialization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* add add_comment, update tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* introduce fine-granular references with span ranges
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* simplify last test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
* Working on ISO standard for IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the pre-commit, still tons of work to do
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated idoctag token class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* need to start fixing the bugs and add proper tests for idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* refactored the testing for dcotags and idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updating the location tokens in idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the DocumentToken
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the idt quot issue
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* Added IDocTagsSerializationMode
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* working on the deserializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the captions for floating items
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* still need to fix some seserialization tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* finally all is working and reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* made the OTSL serialization and deserialization self-contained in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed some unclean code
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* did some fixes and clean up
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests, expanded to deserialize to deal with nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added the deserialization for formatting
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the formatting and the complex inline groups in nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added get_category
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* make IS_SELFCLOSING a set
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the regex
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the if ... return into if ... elif ... return
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the facets parsing
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the footnotes for Tables and Pictures in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated with latest doctags token table
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed private name
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: updating the idoctags serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: adding IDocTagsToken class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the list serializer in the doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added missing document serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added tests for content suppression
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests for serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ran pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* deal with differences between xml serialization in mac and linux
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (3)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (4)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixing the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed all the tests, except the no-content with lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the no-content in lists with doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added the Azure Document Intelligence
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the formatting and MyPy, working on the tests now
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added test document
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* replace all with pydantic dataclasses
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* minor azure serializer refactoring
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
* chore: leverage fixtures in tests and add type annotations
Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(markdown): serialize hyperlink as code always with single backticks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* initial code chunking for docling-core
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* include language detections, add code chunking into hierarchical chunker
* add serializer, internal marking of chunkers, typing
* Update pyproject.toml
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
* Update docling_core/transforms/chunker/hierarchical_chunker.py
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
* run all pre-commit less pytest
* update test files for code ID
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* update uv.lock
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* revert to stricter treesitter versioning due to compatibility
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* remove language detection (to be run by client, i.e. docling)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* align new dependency specs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* address backticks, ABC, and supported languages feedback
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* remove Language class and reuse CodeLanguageLabel
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* refactoring and improvements
- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
Cesar Berrospi RamisandMatteoOmenetti
Panos Vagenas
Panos VagenasandGitHub
MatteoandGitHub
Peter W. J. StaarandGitHub
Michele DolfiandGitHub
ErikandGitHub
