* feat(DocItem): Add comments field for linking annotations to document items
Implements support for linking comments (from Word/PPT documents) to their
annotated content using the established FloatingItem/RefItem pattern.
Changes:
- Add `comments: List[RefItem]` field to DocItem class
- Update `_update_breadth_first_with_lookup()` to handle comment references on deletion
- Bump CURRENT_VERSION to 1.9.0
- Fix version comparison bug (string vs integer for minor version)
- Add 4 new tests for comments functionality
- Update test data files for new schema
Closes: docling-project/docling#464
Related: docling-project/docling#2834
Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
* improve comment Pydantic serialization
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* add add_comment, update tests
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* introduce fine-granular references with span ranges
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* simplify last test
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: s1v4-d <leelasaisivasubrahmanyamdurga@gmail.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
* Working on ISO standard for IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the pre-commit, still tons of work to do
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated idoctag token class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* need to start fixing the bugs and add proper tests for idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* refactored the testing for dcotags and idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updating the location tokens in idoctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the DocumentToken
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the idt quot issue
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* Added IDocTagsSerializationMode
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* working on the deserializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the captions for floating items
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* still need to fix some seserialization tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* finally all is working and reformatted
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* made the OTSL serialization and deserialization self-contained in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed some unclean code
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* did some fixes and clean up
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests, expanded to deserialize to deal with nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added the deserialization for formatting
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the formatting and the complex inline groups in nested lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added get_category
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* make IS_SELFCLOSING a set
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the regex
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the if ... return into if ... elif ... return
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the facets parsing
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the footnotes for Tables and Pictures in IDocTags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* updated with latest doctags token table
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed private name
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: updating the idoctags serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* feat: adding IDocTagsToken class
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fix the get_special_tokens (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the doctags serializer: if the no_content flag is true, no double list_item, no content in tables and captions
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the list serializer in the doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added missing document serializer
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added tests for content suppression
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests for serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added extra tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* ran pre-commit
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* deal with differences between xml serialization in mac and linux
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (2)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (3)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* revert to old doctags serialization (4)
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixing the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed all the tests, except the no-content with lists
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the no-content in lists with doctags
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* removed the prints
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added the Azure Document Intelligence
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the formatting and MyPy, working on the tests now
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* fixed the tests
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* added test document
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* replace all with pydantic dataclasses
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* minor azure serializer refactoring
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
* chore: remove unnecesary copyright headers
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* reformatted, still need to fix a test ...
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
---------
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
* chore: leverage fixtures in tests and add type annotations
Use fixtures for sample DoclingDocument in tests.
Add missing function type annotation when the body contains annotations.
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix(markdown): serialize hyperlink as code always with single backticks
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* fix: DoclingDocument model validator should deal with any raw input
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* docs: mention 3.14 as valid Python version for development
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
---------
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
* initial code chunking for docling-core
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* include language detections, add code chunking into hierarchical chunker
* add serializer, internal marking of chunkers, typing
* Update pyproject.toml
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
* Update docling_core/transforms/chunker/hierarchical_chunker.py
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
* run all pre-commit less pytest
* update test files for code ID
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* update uv.lock
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* revert to stricter treesitter versioning due to compatibility
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* remove language detection (to be run by client, i.e. docling)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* align new dependency specs
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
* address backticks, ABC, and supported languages feedback
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* remove Language class and reuse CodeLanguageLabel
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
* refactoring and improvements
- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
---------
Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
* fix(serializer): the HTML serializer should append nested lists to list items
For a valid HTML document, the serializer should ensure that sub-list items are indented
under their respective main list items.
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
* test: address several warning messages
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
---------
Signed-off-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
* insert_* methods and range manipulation methods
* DCO Remediation Commit for Ryan Fernandes <ryan@fernandes.us>
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: cc0c4b283f
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* insert_node_items method with recursive _append_item_copies helper for cross-document manipulation
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* add_node_items complement to inser_node_items, refactored extract_items_range to be more robust, added insert/add_document
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* Extended table data editing functionality
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* Added new document manipulation tests
* Extra gt files
* Refactor for reusable code - table methods
* Private methods for more concise insert_* methods
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* Docstring changes
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* ValueError for failed insertions
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
* DCO Remediation Commit for Ryan Fernandes <ryan@fernandes.us>
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 01e8b9cb03
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 8c016d9d32
I, Ryan Fernandes <ryan@fernandes.us>, hereby add my Signed-off-by to this commit: 308900270a
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>
---------
Signed-off-by: Ryan Fernandes <ryan@fernandes.us>