Cesar Berrospi Ramis a661bb10cb fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens (#533)
* fix: prevent infinite loop in LineBasedTokenChunker with unbreakable tokens

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* tests: add a pytest fixture to test LineBasedTokenChunker efficiently

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

* style: apply style conventions to test_line_chunker.py test module

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>

---------

Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
2026-03-06 12:52:30 +01:00
2024-07-15 12:55:20 +02:00
2024-07-15 12:55:20 +02:00
2024-07-15 12:55:20 +02:00
2024-07-15 12:55:20 +02:00

Docling Core

PyPI version Python uv Code style: black Imports: isort Checked with mypy Pydantic v2 pre-commit License MIT

Docling Core is a library that defines core data types and transformations in Docling.

Installation

To use Docling Core, simply install docling-core from your package manager, e.g. pip:

pip install docling-core

Development setup

To develop for Docling Core, you need Python3.10 through 3.14 and the uv package. You can then install it from your local clone's root directory:

uv sync --all-extras

To run the pytest suite, execute:

uv run pytest -s test

Main features

Docling Core provides the foundational DoclingDocument data model and API, as well as additional APIs for tasks like serialization and chunking, which are key to developing generative AI applications using Docling.

DoclingDocument

Docling Core defines the DoclingDocument as a Pydantic model, allowing for advanced data model control, customizability, and interoperability.

In addition to specifying the schema, it provides a handy API for building documents, as well as for basic operations, e.g. exporting to various formats, like Markdown, HTML, and others.

👉 More details:

Serialization

Different users can have varying requirements when it comes to serialization. To address this, the Serialization API introduces a design that allows easy extension, while providing feature-rich built-in implementations (on which the respective DoclingDocument helpers are actually based).

👉 More details:

Chunking

Similarly to above, the Chunking API provides built-in chunking capabilities as well as a design that enables easy extension, this way tackling customization requirements of different use cases.

👉 More details:

Contributing

Please read Contributing to Docling Core for details.

References

If you use Docling Core in your projects, please consider citing the following:

@techreport{Docling,
  author = "Deep Search Team",
  month = 8,
  title = "Docling Technical Report",
  url = "https://arxiv.org/abs/2408.09869",
  eprint = "2408.09869",
  doi = "10.48550/arXiv.2408.09869",
  version = "1.0.0",
  year = 2024
}

License

The Docling Core codebase is under MIT license. For individual model usage, please refer to the model licenses found in the original packages.

S
Description
A python library to define and validate data types in Docling.
Readme MIT 276 MiB
Languages
Python 99.9%