Files
docling-core/test/data/doc/constructed_doc.dt
3097645198 feat: add code chunking functionality (#398)
* initial code chunking for docling-core

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* include language detections, add code chunking into hierarchical chunker

* add serializer, internal marking of chunkers, typing

* Update pyproject.toml

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* Update docling_core/transforms/chunker/hierarchical_chunker.py

Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>

* run all pre-commit less pytest

* update test files for code ID

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* update uv.lock

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* revert to stricter treesitter versioning due to compatibility

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove language detection (to be run by client, i.e. docling)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* align new dependency specs

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

* address backticks, ABC, and supported languages feedback

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* remove Language class and reuse CodeLanguageLabel

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com>

I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 63c773907f
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 431d357053
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: f3175c2717
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 1a01de8557
I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 025aea31ab

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>

* refactoring and improvements

- encapsulated code chunking specifics to separate package
- clearly separated public vs internal API via module and method naming conventions
- simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy)
- split chunk data model to separate modules to prevent circular dependencies
- renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy
- fixed some issues (e.g. gen flag in test)

Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>

---------

Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com>
Signed-off-by: Bridget <bridget.mcginn@ibm.com>
Signed-off-by: Panos Vagenas <pva@zurich.ibm.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
2025-11-12 10:28:54 +01:00

73 lines
2.8 KiB
Plaintext
Vendored

<doctag><unordered_list><list_item>item of leading list</list_item>
</unordered_list>
<title>Title of the Document</title>
<text>Author 1
Affiliation 1</text>
<text>Author 2
Affiliation 2</text>
<section_header_level_1>1. Introduction</section_header_level_1>
<text>This paper introduces the biggest invention ever made. ...</text>
<unordered_list><list_item>list item 1</list_item>
<list_item>list item 2</list_item>
<list_item>list item 3</list_item>
<list_item><ordered_list><list_item>list item 3.a</list_item>
<list_item>list item 3.b</list_item>
<list_item>list item 3.c</list_item>
<list_item><ordered_list><list_item>list item 3.c.i</list_item>
</ordered_list></list_item>
</ordered_list></list_item>
<list_item>list item 4</list_item>
</unordered_list>
<otsl><fcel>Product<fcel>Years<lcel><nl><ucel><fcel>2016<fcel>2017<nl><fcel>Apple<fcel>49823<fcel>695944<nl><caption>This is the caption of table 1.</caption></otsl>
<picture><caption>This is the caption of figure 1.</caption></picture>
<picture><caption>This is the caption of figure 2.</caption></picture>
<unordered_list><list_item>item 1 of list</list_item>
</unordered_list>
<unordered_list><list_item>item 1 of list after empty list</list_item>
<list_item>item 2 of list after empty list</list_item>
</unordered_list>
<unordered_list><list_item>item 1 of neighboring list</list_item>
<list_item>item 2 of neighboring list</list_item>
<list_item><unordered_list><list_item>item 1 of sub list</list_item>
<list_item><inline><text>Here a code snippet:</text>
<code><_Python_>print("Hello world")</code>
<text>(to be displayed inline)</text>
</inline></list_item>
<list_item><inline><text>Here a formula:</text>
<formula>E=mc^2</formula>
<text>(to be displayed inline)</text>
</inline></list_item>
</unordered_list></list_item>
</unordered_list>
<text>Here a code block:</text>
<code><_Python_>print("Hello world")</code>
<text>Here a formula block:</text>
<formula>E=mc^2</formula>
<key_value_region><key_0>number<link_1></key_0><value_1>1<link_0></value_1></key_value_region>
<inline><text>Some formatting chops:</text>
<text>bold</text>
<text>italic</text>
<text>underline</text>
<text>strikethrough</text>
<text>subscript</text>
<text>superscript</text>
<text>hyperlink</text>
<text>&</text>
<text>everything at the same time.</text>
</inline>
<ordered_list><list_item>Item 1 in A</list_item>
<list_item>Item 2 in A</list_item>
<list_item>Item 3 in A</list_item>
<list_item><ordered_list><list_item>Item 1 in B</list_item>
<list_item>Item 2 in B</list_item>
<list_item><ordered_list><list_item>Item 1 in C</list_item>
<list_item>Item 2 in C</list_item>
</ordered_list></list_item>
<list_item>Item 3 in B</list_item>
</ordered_list></list_item>
<list_item>Item 4 in A</list_item>
</ordered_list>
<unordered_list><list_item>List item without parent list group</list_item>
</unordered_list>
<text>The end.</text>
</doctag>