mirror of
https://github.com/docling-project/docling-core.git
synced 2026-11-18 20:27:00 +00:00
* initial code chunking for docling-core * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 334811a4edf0c9a8511a0e3cd31acda2df335d62 Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * include language detections, add code chunking into hierarchical chunker * add serializer, internal marking of chunkers, typing * Update pyproject.toml Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com> Signed-off-by: Bridget <bridget.mcginn@ibm.com> * Update docling_core/transforms/chunker/hierarchical_chunker.py Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com> Signed-off-by: Bridget <bridget.mcginn@ibm.com> * run all pre-commit less pytest * update test files for code ID * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 46bb88ae3a00ebbdd33336e51b51e030c748425b I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 10e9ed8a0f28bdd4ae6156a84f19435319f4a5ce I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: d9827c749b72ecc902c66a24eb7fcb4db1b615fa I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 814dc6161134752b08f699eeef58535248893768 Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * update uv.lock Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * revert to stricter treesitter versioning due to compatibility Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: a4a21e90d63da3aa46fd6f0681b065d6bf85305b I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 0266c6387066cd8e4d45ea02fcb417ade5eedd18 I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 336dd6a2078272dd30ade58b4fc6392e7a2edf86 I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 68890e9f48e6f310e27eb20eeb2d98d45c90bd36 I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit: 3c65eef59e817e9bd8cd3c55e98e843e326dd7a2 Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * remove language detection (to be run by client, i.e. docling) Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * align new dependency specs Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> * address backticks, ABC, and supported languages feedback Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * remove Language class and reuse CodeLanguageLabel Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * DCO Remediation Commit for Bridget McGinn <bridget.mcginn@ibm.com> I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit:63c773907fI, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit:431d357053I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit:f3175c2717I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit:1a01de8557I, Bridget McGinn <bridget.mcginn@ibm.com>, hereby add my Signed-off-by to this commit:025aea31abSigned-off-by: Bridget McGinn <bridget.mcginn@ibm.com> * refactoring and improvements - encapsulated code chunking specifics to separate package - clearly separated public vs internal API via module and method naming conventions - simplified or removed parts not stricly necessary for public API (e.g. lang support querying, noopstrategy) - split chunk data model to separate modules to prevent circular dependencies - renamed DefaultCodeChunkingStrategy to Standard... for clarity as it need not be the default strategy - fixed some issues (e.g. gen flag in test) Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> --------- Signed-off-by: Bridget McGinn <bridget.mcginn@ibm.com> Signed-off-by: Bridget <bridget.mcginn@ibm.com> Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com> Co-authored-by: Panos Vagenas <pva@zurich.ibm.com>
73 lines
2.8 KiB
Plaintext
Vendored
73 lines
2.8 KiB
Plaintext
Vendored
<doctag><unordered_list><list_item>item of leading list</list_item>
|
|
</unordered_list>
|
|
<title>Title of the Document</title>
|
|
<text>Author 1
|
|
Affiliation 1</text>
|
|
<text>Author 2
|
|
Affiliation 2</text>
|
|
<section_header_level_1>1. Introduction</section_header_level_1>
|
|
<text>This paper introduces the biggest invention ever made. ...</text>
|
|
<unordered_list><list_item>list item 1</list_item>
|
|
<list_item>list item 2</list_item>
|
|
<list_item>list item 3</list_item>
|
|
<list_item><ordered_list><list_item>list item 3.a</list_item>
|
|
<list_item>list item 3.b</list_item>
|
|
<list_item>list item 3.c</list_item>
|
|
<list_item><ordered_list><list_item>list item 3.c.i</list_item>
|
|
</ordered_list></list_item>
|
|
</ordered_list></list_item>
|
|
<list_item>list item 4</list_item>
|
|
</unordered_list>
|
|
<otsl><fcel>Product<fcel>Years<lcel><nl><ucel><fcel>2016<fcel>2017<nl><fcel>Apple<fcel>49823<fcel>695944<nl><caption>This is the caption of table 1.</caption></otsl>
|
|
<picture><caption>This is the caption of figure 1.</caption></picture>
|
|
<picture><caption>This is the caption of figure 2.</caption></picture>
|
|
<unordered_list><list_item>item 1 of list</list_item>
|
|
</unordered_list>
|
|
<unordered_list><list_item>item 1 of list after empty list</list_item>
|
|
<list_item>item 2 of list after empty list</list_item>
|
|
</unordered_list>
|
|
<unordered_list><list_item>item 1 of neighboring list</list_item>
|
|
<list_item>item 2 of neighboring list</list_item>
|
|
<list_item><unordered_list><list_item>item 1 of sub list</list_item>
|
|
<list_item><inline><text>Here a code snippet:</text>
|
|
<code><_Python_>print("Hello world")</code>
|
|
<text>(to be displayed inline)</text>
|
|
</inline></list_item>
|
|
<list_item><inline><text>Here a formula:</text>
|
|
<formula>E=mc^2</formula>
|
|
<text>(to be displayed inline)</text>
|
|
</inline></list_item>
|
|
</unordered_list></list_item>
|
|
</unordered_list>
|
|
<text>Here a code block:</text>
|
|
<code><_Python_>print("Hello world")</code>
|
|
<text>Here a formula block:</text>
|
|
<formula>E=mc^2</formula>
|
|
<key_value_region><key_0>number<link_1></key_0><value_1>1<link_0></value_1></key_value_region>
|
|
<inline><text>Some formatting chops:</text>
|
|
<text>bold</text>
|
|
<text>italic</text>
|
|
<text>underline</text>
|
|
<text>strikethrough</text>
|
|
<text>subscript</text>
|
|
<text>superscript</text>
|
|
<text>hyperlink</text>
|
|
<text>&</text>
|
|
<text>everything at the same time.</text>
|
|
</inline>
|
|
<ordered_list><list_item>Item 1 in A</list_item>
|
|
<list_item>Item 2 in A</list_item>
|
|
<list_item>Item 3 in A</list_item>
|
|
<list_item><ordered_list><list_item>Item 1 in B</list_item>
|
|
<list_item>Item 2 in B</list_item>
|
|
<list_item><ordered_list><list_item>Item 1 in C</list_item>
|
|
<list_item>Item 2 in C</list_item>
|
|
</ordered_list></list_item>
|
|
<list_item>Item 3 in B</list_item>
|
|
</ordered_list></list_item>
|
|
<list_item>Item 4 in A</list_item>
|
|
</ordered_list>
|
|
<unordered_list><list_item>List item without parent list group</list_item>
|
|
</unordered_list>
|
|
<text>The end.</text>
|
|
</doctag> |