mirror of
https://github.com/docling-project/docling.git
synced 2026-05-17 13:10:38 +00:00
dev/complex_html_renderer_example
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 geoHeil
|
eb4724ee4c |
ci: prototype tach-based modular skipping (#3333)
* ci: prototype tach-based modular skipping Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: modularize ubuntu setup and refine gating Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: adopt metaxy-inspired governance helpers - replace custom aggregate check with re-actors/alls-green - set FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 on every workflow - keep PR concurrency alive when the graphite:merge label is present Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: tune checks and pin action versions Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: split CI suites and heavy examples Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * DCO Remediation Commit for Georg Heiler <georg.kf.heiler@gmail.com> I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: ecaa4777886157d5c2a7b3893c3a820983089dbf I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: d15416f3ca94ac97af2a8317cd6404208db9d896 Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: sharpen tach graph and per-suite path filters - Split docling.pipeline into per-pipeline tach modules (asr, vlm, standard_pdf, threaded_standard_pdf, legacy_standard_pdf, extraction_vlm, base, base_extraction, simple) so pytest --tach-base impact analysis can attribute changes to a specific pipeline rather than the whole package. - Split the asr- and vlm-specific docling.datamodel option files (asr_model_specs, pipeline_options_asr_model, vlm_engine_options, vlm_model_specs, pipeline_options_vlm_model, layout_model_specs, stage_model_specs, backend_options) into their own tach modules so a narrow spec/options change no longer marks the full datamodel as impacted. - Narrow the per-suite pipeline path filters in checks.yml to the concrete pipeline files relevant to each suite, so editing vlm_pipeline.py only triggers the vlm matrix cell and editing asr_pipeline.py only the asr one. - Rekey the model cache in setup-ubuntu-ci to include runner.os and hashFiles(uv.lock, pyproject.toml), with ordered restore-keys fallbacks so a lockfile bump no longer silently stales the cache. Metaxy parity note: layered tach enforcement (layer = "...") is blocked by existing backend<->datamodel and utils<->stages cycles; depot runners, nox dynamic matrices, devenv/nix, dprint and ty are not applicable to docling's stack. All pinned action SHAs are on their latest release as of this commit. Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: introduce pipeline and orchestration tach layers Earlier notes claimed layers were blocked. That was only true for the cyclic core (backend<->datamodel, utils<->stages). The boundary *above* core is clean: - No module under docling/backend, docling/datamodel, docling/models, docling/utils, docling/exceptions, or docling/chunking imports anything from docling.pipeline (verified by grep). - No module anywhere in docling/ imports from docling.cli, docling.document_converter, docling.document_extractor, or docling.service_client (also verified). So we can introduce two real layers on top of the cyclic core: - "pipeline" — docling.pipeline and all nine concrete pipelines (base, simple, base_extraction, asr, vlm, extraction_vlm, standard_pdf, threaded_standard_pdf, legacy_standard_pdf). - "orchestration" — docling.cli, docling.document_converter, docling.document_extractor, and docling.experimental.pipeline. Unlayered modules stay "below" both layers (tach allows them to be depended on freely) and continue to carry the declared-but-cyclic backend<->datamodel and utils<->stages edges. A VLM-only layer was explored but rejected: only docling.pipeline.vlm_pipeline and docling.pipeline.extraction_vlm_pipeline could be cleanly layered as "vlm", because the matching datamodel options (pipeline_options_vlm_model, vlm_engine_options, vlm_model_specs) and model stages (vlm_convert, vlm_pipeline_models) sit inside the datamodel/models cycle and cannot be promoted to a higher layer without first breaking that cycle. Layering only the two pipeline files is not worth the extra config. Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: expand tach layers to entrypoints/pipeline/models/core Follow-up to the two-layer attempt. After verifying via grep that nothing in datamodel/utils/backend imports from docling.models.{extraction,factories,plugins,vlm_pipeline_models} or from the "upper" stages (page_assemble, page_preprocessing, reading_order, picture_description, vlm_convert), those nine modules can be promoted out of the cyclic core into a dedicated "models" layer. The resulting order (highest first): - entrypoints — cli, document_converter, document_extractor, experimental.pipeline - pipeline — docling.pipeline + the nine concrete pipelines - models — model factories, extraction, plugins, vlm_pipeline_models, and the five "upper" stages - core — datamodel*, backend*, utils, exceptions, chunking, models (base), models.utils, inference_engines.*, the six "core stages" that utils cycles with (chart_extraction, code_formula, layout, ocr, picture_classifier, table_structure), and the experimental.* and service_client modules Rename the previous "orchestration" layer to "entrypoints" to match the common docling vocabulary. Every module now carries an explicit layer tag instead of relying on implicit unlayered behaviour, so future additions must pick a layer deliberately. A VLM layer, a stand-alone inference-engines layer, and separating datamodel from backend all remain blocked by the bidirectional backend<->datamodel and utils<->core-stages edges; those need a code-level refactor first. Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: refine tach client and foundation layers Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: add optional windows and macos smoke lanes Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: normalize reusable workflow boolean inputs Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: replace external all-green action Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: use org-allowed setup-uv action Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: install compiler toolchain for ML tests Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * DCO Remediation Commit for Georg Heiler <georg.kf.heiler@gmail.com> I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: bb714afb42cd1b29ab073a7f59cc72874ff2fdcd I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: a1f2761da8f72bfed636bd571ebf77b42c8771b6 Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * DCO Remediation Commit for Georg Heiler <georg.kf.heiler@gmail.com> I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: cc6551b54c5bf4815ae9cd57cf43a98928a74be0 I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: b21b0e7ca12b552dbdd54fac1bda113719c286f1 Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: simplify ML pytest suite patterns Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: gate heavy examples on label, add job timeouts - ci-heavy-examples: run only on main push, schedule, workflow_dispatch, or when a PR is labeled tests:full / tests:heavy-examples. Drops the path-based auto-trigger so that common edits to pyproject.toml, uv.lock, or .github/actions do not kick off the 45-60min matrix on every PR push. Collapses the changes job into a job-level if gate and adds timeout-minutes: 90. - checks.yml: add timeout-minutes to every job so stuck runners cannot burn the full 6h default. Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: tolerate cancelled allowed-skip jobs in check aggregator Intentional cancellations (manual cancel, concurrency replacement) on jobs that are already in ALLOWED_SKIPS should not mark the overall workflow red. Treat `cancelled` the same as `skipped` when the job is listed as an allowed skip; any unexpected cancellation of a required job still fails. Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * docs: make minimal vlm example portable Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * DCO Remediation Commit for Georg Heiler <georg.kf.heiler@gmail.com> I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: 2135051da3ed73d4b8a9130f584f40b56155af1a I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: 4f6d1d7960f7418d0cde6425ae61538da84fda40 Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: install workspace packages in CI syncs Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * DCO Remediation Commit for Georg Heiler <georg.kf.heiler@gmail.com> I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: 492fa9883d4de6d98ebcb40fa863eafe2facff3c I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: 3eefae71643f9ca3df0264690c0c6eb1f67f06f1 Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * DCO Remediation Commit for Georg Heiler <georg.kf.heiler@gmail.com> I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: fe8c9689a0ee94f36eb826da8e2177ef87404f5e I, Georg Heiler <georg.kf.heiler@gmail.com>, hereby add my Signed-off-by to this commit: eabdd24a6734ec873cdaac857718aef2473677e7 Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: remove unused graphite concurrency exception Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: document test labels and gate cross-platform lanes Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: select ml tests with pytest markers Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: fix marker selector typing Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: simplify ml suite scheduling Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: mark cross-platform smoke tests Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: reuse test trigger for ml matrix Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: tighten full ci aggregation Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> * ci: share required job result check Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> --------- Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
 Christoph Auer
|
03532938b5 |
feat: Unified model-family inference engines (including image-classification) and KServe v2 API support (#2979)
* feat: Inference engines abstraction for image classification model family with HF Transformers and ONNX runtime Implements runtime abstraction for image classification models with support for both ONNX Runtime and HuggingFace Transformers engines. Users can switch between engines without model retraining, similar to the object detection abstraction (#2959). Key components: - BaseImageClassificationEngine with factory pattern - OnnxRuntimeImageClassificationEngine and TransformersImageClassificationEngine implementations - Shared HfVisionModelMixin for common HF model utilities - Engine-specific configuration options - Test suite and example demonstrating runtime engine switching Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Add missing files and re-export for backward compat Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Don't run with OCR in the example. Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Remove excess onnxruntime related options for inuts and outputs Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat: centralize torch compile defaults with DOCLING_INFERENCE_COMPILE_TORCH_MODELS Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat: Add Kserve2 API engine for image classifier and object detection models (#2999) * fix: add failed pages to DoclingDocument for page break consistency (#2939) * fix: add failed pages to DoclingDocument for page break consistency When some PDF pages fail to parse, they were not added to DoclingDocument.pages, causing page break markers to be incorrect during export. This adds failed/skipped pages with their size info (if available) to maintain correct page numbering and structure. - Add _add_failed_pages_to_document() method in StandardPdfPipeline - Add test cases for failed page handling - Add test cases for normal page handling (regression test) - Add test PDF files Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com> * fix: ensure resource cleanup and simplify type hints - Wrap page_backend usage in try-finally to guarantee unload (prevents resource leaks). - Simplify redundant 'float | None | None' type hint. Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com> * fix: add groundtruth for normal_4pages.pdf and exclude failing PDFs from e2e test Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com> * fix: ensure correct status assertion for failed pages in tests Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com> --------- Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com> * fix: Use timezone-aware datetime (#2947) * Use timezone-aware datetime for profiling timestamps Updated timestamp recording to use timezone-aware datetime. Signed-off-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com> * run formatter Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> --------- Signed-off-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com> Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> Co-authored-by: Michele Dolfi <dol@zurich.ibm.com> * fix(asciidoc): handle commas in image alt text (#2983) * Fix: Handle commas in AsciiDoc image alt text - Modified _parse_picture() to gracefully handle alt text containing commas - Commas in alt text are now preserved instead of causing ValueError - Added test case with realistic auto-generated alt text - split('=', 1) prevents issues when values contain '=' characters * DCO Remediation Commit for n0rdp0l <n90.w135@gmail.com> I, n0rdp0l <n90.w135@gmail.com>, hereby add my Signed-off-by to this commit: |
||
|
Christoph Auer
|
aed772ab33 |
feat: Threaded PDF pipeline (#1951)
* Initial async pdf pipeline Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * UpstreamAwareQueue Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Refactoring into async pipeline primitives and graph Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Cleanups and safety improvements Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Better threaded PDF pipeline Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Pin docling-ibm-models Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Remove unused args Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Add test Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Revise pipeline Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Unload doc backend Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Revert "Unload doc backend" This reverts commit |
||
 Alex Sokolov
|
2efb7a7c06 |
fix(settings): fix nested settings load via environment variables (#1551)
Signed-off-by: Alexander Sokolov <alsokoloff@gmail.com> |