docs(service-client): show chunk export through convert - #3983
Draft
pgustafs wants to merge 4 commits into
Draft
Conversation
Signed-off-by: Peter Gustafsson <peter.gustafsson6@gmail.com>
Contributor
|
✅ DCO Check Passed Thanks @pgustafs, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
dolfim-ibm
requested changes
Aug 14, 2026
dolfim-ibm
left a comment
dolfim-ibm
left a comment
Member
There was a problem hiding this comment.
@pgustafs we are about to deprecate the chunking endpoint for docling-serve.
The chunking capabilities are now directly in the convert endpoint. You can add to_formats=["chunks"] or also stream directly to a database.
Can you please validate if the normal convert endpoint is already enough for your use case? In case, update your PR for it.
added 3 commits
August 14, 2026 14:25
…-chunking-options Signed-off-by: Peter Gustafsson <peter.gustafsson6@gmail.com>
This reverts commit d5ef475. Signed-off-by: Peter Gustafsson <peter.gustafsson6@gmail.com>
Signed-off-by: Peter Gustafsson <peter.gustafsson6@gmail.com>
pgustafs
changed the title
Author
|
Thanks for pointing this out. I validated the normal convert endpoint, and it is enough for this use case. I tested use_markdown_tables and max_tokens with Docling Serve v1.30.0. An inline target does not include the chunk artifact, and sending chunks directly to a database would bypass a caller's own vector-store flow. ZipTarget works well here: the result contains a .chunks.jsonl file that the caller can read and store through its normal vector-store API. I reverted the submit_chunk() code change. The PR now only updates the documentation and example, and adds sync and async regression tests for the normal convert endpoint. |
This was referenced Aug 14, 2026
Member
|
@pgustafs do you consider this final? If yes, please mark it as ready for review. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
rhdedgar
pushed a commit
to rhdedgar/ogx
that referenced
this pull request
Aug 26, 2026
…arkdown (ogx-ai#6408) # What does this PR do? This PR lets callers ask the `remote::docling-serve` file processor to serialize tables inside chunks as Markdown tables instead of row triplets. The option is set for each request: ```python options={"use_markdown_tables": True} ``` The option works in both places that can start file processing: - Direct File Processors API requests. - Files attached through the Vector Stores API. Vector-store attachment already uses the File Processors API internally. This PR adds an optional `options` field to `OpenAIAttachFileRequest` and forwards it to the internal `ProcessFileRequest`. This keeps ingestion behavior consistent across both API entry points. The option is optional. Existing requests that do not send it keep the current behavior. The Docling Serve provider config stays limited to connection and operational settings. The File Processors API uses multipart form data. This PR also makes its existing `options` field accept a JSON object from multipart HTTP requests. This lets deployed OGX servers pass provider-specific options into `ProcessFileRequest.options`. Invalid values, such as the string `"true"`, return a validation error instead of silently enabling the option. This PR also changes chunking to use Docling's normal convert endpoint with chunk output. The old chunking endpoint is planned for deprecation, as discussed in [docling-project/docling#3983](docling-project/docling#3983). The implementation: - Supports both automatic and static OGX chunk sizes. - Reads chunk output from the ZIP archive returned by Docling. - Keeps headings and page numbers from Docling. - Accepts Docling options on direct file-processing requests and vector-store file attachments. - Returns a clear error when the Docling Serve version does not support chunk output through the convert endpoint. - Updates the minimum `docling-slim` version to `2.117.0`. - Regenerates the OpenAPI specifications. Chunking requires a compatible Docling Serve version, such as v1.30.0. Closes ogx-ai#6403 ## Test Plan Focused file processor, multipart request, vector-store router, and attachment propagation tests: ```bash uv run pytest -q \ tests/unit/core/jobs/test_file_processor_routes.py \ tests/unit/providers/file_processor/test_docling_serve.py \ tests/unit/providers/file_processor/test_docling_metadata.py \ tests/unit/providers/utils/memory/test_openai_vector_store_mixin.py \ tests/unit/core/routers/test_vector_stores_abac.py ``` Result: ```text 75 passed ``` All code, formatting, type, generated-schema, API compatibility, and API coverage pre-commit checks passed: ```bash uv run pre-commit run --all-files ``` The complete GitHub unit test job passed on commit `acc30038c`: ```text unit-tests (3.12): passed ``` The unit tests check that: - Multipart JSON is available in `ProcessFileRequest.options`. - Invalid multipart options are rejected. - Missing `use_markdown_tables` keeps the default behavior. - Boolean values are passed to Docling. - Non-boolean values are rejected. - `OpenAIAttachFileRequest.options` reaches the internal `ProcessFileRequest` unchanged. - Existing attachment requests without `options` still use automatic chunking. The script below verifies the new attachment path against a running OGX server with Docling Serve and pgvector: ```python #!/usr/bin/env -S uv run --script # /// script # requires-python = ">=3.11" # dependencies = [ # "httpx>=0.27,<1", # "reportlab>=4.2,<5", # ] # /// import io import os import re import time import httpx from reportlab.lib import colors from reportlab.lib.pagesizes import letter from reportlab.platypus import SimpleDocTemplate, Table, TableStyle def make_pdf() -> bytes: output = io.BytesIO() document = SimpleDocTemplate(output, pagesize=letter) table = Table( [ ["Probe Code", "Probe Region", "Probe Units"], ["MDTABLE-ALPHA-7319", "North Harbor", "17"], ["MDTABLE-BETA-8426", "South Ridge", "29"], ] ) table.setStyle( TableStyle( [ ("GRID", (0, 0), (-1, -1), 1, colors.black), ("FONTNAME", (0, 0), (-1, 0), "Helvetica-Bold"), ] ) ) document.build([table]) return output.getvalue() base_url = os.environ.get("OGX_URL", "http://localhost:8321/v1").rstrip("/") headers = {"Authorization": f"Bearer {os.environ.get('OGX_API_KEY', 'none')}"} client = httpx.Client(base_url=base_url, headers=headers, timeout=300) vector_store_id = None file_id = None try: vector_store = client.post( "/vector_stores", json={ "name": "markdown-table-attachment-test", "provider_id": "pgvector", "embedding_model": "sentence-transformers/nomic-ai/nomic-embed-text-v1.5", "embedding_dimension": 768, }, ) vector_store.raise_for_status() vector_store_id = vector_store.json()["id"] upload = client.post( "/files", data={"purpose": "assistants"}, files={"file": ("markdown-table-test.pdf", make_pdf(), "application/pdf")}, ) upload.raise_for_status() file_id = upload.json()["id"] attachment = client.post( f"/vector_stores/{vector_store_id}/files", json={ "file_id": file_id, "options": {"use_markdown_tables": True}, "chunking_strategy": { "type": "static", "static": { "max_chunk_size_tokens": 256, "chunk_overlap_tokens": 32, }, }, }, ) attachment.raise_for_status() attachment_body = attachment.json() while attachment_body["status"] == "in_progress": time.sleep(1) attachment = client.get(f"/vector_stores/{vector_store_id}/files/{file_id}") attachment.raise_for_status() attachment_body = attachment.json() assert attachment_body["status"] == "completed", attachment_body content = client.get(f"/vector_stores/{vector_store_id}/files/{file_id}/content") content.raise_for_status() text = "\n".join(item["text"] for item in content.json()["data"]) assert "| Probe Code" in text assert "| MDTABLE-ALPHA-7319" in text assert re.search(r"\|\s*-{3,}", text) assert not re.search(r"Probe\s+(Code|Region|Units)\s*=", text) search = client.post( f"/vector_stores/{vector_store_id}/search", json={"query": "MDTABLE-ALPHA-7319 North Harbor 17"}, ) search.raise_for_status() search_text = "\n".join( item["text"] for result in search.json()["data"] for item in result["content"] ) assert "MDTABLE-ALPHA-7319" in search_text print("PASS: vector-store attachment forwarded the Markdown-table option to Docling.") finally: if vector_store_id: client.delete(f"/vector_stores/{vector_store_id}").raise_for_status() if file_id: client.delete(f"/files/{file_id}").raise_for_status() client.close() ``` Execution output from OGX commit `acc30038c`, Docling Slim `2.120.1`, Docling Serve `v1.30.0`, and `remote::pgvector`: ```text Providers verified: remote::pgvector and remote::docling-serve Created vector store: vs_c93cb311-ceb8-4042-8ab6-e6718084bc09 (provider=pgvector) Uploaded table PDF: file-f2fa8599b08e49338159e70210783203 Vector-store attachment completed with options.use_markdown_tables=true Retrieved 3 stored Docling chunk(s) from pgvector | Probe Code | Probe Region | Probe Units | |--------------------|----------------|---------------| | MDTABLE-ALPHA-7319 | North Harbor | 17 | pgvector search returned the uniquely marked table row PASS: vector-store attachment forwarded the Markdown-table option to Docling. ``` --------- Signed-off-by: Peter Gustafsson <peter.gustafsson6@gmail.com> Co-authored-by: Peter Gustafsson <peter.gustafsson6@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The normal convert endpoint already supports chunk output and custom hybrid chunking options. This PR updates the service-client documentation and example to show that supported path.
The example uses OutputFormat.CHUNKS, HybridChunkerOptions, and ZipTarget. The returned zip contains a .chunks.jsonl file that the caller can read and store where it wants.
This PR no longer changes submit_chunk(). It adds regression tests showing that both the sync and async clients send the chunk format, custom chunking options, and zip target through the normal convert endpoint.
Issue resolved by this Pull Request:
Resolves #3982
Test plan:
Checklist: