Update from code changes: document write_parallelism on add() - #312
Closed
mintlify[bot] wants to merge 1 commit into
Closed
Update from code changes: document write_parallelism on add()#312mintlify[bot] wants to merge 1 commit into
mintlify[bot] wants to merge 1 commit into
Conversation
Contributor
Author
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Contributor
Author
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
2 tasks
Contributor
|
Opened an alternative in #313 — reframes |
AyushExel
pushed a commit
that referenced
this pull request
Jul 26, 2026
* docs: rewrite write performance/ingestion section Reworks the Ingestion section on the Performance Tips page: - Frames write_parallelism around its main benefit (bandwidth), not just memory, and documents the progress=True tqdm bar (throughput and active worker count) that was previously undocumented. - Clarifies that for larger-than-memory data, scanning a file-backed Dataset is preferable to a hand-built RecordBatchReader: only a Dataset can be counted and rescanned, which enables auto-sized parallelism and retry-on-failure. - Adds write_parallelism guidance for iterator ingestion, where LanceDB can't auto-size it, including a rule of thumb to avoid over- fragmenting small datasets. Alternative to #312, which framed write_parallelism as primarily a memory-capping knob and didn't mention the progress bar. * docs: show sample tqdm output for progress=True * docs: narrow iterator ingestion to on-the-fly transforms Drops the memory-bound framing (iterator ingestion isn't really about memory) and scopes the section to data that needs per-row work before writing, rather than streaming sources generally. Calls out setting write_parallelism manually for large inputs in its own note. * docs: tighten ingestion section prose Trims repeated reasoning (bandwidth benefit, Dataset vs RecordBatchReader rescan/retry logic) that was restated across paragraphs. * docs: fix embeddings example, drop wide-row scan memory note Embedding computation is built into the ingestion pipeline, so it's a misleading example of when you'd need to hand-roll an iterator instead of scanning a file-backed Dataset. Swaps it for a generic "custom data transformations" example in both places it appeared. Also drops the wide-row scan memory bounding note for now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Python
add()method now exposes awrite_parallelismparameter to cap the number of parallel write partitions, and automatically bounds scan batch size and read-ahead for wide-rowpyarrow.dataset.Datasetinputs so bulk ingestion no longer OOMs the client on wide embedding columns. This PR documents the new knob and the auto-bounded behavior in the Ingestion section of the Performance Tips page. See lancedb/lancedb#3625.Changes
docs/performance.mdx, covering when to usewrite_parallelism, its default, and a code example.Datasetinputs are auto-sized against a ~1 GiB in-flight budget while narrow inputs keep PyArrow defaults.