From 81036ddb6e5eee9e1edaafe7f16bd0abad81cc86 Mon Sep 17 00:00:00 2001
From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com>
Date: Tue, 28 Jul 2026 22:39:53 +0000
Subject: [PATCH] docs: document transform_parallelism option for
StreamingDataset
---
docs/training/index.mdx | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/docs/training/index.mdx b/docs/training/index.mdx
index 85078a0..1948976 100644
--- a/docs/training/index.mdx
+++ b/docs/training/index.mdx
@@ -115,7 +115,10 @@ ds = StreamingDataset(
Many model training workloads require a transformation step between loading the data and training the model. For
example, we may need to decode images, tokenize text, or normalize data. A transformation function can be provided
using the `transform` parameter. Transformations can be expensive, so `StreamingDataset` applies them with a
-`ThreadPoolExecutor` whose worker count equals the number of available CPUs.
+`ThreadPoolExecutor`. By default the worker count equals the number of available CPUs (falling back to one worker
+when the CPU count cannot be determined). Use the `transform_parallelism` parameter to override this limit — for
+example, to cap concurrency on shared machines or to raise it when transforms release the GIL and CPUs are plentiful.
+`transform_parallelism` must be a positive integer.
Transformations are applied to batches, not individual samples, to amortize per-batch overhead. A transformation
function receives a PyArrow `RecordBatch` and must return an iterable with exactly one output sample for every input
@@ -140,6 +143,24 @@ ds = StreamingDataset(table, shuffle_seed=42, transform=normalize)
```
+To pin the transform pool to a specific size, pass `transform_parallelism`:
+
+
+```py Python icon=Python
+ds = StreamingDataset(
+ table,
+ shuffle_seed=42,
+ transform=normalize,
+ transform_parallelism=4,
+)
+```
+
+
+
+`transform_parallelism` also bounds how many raw batches are held in flight for transformation, so a smaller value
+reduces peak memory use at the cost of throughput.
+
+
#### DataLoader workers
The thread-based transformation model that `StreamingDataset` uses by default is only effective when the transform