timely-util: UnloadChunk, bulk probe extraction for chunk batches - #37954
Merged
Conversation
UnloadChunk is the bulk-read capability for Chunk families: look up a sorted set of keys in a chunk, copying the matching updates out into caller-owned staging. Extraction is finished with a chunk's body when the call returns, so a spilled body is read for the scope of one call and no reference into pool memory exists outside it -- the contract that lets a buffer pool evict with no reader accounting. The trait has no key/val/time/diff opinions (Staging and Probes are family-chosen), and the one comparison the batch driver needs is delegated to the chunk via locate, answerable from resident metadata. Chunk boundaries are carried as the consume-index protocol: a chunk consumes every probe strictly below its last key and extracts-but-does-not-consume one equal to it, re-offered to the next chunk as a legal straddle. Ported from the differential fork (TimelyDataflow/differential-dataflow PR 782) so consumers here depend on a local definition rather than a git pin. One delta from the upstream shape: the batch driver is the UnloadBatch extension trait rather than inherent methods on ChunkBatch (not available to this crate); names and signatures match, so call sites survive a future switch to upstream re-exports unchanged. If the PR lands, this module retires in favor of re-exports. Contract test: a miniature row family driven through every (chunk cut x contiguous probe range) placement with a multi-chunk-spanning key, against the reference filter.
DAlperin
added a commit
that referenced
this pull request
Aug 4, 2026
### Motivation Part of the buffer-managed dataflow state design (`doc/developer/design/20260610_buffer_managed_state.md`). The pool merged in #37718 and its configuration in #37719. This PR adds the chunk type that spills through it. Stacked on #37954 (the UnloadChunk trait); follow-ups adopt it in storage's upsert stash and compute's arrange sites, staged with CI on #37805. Part of [CPU-184](https://linear.app/materializeinc/issue/CPU-184). ### Description Three commits, reviewable separately. 1. **Monotone columnar ship threshold.** The ship signal was a 10% window below each 2 MiB boundary. A single record wider than the window steps clear over it, the signal un-fires, and a chunk can grow past the pool's largest size class (8 MiB), where a spilled body degrades to permanently resident. The threshold is now monotone at 10% under 2 MiB. This touches live paths (`ColumnBuilder`'s ship point, merge/extract cuts, `Column::at_capacity`): rows under ~200 KiB serialized behave identically, wider rows now ship at the first boundary instead of drifting. 2. **ColumnChunk.** Differential's `Chunk` over `Column`-shaped updates: sorted, consolidated `(D, T, R)` runs with the merge/extract/advance/settle transducers, plus the UnloadChunk implementation for `(K, V)` data (locate from key fences, gallop-based extraction). Grading is by serialized bytes rather than the record-count `TARGET`, since record count does not bound bytes for variable-width data. 3. **Pool spilling and depth hints.** A chunk is Resident (`Rc`-shared `Column`) or Spilled (serialized body in the process pool, with record count and first/last fences resident). `settle` is the commit point: bodies at or above 64 KiB spill when the compute or storage gate is set and a pool is configured. Reads are copy-out and call-scoped, which is what lets the pool evict with no reader accounting. Each chunk carries a generational depth (fresh 0, merge output one past its deepest input) that becomes the pool's eviction-band hint, so repeatedly merged, colder data evicts first. The UnloadChunk probe path reads spilled bodies for the scope of one call and deliberately does not re-admit them. Nothing in production sets the spill gates yet. The storage and compute wiring comes in the follow-up PRs, so apart from commit 1 this is inert until then. ### Verification Property tests drive the trait methods the way the differential harness does and compare against brute-force references, in resident and force-spilled variants (batcher round trip, seal partitioning at intermediate frontiers, advance, extraction with straddled keys). Deterministic large-data tests cover the cut paths the proptests cannot reach: advance's multi-chunk cut, the giant-group carry, extract's mid-chunk cut on both sides. The threshold change carries a regression test that fails against the windowed predicate. The spill-gate matrix (compute/storage OR, no clobber) runs against a real installed pool, and the spill round trip is asserted byte-identical, including re-spilling an already-serialized body. The full stack runs CI on #37805.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Part of the buffer-managed dataflow state work (
doc/developer/design/20260610_buffer_managed_state.md, pool in #37718, configuration in #37719). This is the read capability the pool-spillable chunk stack builds on: the ColumnChunk PR stacks directly on this one, and upsert v2's stash drain is the production consumer. Upstream counterpart: TimelyDataflow/differential-dataflow#782.Part of CPU-184.
Description
UnloadChunk is a second way to read a chunk, alongside the cursor: hand in a sorted, deduplicated column of probe keys and the matching updates are appended to caller-owned staging.
locateanswers where a probe falls relative to a chunk's key span from resident metadata only, so the batch driver can gallop the chunk list and open only chunks whose span contains a probe. Probes that fall in a gap between chunks are consumed without faulting any body. A probe equal to a chunk's last key is extracted but re-offered to the next chunk, since its group may continue there (the straddle protocol).This PR is the trait and the batch-level driver only. The ColumnChunk implementation, including the spilled-body read path, is the next PR in the stack.
Verification
Contract tests over a miniature row family drive extraction across arbitrary chunk cuts and probe placements, straddled keys included, and compare against a reference filter of the raw rows.