[Feature Request]: Support single-GPU vector-index builds at wiki_all 88M scale on 45 GB VRAM - #27357
Conversation
…ty bounds the build
IndexCapacity is supposed to cap how much of a table a single cuVS sub-index
covers, but setting it bought no memory back. getOrCreateCurrent finalized the
full sub-index with Build() and appended it to b.indexes, and nothing was
released until ToInsertSql at the very end of the scan. Every finished
sub-index therefore stayed resident for the whole build.
The host side was already fine: Build() clears flattened_host_dataset at the end
of the C++ build(). What accumulated was DEVICE memory -- the cuVS index plus
dataset_device_ptr_, the f32 build matrix that SINGLE_GPU keeps until destroy()
(cgo/cuvs/ivf_pq.hpp:581, released at :1943). That is capacity*dim*sizeof(Q) per
retired sub-index.
Now the sub-index is packed to its temp tar and freed at rotation. saveToFile
already does exactly this teardown -- Pack, checksum, Index.Destroy(), Index=nil
-- and is idempotent, so the later ToSql call is a no-op. This is the pattern
HnswBuild has always used.
Measured on a 8 GB card, 20000 rows at dim 256, capacity 4000 (5 sub-indexes),
free VRAM at each sub-index's SINGLE_GPU:before-alloc:
before: 6875 6863 6859 6853 6849 MB (-26 MB, monotonic)
after: 6875 6869 6869 6869 6869 MB (flat)
The A/B was run both ways against real builds, not inferred: reverting the
change reproduces the decline, restoring it reproduces the flat line.
At wiki_all scale the difference is not incremental. dim 960 with a
VRAM-sized capacity puts ~11 GB in each retired sub-index, so on a 20 GB card
the second one alone would exhaust the device -- without this, capacity
splitting cannot work at all.
Two smaller things ride along:
- A capacity > 0 guard on the rotation predicate. With IndexCapacity == 0,
`b.count >= capacity` holds on every AddRow, which after this change would
pack a sub-index per row. Only the create TVF's auto-detect keeps that from
firing today, and capacity is about to gain a second provenance.
- The retired model is appended to b.indexes BEFORE saveToFile, so a pack
failure still leaves it reachable from Destroy() and its GPU handle is
freed. IvfpqBuild.Destroy already iterates b.indexes and IvfpqModel.Destroy
removes idx.Path, so the abort path leaks neither handle nor tar.
ivfpq and cagra change together and the two edits are textually identical;
hnsw already did this and ivfflat has no GPU builder, so this closes the set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for bounding the index build by VRAM. No behaviour change yet: the new C shim has no Go caller, and the quantizer path computes exactly what it did. cap_train_rows_to_gpu_mem already knew how many rows fit ~60% of free VRAM, but it was a private member of gpu_index_base_t and unreachable from Go, where the build capacity is decided (cudaMemGetInfo appears nowhere outside C++). The arithmetic moves to matrixone::rows_fitting_gpu_mem in helper.h/.cpp and the member becomes a three-line delegate, so there is ONE implementation of the rule. A second, independently written VRAM heuristic that could disagree with the first would be a defect in itself, which is why this is extracted rather than reimplemented. The 60% figure and its reasoning are unchanged and now documented on the shared function: the caller uploads one contiguous device allocation, a single block rarely fits 80% of free memory once the pool is fragmented, and cuVS needs scratch on top. gpu_rows_fitting_free_mem exposes it to Go. It calls cudaSetDevice first -- cudaMemGetInfo reports the CURRENT device and the Go caller runs on an arbitrary thread with nothing bound, so without that it would measure whichever device happened to be current. cuvs.RowsFittingFreeMem wraps it and returns the free-byte reading alongside the row count so callers can log what they measured. Both keep the throw-never-guess contract: a caller that cannot measure the device cannot size an upload for it, and the fallback that would otherwise be natural here -- assume the whole table fits -- is precisely the OOM this work exists to prevent. The quantizer's log line changes shape (it is now emitted by the shared function and names per-row bytes rather than dim). Nothing greps that text. Verified: MO_CL_CUDA=1 make cgo, go build -tags gpu ./pkg/cuvs/..., go vet, gofmt all clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the VRAM ceiling. IndexCapacity now means "rows per sub-index that this
device can actually hold", not "the whole table".
Previously capacity defaulted to the full source row count, so building wiki_all
(88M rows, dim 960) asked for ~315 GiB of f32 on a 20 GB card and died in training
even though the finished PQ index fits the card several times over. The bound
applies to EVERY build, not only the default one: an explicit max_index_capacity is
a request, and honouring a request larger than the device can take would reintroduce
exactly the OOM this exists to prevent.
capacity = min(explicit_or_srcRowCount, rowsFittingFreeVRAM, srcRowCount)
The trailing min(., srcRowCount) is load-bearing. InitEmpty preallocates
capacity*dim*sizeof(Q) host bytes before a row arrives, so without it a 20-row table
would reserve for millions. TestPlanCapacity covers it, and non-vacuously: with the
clamp disabled the suite fails.
Per-row cost is not just the dataset, and getting that wrong would put the OOM back
at the DERIVED default, which is worse than today's failure because nobody asked for
it. ivfpq adds dim*4*kmeans_trainset_fraction (cuVS subsamples the k-means trainset
as float32 whatever the storage type, so a quantized build still pays it — omitting
it under-counts an int8 build by ~40%). cagra adds intermediate_graph_degree*8 for
the intermediate kNN graph.
Both create TVFs resolve capacity later in start() than before, because sizing
against VRAM needs the dimension, the storage type and the device, and all three
were established after the old capacity block. The VRAM reading is sampled once,
before any index is constructed, since the RMM pool that worker->start() creates
counts as used memory.
On cudaMemGetInfo failure the statement errors instead of guessing. The natural
fallback -- assume the whole table fits -- is the failure being fixed, so the message
names the lever (set max_index_capacity explicitly) rather than silently proceeding.
Three latent bugs this made reachable, fixed here since capacity is no longer
almost-always the row count:
- ivfpq read `lists` straight into the small-tail threshold, and lists is only set
when the user passed it. A 0 threshold silently disabled the k-means minimum
check; it now falls back to the cuVS default, as cagra already did.
- capacity < the k-means minimum used to route EVERY row into the brute-force CDC
tail with only a log line -- for a large table that is the whole dataset as
per-row host copies plus an index that cannot answer a vector search. It is now
an error, and the message distinguishes an over-tight knob from a device too
small for the configured lists, so the operator is sent to the right lever.
- distribution_mode 'sharded' combined with a split is rejected. Sharded
sub-indexes are packed with a "shards" manifest key and reloaded by a loader that
branches on the configured mode, so silently building them single-GPU would
produce indexes that cannot be loaded back at all.
The decision logic lives in planCapacity in index_create_helper.go, the only file on
this path without a gpu build tag, so all of it is covered by CI without CUDA. That
is the whole off-GPU safety net -- everything else here is //go:build gpu and can
only be exercised on a GPU host.
Verified: builds with and without the gpu tag, go vet clean, gofmt clean, package
tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Clean merge, no conflicts: 11 commits, 76 files. Main touched none of the files this
branch changes -- the vector index build path (pkg/vectorindex/{ivfpq,cagra},
pkg/sql/colexec/table_function/*_create_gpu.go, cgo/cuvs/helper.*, index_base.hpp) is
untouched upstream, so the overlap is empty rather than merely conflict-free.
Verified after merging: builds with and without the gpu tag; go vet clean; the
off-GPU planCapacity suite and the whole table_function package pass; pkg/vectorindex
and pkg/cuvs pass under MO_CL_CUDA=1 through the wrapper (a plain wrapper run is
correctly refused here -- this tree's libmo is CUDA-built).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…stop the bogus cap log Two findings from the self-review gate on this branch. 1. The per-row VRAM model used the CONFIGURED k-means trainset fraction, but ivfpqConfig only forwards that value to cuVS when it is > 0 (ivfpq/model_gpu.go), so `kmeans_train_percent = 0` leaves cuVS on its own default of 0.5 while the model assumed 0. At dim 768 that is 3072 B/row modelled against 4608 B/row actually allocated -- a third under -- which spends the 60% headroom and can OOM at the DERIVED capacity, precisely the failure this bound exists to prevent. It now reads the effective fraction, falling back to DefaultIvfPqBuildParams() exactly as the build does. Confirmed against a real build rather than argued: with kmeans_train_percent=0 at dim 8 the log reported 32 B/row before and reports 48 after (32 dataset + 8*4*0.5), matching what cuVS allocates. CAGRA has no equivalent hole: its intermediate_graph_degree fallback of 128 already matches the value that reaches cuVS. 2. Every GPU index build logged "[index capacity] capped 9223372036854775807 -> N rows". The Go shim passed INT64_MAX as the requested row count so the shared helper believed it had truncated a request, when the caller was only asking how many rows fit. It read as data loss and printed a nonsense number. rows_fitting_gpu_mem is now a pure query that never logs, and cap_rows_to_gpu_mem caps a real request and logs only when it actually caps. The quantizer keeps its capping semantics and its log line; the capacity query is silent. There is still exactly one implementation of the 60% rule. Verified: the line is gone from a real build and the informative "N MB free ... B/row -> R rows fit" line remains. Verified after the change: MO_CL_CUDA=1 make clean build; go vet clean; the off-GPU planCapacity suite passes; pkg/cuvs, pkg/vectorindex/ivfpq and pkg/vectorindex/cagra pass through the wrapper; both behaviours re-checked against a live index build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ation Two cleanup defects from the self-review decision log, plus the tests that were missing for the rotation mechanism this branch depends on. Destroy stopped at the first error. All three models -- ivfpq, cagra AND hnsw -- returned as soon as Index.Destroy() failed, so the packed tar was never removed and leaked for the lifetime of the process; hnsw could also skip clearing its buffer. The handle, the file and the buffer are independent, so each is now released and the outcomes joined. hnsw additionally clears Path afterwards, matching its siblings. My decision-log entry understated this: I recorded it as reachable only through b.current, and missed that hnsw's variant could strand the buffer too. Each build now owns a private temp directory. Tars were written straight into $TMPDIR as ivfpq*/cagra*, so cleanup was N per-file removals -- any one of which could fail and strand a tar -- and an orphan could not be attributed to a process. The builder creates mo-<algo>-<pid>-<random>/ up front, models pack into it, and Destroy reclaims it with a single RemoveAll. The pid is a label, not the uniqueness mechanism: MkdirTemp appends a random suffix and creates the directory atomically, so concurrent CREATE INDEX statements in one process get distinct directories and a builder can only ever delete its own. That is also why a $TMPDIR sweeper was rejected instead: several mo-service processes share /tmp, and a sweeper matching ivfpq* would delete another live instance's in-flight tar. A crash still orphans a directory -- unavoidable -- but it is now one identifiable directory per build rather than loose files, which is what would make a pid-liveness sweeper writable later. Tests. build_tmpdir_test.go pins the directory contract: created up front under $TMPDIR with the owning pid in its name; distinct across builders in one process (the property the pid alone does NOT give); propagated to the model so the tar really lands there; and RemoveAll reclaiming a file that a per-file remove could not. vector_ivfpq_multi_subindex.sql is the first BVT to exercise rotation at all -- vector_ivfpq.sql sets capacity 99999 against 20 rows and always builds one sub-index. 256 rows at capacity 64 gives exactly 4 sub-indexes; the case asserts the metadata row count, that every sub-index was actually persisted (non-empty checksum, positive filesize), exact top-1 recall for a probe landing in each of the four chunks, a top-3 that merges across them, and the new error when capacity falls below lists. Writing it turned up that the existing vector_ivfpq.sql "storage layout" check asserts nothing: mo-tester records no output for `prepare s from @q; execute s; deallocate prepare s;` on one line. The new case puts execute on its own line, which does capture the result -- hence real values (4 and 4) in its .result rather than a silently empty assertion. Verified: gpu and cpu builds, go vet, gofmt; pkg/vectorindex/... green through the wrapper; the full gpu_cases/vector suite 927/928 with -n as CI runs it. The single failure is vector_gpu_negative.sql expecting the old spelling of an INCLUDE column-type error from ivfflat/plugin/plan/schema.go, which this diff does not touch -- main changed how those type names render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MO builds an IVF-PQ index by allocating count*dim device memory, copying the
host buffer up, and calling the DEVICE build overload (ivf_pq.hpp:555-571).
cuVS also accepts a raft::host_matrix_view, and that path samples the k-means
trainset on the host (sample_rows.cuh:47-63) and streams the encode through
batch_load_iterator with a batch size that halves until it fits free memory
(ivf_pq_build.cuh:1050-1097, :1374). Nothing measured which of the two MO
should be on.
dim 960, trainset_fraction 0.1, n_lists 4*sqrt(N), n_probes 32, k 10, on an
RTX 5070 Laptop with 7.34 GB free:
rows dataset device host host+pool dev peak host peak
250k 0.96 GB 9.9/11.6s 8.3s 8.2s 1.63 GB 0.92 GB
500k 1.92 GB 21.0/19.8s 20.2s 20.3s 2.74 GB 1.07 GB
1M 3.84 GB 25.1/25.1s 25.9s 25.9s 4.94 GB 1.35 GB
2.5M 9.60 GB 147.9/144.7s 73.8s 40.8s 7.33 GB 2.88 GB
recall@10 is 1.000 for every cell. While the dataset fits, streaming costs
nothing (+0.8% at 1M) and peak VRAM drops by the whole dataset — host peak is
near flat, 0.92 -> 1.35 GB across a 4x increase. Once the dataset exceeds VRAM
the host path is 2.0x faster, 3.6x with a stream pool; the device path did not
hard-OOM here only because WSL2 backs GPU memory with host RAM, so it paged
instead.
Queries perturb a known row so the correct top-1 is that row's id. An earlier
draft compared A-vs-B neighbor overlap on uniform random vectors and was
unreadable — an A-vs-A control scored 0.02 at 1M, because distances concentrate
in 960 dims and top-k is then near-arbitrary. An equivalence metric needs an
absolute oracle or a same-path control.
Standalone (cuVS/RAFT/RMM only), like test_dynb.cu and test_ivfpq_filter.cu.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…taset All three branches of build_internal allocated count*dim (or num_rows*dim) device memory, copied flattened_host_dataset up, and called the DEVICE build overload. That upload is what makes build memory scale with the table: at 88M x dim 960 it is ~315 GiB of f32 against a 20 GB card. cuVS also accepts raft::host_matrix_view, and detail::build is templated on the mdspan accessor (ivf_pq_build.cuh:1223) so the host view runs the same code with two differences that are exactly what a large build needs: sample_rows branches on cudaPointerGetAttributes and gathers the k-means trainset on the HOST (sample_rows.cuh:47-63), and build_impl ends with detail::extend(dataset.data_handle(), ...) (:1374), which streams through utils::batch_load_iterator with a batch size that halves until it fits free memory (:1050-1097). Measured by test/bench_hostview.cu at dim 960: within noise while the dataset fits the card (+0.8% at 1M), 2.0x faster once it does not, peak VRAM down from 4.94 GB to 1.35 GB at 1M and near flat as rows grow. recall@10 identical at 1.000 across 250k..2.5M. SHARDED gives each rank a view of its own disjoint slice of the one immutable host buffer, so ranks need no copy and no coordination. Because no build dataset is left on the device, neither dataset_device_ptr_ nor replicated_datasets_ is populated here any more. Both members stay in index_base.hpp for the algorithms that do need a resident dataset (cagra keeps its to search). The resets/erases on the extend paths and the SHARDED post-build clear in build() are now no-ops; that post-build clear existed only to stop non-last shards leaking GB-scale device memory for the index lifetime, which can no longer happen. test_cuvs_worker: 162/162 pass, covering all three distribution modes (GpuIvfPqTest, ManualSharded*, ReplicatedModeSimulation), extend in each mode, GpuIvfPqRecall::Int / PureCuvsIvfPqRecall::Int, and save/load round-trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uses the real tuning from mo_vector_benchmark/cfg/templates/88M.json — dim 768,
m 192, bits 8, lists 6000, kmeans_train_percent 2, float16 — rather than cuVS
defaults, which differ enough to change the answer (calculate_pq_dim(768) is
384, twice the configured 192).
IVF-PQ build has two phases whose allocations do NOT overlap: the trainset and
k-means scratch are scoped to a block closing at ivf_pq_build.cuh:1369, before
detail::extend at :1374 allocates the PQ list data. Peak is therefore
max(train, encode), not their sum. This measures each separately via
add_data_on_build=false followed by an explicit extend — which is also the
train-then-extend split expressed through cuVS's own API, with no hand
sampling and so no ord/pk permutation.
Measured (RTX 5070 Laptop, 7.34 GB free), conservative_memory_allocation false:
rows trainA peakA extB peakB recall@10
1M 4.3s 0.14GB 4.8s 0.75GB 1.000
2.5M 10.8s 0.28GB 10.4s 1.21GB 1.000
5M 7.9s 0.51GB 19.5s 1.58GB 1.000
8M 10.6s 0.80GB 38.9s 2.63GB 1.000
Training is not the constraint: phase A is 8.1 GB projected at 88M. The encode
phase is, and its floor is fixed by the config — 88M x (192 code + 8 index)
bytes = 17.6 GB, i.e. 88% of a 20 GB card before any workspace. Measured
bytes-per-row runs above that floor and is not cleanly linear (207 B/row
fitting 1M-5M, 258 B/row fitting 2.5M-8M, against a 200 B/row floor), so the
whole-range fit projects 24.1 GB. conservative_memory_allocation=true does not
account for the gap (8M: 2.62 vs 2.63 GB), so it is not IVF list over-growth.
Extrapolating 11x is not a verdict. What the numbers do support: a single 20 GB
card is marginal at best for this config, the 88M template already specifies
distribution_mode "sharded", and sharded over 4 GPUs puts ~4.4 GB of codes on
each. recall@10 is 1.000 at every measured size.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-on to the host-view build. Three things were wrong once the dataset stopped being uploaded. 1. Capacity was bounded by the dataset. It is now bounded by what actually stays resident -- the PQ codes plus the int64 payload cuVS stores beside each one. That is also the honest bound: a search reaches every list, so the whole index must be loaded, and splitting into sub-indexes does not shrink the total. pqCodeBytes mirrors cuVS calculate_pq_dim (ivf_pq_index.cu:611) for builds that leave m unset, where cuVS picks dim/2 rounded down to a multiple of 32 -- 384 at dim 768, twice the 192 the wiki_all template configures. 2. The trainset was folded into the per-row cost, which traded capacity away to buy something that is not competing for the same bytes. The trainset and its k-means scratch are scoped to a block closing at ivf_pq_build.cuh:1369, before detail::extend at :1374 allocates the list data, so peak is max(train, index). It is now budgeted separately and the FRACTION is clamped instead of the capacity -- capacity is bounded by the index, which nothing can shrink, while the sample can simply be made smaller. 3. The trainset per-element cost used the storage width. cuVS keeps the trainset in float32 whatever the storage type, plus a second copy in T for non-float T (:1288-1307), both live at the peak. So f16 costs 6 bytes per element against f32's 4 -- a narrower storage type makes this term BIGGER, and sizing it the old way under-counted f16 by a third. The effective fraction is now written back to the config and logged against the requested one. Without that the clamp is invisible: a request for 20% that the device can only honour at 3.7% still builds, still succeeds, and surfaces only as recall nobody can explain. Thin centroids (under ~39 training rows per list) warn rather than refuse. The floor is a rule of thumb, not a cuVS constraint -- validate_build_params only checks rows >= n_lists -- so refusing would break configurations that work today. The warning names both levers and leaves the choice with the operator. CAGRA keeps its dataset term, and the comment now says why instead of pointing at an ivfpq twin that has diverged: cagra searches by walking the graph and reading the actual vectors, so its dataset is resident for the index's whole life, not just the build. Streaming the build would not change that. Also adds vector_cagra_multi_subindex.sql, the cagra half of the rotation pair. Every other cagra case sets capacity 99999 against 20 rows and builds exactly one sub-index, so nothing covered rotation -- which matters more for cagra than for ivfpq, since freeing each retired sub-index is the only thing bounding a resident dataset. Tests: 12 new subtests over calculatePqDim, pqCodeBytes, trainsetBytesPerElem and planTrainFraction, in the tag-free helper so CI can reach them. Package passes; go vet clean under both the default and gpu build tags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Generated against mo-service at 72f99d4. Asserts what rotation is supposed to produce and nothing else: 4 metadata rows for 256 rows at capacity 64, all 4 with a checksum and a positive filesize (a sub-index freed without being persisted reads as 0 here), and exact top-1 for a probe drawn from each of the four chunks -- 7, 80, 150, 233 -- which only holds if every sub-index was built, saved and reloaded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unrelated to the ivfpq work in this branch; found because it was the one case failing the GPU vector suite (954/955). The message from pkg/vectorindex/ivfflat/plugin/plan/schema.go now renders SQL type names where the recorded expectation still had Go ones: - (supported: int32, int64, float32, float64) + (supported: INT, BIGINT, FLOAT, DOUBLE) The behaviour is right and arguably better -- a user reading the error writes INT, not int32 -- so the expectation is what is out of date. Nothing in this branch touches that file; `git diff --name-only 7940a50..HEAD` does not include it. Suite is 955/955 after this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ment Rewrites the plan around SINGLE as the target. The 88M template says sharded, but that reflects a build that could not fit otherwise; the work on this branch is what makes one card plausible. Supersedes a draft that proposed hand-writing a train/extend split in MO. cuVS already implements it and MO was opting out, so the plan became "stop opting out" plus the sizing corrections that follow from it. States the three resources separately, because the build was failing for one reason and being sized for another: the dataset (streamed, done), the k-means trainset (fraction clamped, done), and the index itself (m, still open). The first two never coexist with each other -- the trainset block closes at ivf_pq_build.cuh:1369 before extend at :1374 -- so peak is max(train, encode) and m alone decides whether SINGLE works. The open question is narrowed to one number. Measured overhead above the raw codes+ids floor is ~0.56 GB flat through 5M and then 1.02 GB at 8M. Constant means 88M at m=192 is 18.2 GB and the template's tuning already works; proportional means 28.9 GB and m must drop to 128. An 8 GB card cannot tell those apart, and conservative_memory_allocation was tested and is not the cause. bench_wiki88 already reports the phases separately, so the AWS run extends the table directly; host RAM (1536 B/row at f16) is what caps how far it goes. Keeps the two arithmetic corrections that were wrong in every earlier estimate: cuVS picks pq_dim 384 at dim 768 when m is unset, twice the configured 192; and the trainset is float32 plus a copy in T, so f16 costs 6 bytes per element and narrow storage makes that term bigger, not smaller. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Self-review blocker against the previous commit.
capacity is a HOST allocation before it is a device one: InitEmpty reaches the
C++ constructor, which does flattened_host_dataset.resize(capacity * dim) up
front (ivf_pq.hpp:226,258), and resize value-initializes, so it is committed
RSS rather than lazy pages.
That bound used to come for free. While the per-row cost included the dataset
term (dim * sizeof(Q)), bounding capacity against VRAM incidentally bounded the
host buffer at roughly the same figure -- they were the same bytes. Sizing
ivfpq against the PQ codes instead is correct, since the dataset is now
streamed and never resident, but it decoupled the two and multiplied the host
allocation by dim*sizeof(Q)/(m+8):
dim 768, f16, m 192, 60% of a 20 GB card
before perRow 1597 B capacity 7.9M host prealloc 12.1 GB
after perRow 200 B capacity 63.0M host prealloc 96.8 GB (7.7x)
Concrete failure: an 88M-row table on a box with less than ~97 GB free RAM gets
std::bad_alloc or the OOM killer, where before the change it built as 12
sub-indexes of 12.1 GB. Nothing in the create path read host memory at all.
planCapacity now takes hostRowsFit alongside rowsFit and records which one
decided the result, so the "below the k-means minimum" error names the resource
that actually bound it -- host and device want opposite fixes. hostRowsFittingMem
takes 60% of gopsutil's MemAvailable, the same fraction as
cap_train_rows_to_gpu_mem, deliberately not a second heuristic.
Unlike the device query, a failure here is non-fatal: cudaMemGetInfo failing
means the device is unusable and guessing reintroduces the OOM, but gopsutil
failing just means this extra bound cannot be applied on this platform, and the
device bound plus the srcRowCount clamp still hold. It returns 0, which
disables the bound rather than failing a build that would have worked.
Applied to cagra too. Its per-row cost still includes the dataset, so its VRAM
bound already keeps the host buffer under the device budget; the explicit bound
costs one syscall and stops the two from drifting apart.
Also corrects a factual error this document had been repeating: host RAM for
88M in f16 is not 135 GB. flattened_host_dataset is sized by capacity, not N,
and is cleared after each sub-index builds (ivf_pq.hpp:418-419), so only one
buffer is live at a time; 135 GB would require capacity == N. It IS 135 GB for
cagra, where every sub-index must stay loaded to search.
Tests: 6 new subtests for the host bound (tighter-than-VRAM, looser, explicit
request clamped, srcRowCount clearing both flags, error naming the host lever,
zero disabling it) plus 3 for hostRowsFittingMem, and the 16 existing
planCapacity cases updated for the signature. Package passes; go vet clean
under default and gpu tags.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Checked whether passing a host pointer is documented behaviour or an implementation detail we happened to find. It is documented, and it settles two items this plan listed as open. The out-of-core path is intended. wiki_all_dataset.rst says the ~251 GB is "intentionally larger than the typical memory of GPUs ... to promote the use of compression and efficient out-of-core methods for both indexing and search." The host_matrix_view overloads are that mechanism. The index-size formula capacity is sized against is the documented one -- n_vectors * (pq_dim * pq_bits/8 + sizeof_idx) + n_clusters -- not something reverse-engineered. The same page's "Build peak memory usage (device)" is trainset + labels + centroids with the index absent, independently confirming the two phases do not overlap. large_workspace_resource is struck from the risks: the doc says build "automatically switches to the managed memory for the training set and labels" when the workspace is short, so it pages rather than throws. Most usefully it reframes the open 88M question. "A heuristic controls the batch size to make sure the workspace fits get_workspace_free_bytes" means the gap between measured peak and the raw floor is cuVS expanding to fill free workspace, not a fixed cost scaling with rows -- which also explains the 8M jump that conservative_memory_allocation could not. The 24.1 GB linear extrapolation is likely wrong in the safe direction. Also records the host build buffer as bounded now that 13bf845 landed.
Completes what the IVF-PQ change started. All three uploaded the dataset to the
device and called the device build overload; all three can take a
raft::host_matrix_view instead. The reason it pays off differs per algorithm, and
so does what happens to ownership.
IVF-Flat copies vectors INTO its lists (interleaved). The documented index
footprint is n_vectors*dim*sizeof(T) + n_vectors*sizeof(idx) + ... , so the build
input was never referenced after build returned -- yet we uploaded it and then
retained it in replicated_datasets_/dataset_device_ptr_ for the index's whole
life. That was the dataset carried TWICE on the device. Now nothing is retained.
CAGRA genuinely needs a resident dataset: it searches by walking the graph and
reading the actual vectors. Only the owner changes. With a device view
update_dataset stores a REFERENCE when rows are 16B-aligned (cagra.hpp:524-527),
which is why the buffer had to be retained; with a host view it "creates a copy
of the dataset on the device" and "the index manages the lifetime of this copy"
(:556-558). Steady state is identical. The win is that cuVS attaches the dataset
LAST, after optimize (cagra_build.cuh:2264 then :2289), so the raw vectors are
not resident during the optimize step, which is the device peak: peak goes from
dataset + optimize to max(optimize, dataset + graph).
Also fixes a silent failure CAGRA could produce. cuVS catches std::bad_alloc
while attaching the dataset, logs "Only the graph will be added to the index",
and returns a valid index object with no vectors (cagra_build.cuh:2291-2297).
size() does not reveal it -- it falls back to the graph's row count when the
dataset is empty (cagra.hpp:341-346) -- so the first sign would have been every
search failing at serve time. require_dataset_attached() checks
dataset().extent(0) after each build and throws with the levers named.
Measured by the new bench_upload_cost.cu (200k x 768, cuVS/RAFT/RMM only):
algo dataset via build(s) peak recall@10
cagra device (old) 9.76 1.44 GB 1.000
cagra host view 10.14 0.82 GB 1.000
ivfflat device (old) 1.88 1.69 GB 1.000
ivfflat host view 2.07 1.31 GB 1.000
1.76x lower peak for cagra, 1.29x for ivfflat, recall unchanged, build time
within a few percent.
Two things that benchmark had to get right and are recorded in it. Its first
version let the device_uvector fall out of scope on the CAGRA device path and
died with cudaErrorIllegalAddress on the first search -- a faithful reproduction
of the dangling reference that MO's retention existed to prevent, and of exactly
what the host view removes the need for. And at intermediate_graph_degree 64 /
graph_degree 32 CAGRA scored 0.07 recall on 768-dim data while IVF-Flat scored
1.000 on the same corpus; that is too thin a graph for the dimensionality, not a
build-path defect. The template's 256/64/256 gives 1.000.
Note IVF-Flat has no GPU create path in the SQL layer today (no
ivf_flat_create_gpu.go, no GPU plugin registration), so that half is a latent fix
covered by test_cuvs_worker and this benchmark rather than by BVT.
test_cuvs_worker 162/162; benchmark_cuvs rc=0 with recall 1.0 across cagra,
ivfflat and brute_force.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A GPU index build packs each finished sub-index to a tar before freeing its
device memory, so the bytes written to this directory are the whole index --
~17.6 GB per build at 88M rows with m=192. That was going to $TMPDIR. /tmp is
frequently a small or slow mount; on AWS it is the ~128 MB/s root volume, which
would add minutes of pure write stall to every large build. The LOCAL
fileservice is by definition the data directory the operator provisioned for
exactly this traffic. fulltext2 moved its spill for the same reason and for the
same measured number (pkg/fulltext2/storage.go:420-427).
vectorindex.LocalSpillDir mirrors fulltext2's helper but uses its OWN
__vectorindex subdir rather than sharing __fulltext2, so one feature's cleanup
cannot delete another's in-flight files. It returns "" when no LOCAL fileservice
is attached (unit tests, one-shot tools), and "" is already what
os.MkdirTemp/os.CreateTemp read as $TMPDIR -- so the fallback is exactly the
previous behaviour and no caller needs a branch. A failed MkdirAll also returns
"" rather than an error: a scratch directory that cannot be created should not
fail a build that would otherwise succeed.
The directory is passed as an explicit parameter, NOT added to
IndexTableConfig. That struct is JSON marshalled into the table-function
argument list and travels between CNs, whereas this path is only meaningful on
the node that resolved it -- shipping one CN's local path to another would be
wrong. The 14 call sites are the two 7-way [B, Q] switches, which already thread
everything else the builders need.
Verified end to end: with mo-data/local2 as the LOCAL root, the rotation BVT
(256 rows at capacity 64 -> 4 sub-index tars) creates
mo-data/local2/__vectorindex, leaves it empty afterwards (each build's private
subdir is reclaimed by Destroy), and writes no mo-ivfpq-*/mo-cagra-* into /tmp.
Tests: TestLocalSpillDir{FallsBackWithoutLocalFS,CreatesUnderLocalRoot} are
tag-free so CI reaches them; TestBuildTmpDirHonoursSpillDir and
TestBuildTmpDirsDistinctWithinOneSpillDir pin that the builder nests under the
supplied dir, keeps its pid-tagged naming, reclaims only its own subdirectory
and never the shared parent, and that concurrent builds stay distinct within one
spill dir. GPU BVT 955/955; go vet clean under default and gpu tags.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The spillDir parameter went into both builders, but only ivfpq had a build_tmpdir_test.go. Half the pair was carrying an untested contract, so a change could pass CI while breaking cagra's side of it -- which is exactly the shape of gap that made the cagra rotation BVT necessary earlier. The twin covers the same five properties: one private directory per build under $TMPDIR when no spill dir is given, pid-tagged so a crash leaves attributable orphans, RemoveAll-able in one step (including a tar that individual removal would have stranded), idempotent Destroy, distinct directories for concurrent builds sharing a pid, the directory reaching the model so saveToFile packs into it, and -- the new behaviour -- nesting under a supplied LOCAL spill dir while Destroy reclaims only its own subdirectory and never the shared parent. Also strengthens ivfpq's TestBuildTmpDirReachesTheModel, which was still exercising the "" fallback: it now builds under a spill dir and asserts the tar lands two levels down, so both packages prove the tar reaches the LOCAL directory rather than just the builder's. 5/5 pass in each package. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The size ladder was hardcoded to {1M, 2.5M, 5M, 8M} and the CLI argument only
filtered it, so `./bench_wiki88 40000000` -- which is what the plan document told
an operator to run on the AWS box -- silently produced the same four points a
laptop already had. The whole purpose of that run is the sizes above 8M.
The ladder now goes 1M, 2.5M, 5M, 8M, 16M, 32M, 64M, 88M and the argument is a
ceiling: `8000000` reproduces the original four, `88000000` measures the target
directly instead of extrapolating 11x to it.
Host cost is 1536 B/row at dim 768 in f16, so 88M needs ~135 GB and 32M ~49 GB.
Each size frees its buffer before the next, so peak host is the largest single
size rather than the sum -- comfortable on the 512 GB g6e.16xlarge this is aimed
at.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "one open question" from ivfpq_train_extend.md — whether workspace overhead scales linearly with N or is self-limited by cuVS's workspace heuristic — is now answered. On an L40S (44.6 GB free) at 87.5M / dim 768 / f16 / m=192: tar on disk 18.98 GB (17.68 GiB) +0.5% vs N·(m+8) device peak 24.7 GB ~7 GB workspace over floor peak / card 53% inside the 60% VRAM rule SINGLE=1 held (ivfpq_meta=1) build wall 3795 s (57 min scan + 1 min build + 4 min pack) The self-limiting hypothesis holds — workspace is real but bounded, not runaway. That answers the question the doc left open. Also annotate pqCodeBytes: real peak runs ~30-40% above this "documented" floor because of cuVS workspace that isn't folded in. The 60% VRAM rule absorbs the gap on every workload measured so far, but a user tuning a build to exactly the advertised rowsFit will OOM before the check says they should. Names the L40S data point and points at ivfpq_train_extend.md. Doc-only + one comment. No behaviour change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… Destroy failure
Adds instrumentation to pinpoint where a CAGRA/IVFPQ/HNSW commit dies
when the process exits silently — we hit exactly this with an 88M CAGRA
build that crashed at the metadata-commit stage with no error in the
log, leaving 11 completed tars on disk and one 0-byte tar for the
small-tail sub-index.
model_gpu.go (cagra + ivfpq) and model.go (hnsw):
- Log ENTER/exit around Pack (or Save for hnsw), CheckSum, and Destroy
with the sub-index id, size, and elapsed. If the process dies inside
any of these, the last surviving log line names the offending call.
- REORDER: record idx.Path = tarPath after Pack+CheckSum succeed, BEFORE
calling Destroy. A Destroy failure does not invalidate the on-disk
artifact; removing the tar here would drop committed data. For HNSW
the deferred cleanup now checks a `destroyed` flag so a save that
reached the Destroy step keeps its artifact even if Destroy fails.
cagra_create_gpu.go and ivfpq_create_gpu.go end():
- Log "executing N SQLs (M bytes)" once, then per-SQL start / done with
index and elapsed. If the process dies during commit, we now know
which specific INSERT (chunk index N of M) was executing.
Answers we could not answer from the CAGRA 88M crash:
- was it Pack for the not-yet-rotated tail sub-index?
- was it one of ~1,380 chunk INSERTs?
- was it the final 12-row metadata INSERT?
With this patch the next crash tells us which.
No behaviour change beyond logging; the Destroy-order fix is strictly
safer (never worse) than the prior code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… tar loop)
Previous diagnostic patch narrowed the CAGRA tail-Pack crash to
CagraModel.saveToFile's Index.Pack() call — but Pack itself was still a
black box. Refined instrumentation splits it:
GpuCagra.Pack:
log -> gpu_cagra_save_dir (about to enter cuVS via cgo)
log gpu_cagra_save_dir returned + list of files cuVS wrote
log -> Pack (tar) (Go-side tar-packing of the directory)
log Pack (tar) done
cuvs.Pack (consolidate.go):
log per-file: taring <name> (<size>)
error logs on Stat / Open / io.Copy failures
If the crash reappears we now know exactly which stage died:
- no "returned in ..." line => cuVS's cagra::serialize C++ crashed
(SIGSEGV / SIGABRT / uncaught CUDA — not catchable by the C++ try/catch)
- "returned in ..." but no "taring" => something between C call and tar loop
- "taring <name>" but no "done" => Go tar packing crashed on that file
Confirmed on 10M CAGRA build (2 sub-indexes, tail=2.17M rows): the
tail sub-index crashes inside Pack, exact same signature as the 88M
crash. Root-cause is a not-yet-rotated tail sub-index; this patch
identifies which sub-step of Pack.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 26.02 build ships a non-deterministic OOB read in
cuvs::neighbors::cagra::detail::graph::kern_prune during graph::optimize.
On the gpu_single_mode branch this surfaced as an intermittent SIGSEGV
inside cagra::serialize when packing a built index — corruption in
kern_prune wrote a wild pointer into the strided_dataset descriptor;
serialize's later cudaMemcpy2DAsync dereferenced it. 26.06+ fixes it.
Code migration (cgo/cuvs):
- cagra_index typed on device_padded_dataset_view (26.06 API rework).
- deserialize: capture out_dataset unique_ptr into dataset_device_ptr_ /
replicated_datasets_ so the view in index_->dataset() outlives cuvs's
return (previously left dim=0 dangling).
- extend: concat-and-rebind. Reads old view from index_->dataset(),
allocates a fresh padded (n_old + n_new) x stride device_matrix,
stride-aware copies old (D2D contiguous) and new (H2D via memcpy2D),
wraps as padded_dataset_t, calls cuvs::cagra::extend, swaps owner.
Peak VRAM 2x current dataset during transition — same cost cuVS paid
internally pre-26.06, just moved to the caller. Drops the old
"not supported for half" throw (26.08 supports it).
- merge: same concat-and-rebind pattern across sub-index datasets;
threads the merged buffer out through the worker lambda so the new
gpu_cagra_t's dataset_device_ptr_ owns it.
- -std=c++20 (raft needs atomic<shared_ptr>); RMM
device_async_resource_ref replaces the removed device_memory_resource
base class; pool_memory_resource swaps to the non-templated ctor
with any_resource<device_accessible> upstream.
Docker + env:
- go_cuda-133_arch-x86_64.yaml: cuda-version=13.3, libcuvs 26.08.*,
libraft 26.08.*, drops the 26.02-only cudf/kvikio/nvcomp deps.
- Dockerfile / Dockerfile.dev: nvidia/cuda:13.3.1-cudnn-{devel,runtime}
(previously 13.2.0 / 13.0.2), point at the new yaml, make -j8.
- Root Makefile comment refreshed to the new yaml path and CUDA floor.
- Deletes stale go_cuda-130_arch-x86_64.yaml.
Verified: test_cuvs_worker 162/162, benchmark_cuvs clean, pkg/cuvs Go
tests pass, 10M CAGRA build no crash (12 min, previously SIGSEGV),
recall@10 = 0.8861 on wiki_all 10M, docker build + smoke test green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Only cgroup v2 signals "no limit" with the string "max". v1 writes PAGE_COUNTER_MAX into memory.limit_in_bytes, which parses as a perfectly good integer -- so an unlimited v1 hierarchy was read as a REAL limit of ~9.2 EB. MemoryAvailableIncludingCache then returned 9223372035781029888 with measured==true, and measured is exactly the signal callers use to decide the figure is trustworthy enough to size a bulk allocation from. HostRowsFitting derived a 6.9 EB budget, the host governor admitted everything, and the bound that exists to stop a build overcommitting the node silently stopped existing -- on the hosts still running cgroup v1, which is the configuration the v1 branch is there to serve in the first place. Both walks are affected: minHierarchicalHeadroom feeds MemoryAvailableIncludingCache directly, and minHierarchicalLimit feeds it via CgroupMemoryLimit as the second tier. readCgroupLimit now rejects the sentinel for both; usage files keep reading through readCgroupUint, where a large value is not a sentinel. `>=` rather than `==` so kernels reporting a bare LONG_MAX are covered. Fixed in C++ as well: host_meminfo.cpp mirrors these rules deliberately, so it mirrored the defect. Both sides get the regression test, including that a real v1 limit still binds. Not reachable on the dev box, which is cgroup v2 -- proven instead with the temp-directory fixtures the existing hierarchy tests already use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit filtered cgroup v1's unlimited sentinel inside the hierarchy walks, but CgroupMemoryLimit falls back to gosigar when the walk finds nothing, and gosigar returns the sentinel verbatim -- its own tests assert a limit of 9223372036854771712. So on an unlimited v1 host the walk now returned 0, the fallback returned 9.2 EB, and MemoryAvailableIncludingCache's second tier derived a budget from it exactly as before. The fix was half a fix. normalizeCgroupLimit puts "0 means no bound" in one place, on both the error and the sentinel, and is unit-tested directly rather than through a gosigar mock. Found by tracing the closure of the previous commit rather than by a new symptom: CgroupMemoryLimit has two other consumers (remote-compile mpool sizing, execution resource budget) that were also being handed 9.2 EB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aptend
left a comment
aptend
left a comment
There was a problem hiding this comment.
Deep re-reviewed the complete diff, all prior reviews/comments/replies, and every resolved/unresolved thread at exact head e6447112ae9678a5347bd545191a56445d97c976. The earlier IVF-PQ budget mismatch, SINGLE_GPU bystander-device veto, and cgroup-v1 unlimited-sentinel defects are closed. Two deterministic capacity/admission mismatches still reject supported builds. Validation on this head: git diff --check; normal tests for pkg/common/system, pkg/vectorindex/memory, and pkg/sql/colexec/table_function; and focused race tests for their hierarchy, reservation, host, device, and capacity paths all pass. The CUDA/cuVS paths were audited statically because this host has no CUDA runtime.
|
fixed |
1 similar comment
|
fixed |
… aggregate Two admission figures that described a build other than the one that runs. STAGING CHARGE was bounded by the whole source. The arena is per SUB-INDEX -- staging_bound_rows() caps it at this->count, and every rotated model is constructed with idxcfg.IndexCapacity -- so a split build was charged for rows no single sub-index ever stages. At 88M rows with max_index_capacity 1M, dim 768 f32 base and train limit 50M, that charges ~29.25 GiB against a 24 GiB budget and refuses a build whose real sub-indexes stage 1M rows (~2.86 GiB) apiece. stagingRowBound narrows it with the terms already known before the plan -- requested capacity and the VRAM fit. That is what keeps it out of a circle: the plan can only narrow capacity further (hostRowsFit is its one remaining input), so the bound is never smaller than the capacity finally chosen and the charge can never come out short. The test asserts exactly that property against planCapacity rather than just checking the arithmetic. SHARDED AGGREGATE did not survive the split it feeds. min_rows * distinct advertises a capacity that index_base.hpp then cuts as (total/N) & ~31 with the remainder on the LAST shard: four cards holding 1001 rows each advertise 4004, which splits 992/992/992/1028, and the last card is asked for 27 rows more than the card the figure came from. The build sizes its claim from the real shard and is refused at exactly the advertised capacity. CAGRA's 128-row training minimum does not catch it -- 992 clears that comfortably. Aligning before multiplying makes every shard equal, so the last one absorbs no remainder. Below one aligned shard the unaligned figure stands rather than 0, because 0 reads as "not measured" and would disable the VRAM bound instead of refusing. Covered by a sweep over shard counts and sizes, plus the reported case and the figure it used to fail on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
XuPeng-SH
left a comment
XuPeng-SH
left a comment
There was a problem hiding this comment.
Deep re-review on exact head e6447112ae9678a5347bd545191a56445d97c976. The earlier IVF-PQ budget-policy mismatch, bystander-GPU veto, and cgroup-v1 unlimited-sentinel defects are closed, but this PR is not approvable yet.
-
[Design gate / blocking] This feature has no stable, versioned, approved design revision. The complete change is 91 files / 12.5k added lines and changes GPU/host admission, cross-language ownership, concurrent build/load lifecycle, persisted artifacts, configuration semantics, and the CUDA/cuVS runtime. It independently exceeds both mandatory design triggers (>500 production lines and >5 owning components), but the PR links only issue #27356; the diff contains no design/RFC/ADR, and the review history has no traceable design-first approval of an exact revision. The issue/PR descriptions also leave a material scope contradiction unresolved: they claim 88M single-GPU CAGRA builds “land”/work end-to-end, while the PR explicitly says CAGRA 88M cannot be searched, and
end()now callsDeviceAggregateFitsHardwareafter all sub-indexes have been built and packed, so that 135-GB aggregate is rejected and CREATE persists nothing. Please provide and obtain approval for a stable design revision that closes the supported end-to-end scope, resource ownership/invariants, alternatives, rollout/compatibility, and validation plan; then link that exact revision here. If CAGRA 88M is intentionally out of scope, the owning issue/claims and fail-fast behavior need to agree with that approved decision. -
[P1] Quantizer host staging is still sized from the whole source instead of one planned rotated sub-index. Both create paths call
QuantizerStagingBytes(..., srcRowCount, ...)beforeplanCapacity(ivfpq_create_gpu.go:522-560,cagra_create_gpu.go:465-515), but nativestaging_bound_rows()caps each arena by that model constructor count, i.e. the finalIndexCapacity. Concrete supported case: 88M source rows, explicit 1M capacity, dim-768 f32→int8, train limit 50M, 45-GiB GPU, 32-GiB host availability. Planning charges 29.25 GiB of staging and rejects it against the 24-GiB host budget, although each real rotated model can stage only 1M rows (2.86 GiB) plus a small int8/ID capacity buffer. Resolve the staging/capacity dependency without charging rows that a sub-index cannot hold, and add asrcRowCount > IndexCapacityregression for both algorithms. -
[P1] SHARDED capacity does not preserve the per-device bound after 32-row alignment.
index_cost_base::rows_fittingreturnsmin_rows * distinct(index_cost.hpp:130-173), while native CAGRA/IVF-PQ split computesrows_per_shard=(total/N)&~31and places the remainder on the last card (cagra.hpp:727-729,ivf_pq.hpp:540-545). If four cards each fit 1001 rows, planning advertises 4004; build splits 992/992/992/1028, and the last device exact claim refuses 1028 against a 1001-row budget. The 992-row regular shard already clears the CAGRA default minimum, so the existing minimum-shard check does not catch this. Return an alignment-safe aggregate that keeps the largest shard <=min_rows, and add this boundary regression. -
[P1 unhappy path] The new staging governor still releases its claim before geometrically reserved slack is physically accounted.
stage_rows_lockedclaims the full vector-capacity growth, callsreserve, inserts only the currentn_rows, then releases (index_base.hpp:1490-1550). On Linux, the unusedvector::reservepages are not charged to cgroup usage until faulted. Subsequent inserts that fit the spare capacity havegrow_data == false, take no claim, and can fault a large amount of that slack. Two builders can therefore both pass growth admission, release after touching only their current rows, and later consume their unclaimed slack concurrently until the cgroup OOMs. The comment acknowledging “one doubling ... accounted in neither place” is precisely the uncovered window. Retain a claim for unfaulted capacity, materialize the claimed range before release, or claim at write-time; add a barrier-synchronized two-builder test that advances through already-reserved slack.
Validation: exact-head git diff --check is clean; the branch merges cleanly with current origin/main; exact-head ordinary CI is green. The arithmetic counterexamples above are deterministic. GPU-tagged execution/real-scale evidence is not available on this macOS host, and ordinary CI does not compile or execute these CUDA/cuVS paths.
…design doc The staging arena grows geometrically, so a growth reserves roughly twice what is in use. reserve() leaves that slack allocated but UNFAULTED, and cgroup usage only moves on fault -- so the claim covering it was released against memory the kernel had not charged, and every later call that fits inside the slack takes no claim at all (grow_data is false) while faulting it. Two builders could each pass a growth admission, release having touched only their current rows, and then consume their unclaimed slack concurrently. The previous comment named that window and left it open. It is now closed with the idiom allocate_host_capacity already uses: resize() value-initialises, which faults every page, and restoring the size afterwards keeps the capacity. By the time the claim drops, the bytes it stood for are charged to the availability the next caller reads. Also adds docs/design/gpu_vector_index_memory_admission.md: goal, supported scope, ownership (C++ owns both ledgers, Go plans capacity), invariants, alternatives, rollout and validation. It records the decision that a single CAGRA index at 88M rows is OUT of scope -- and that rotation does not change that, since a query reads every sub-index at once so per-device demand sums them; only SHARDED spreads the footprint across cards. The aggregate gate's timing in end(), which refuses only after the whole build is paid for, is recorded as a known gap rather than silently accepted. One test is deliberately absent. A check that the growth claim is sized from capacity rather than from the inserted rows passed against a deliberately broken claim, twice, so it was measuring something else; shipping it would have looked like coverage without being any. The two-builder barrier test, which does discriminate, stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
fixed |
The rejected alternative was one glib sentence about throughput, which is the weakest of its reasons and did not say what the alternative WAS. Nor was "widen device_build_mutex" right: that mutex is taken inside build(), around the cuVS call, because the kmeans workspace is not re-entrant -- while the capacity buffers are allocated in the index constructor, from InitEmpty at the start of CREATE INDEX. The allocations were never on that lock's path, so there is nothing to widen. Serialising them would mean holding a lock from the constructor through the whole ingest to the end of the build: two independent CREATE INDEX statements, each running for minutes, serialised at statement level. That is a DDL concurrency change rather than a locking detail, it still does not cover loads (which run on the search path from unrelated sessions and take no build lock), and it does not fix a single oversized build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aptend
left a comment
aptend
left a comment
There was a problem hiding this comment.
Deep re-review of exact head 6532ec70d7b7bdd828792c9fd0436e03cfdda4d7, including the complete diff, every prior review/comment/reply/thread, and the delta after my previous reviewed head e6447112ae9678a5347bd545191a56445d97c976. The explicit/VRAM staging bound, 32-row shard alignment, and staging-page materialization close the previously reported cases, but two host-admission blockers remain: the planner still rejects valid host-limited rotation, and native geometric growth underclaims the transient vector reallocation peak. CPU-visible build/vet/tests and focused race tests pass; a focused injected-availability counterexample fails as described inline. CUDA execution was unavailable on this host, while exact-head CI is green.
aunjgr
left a comment
aunjgr
left a comment
There was a problem hiding this comment.
Reviewed exact head 6532ec70d7b7bdd828792c9fd0436e03cfdda4d7 against base 705d5fbb625d32090b9a8acadcead08c3e1d4535. The aligned SHARDED bound and staging-page materialization close the previous cases, but two deterministic host-admission failures remain.
-
The planner still cannot admit a valid host-limited rotated build because
stagingRowBoundknows only source rows, explicit capacity, and the VRAM bound. The missing host capacity is then calculated only after subtracting the oversized staging charge. Concrete case: 1 GiB measured host availability gives an 805,306,368-byte budget; IVF-PQ f32 dim=768 -> int8, 1,000,000 source rows, no explicit capacity, VRAM fit >=1,000,000, and a large train limit charges 3,072,000,000 staging bytes and rejects immediately. Yet a 209,279-row sub-index fits: about 642,905,088 staging bytes plus 162,400,504 bytes of int8 vectors/IDs, under the same budget and above the 1,024-list minimum, so five rotated sub-indexes are valid. Solve the staging/capacity dependency as one bounded capacity equation and cover host-limited rotation for both algorithms. -
Native geometric growth claims only the net capacity delta, but
std::vector::resizeallocates/materializes the complete replacement buffer while the old buffer remains live for relocation. Availability already accounts for the old buffer, so admission must cover the full new allocation peak, notnew-old. For intended final arena S=75% of initial free F with old capacity S/2 resident, current availability is 0.625F. The net S/2 claim fits the 75%-of-current budget (0.46875F), then resize must allocate S=0.75F while only 0.625F is available and can hit cgroup OOM before freeing old. Claim the complete replacement buffers or use a no-overlap growth strategy, and regress the real geometric reallocation boundary.
The new design document clarifies the CAGRA 88M scope, and the previous budget-policy, shard-alignment, and unfaulted-slack issues are otherwise closed.
XuPeng-SH
left a comment
XuPeng-SH
left a comment
There was a problem hiding this comment.
Deep re-review of exact head 6532ec70d7b7bdd828792c9fd0436e03cfdda4d7, including the complete prior change map and the delta after my previous review at e6447112ae9678a5347bd545191a56445d97c976. The design/scope gate is now documented, and the previous per-source staging charge, SHARDED 32-row alignment, and unfaulted staging-slack cases are materially addressed. Two deterministic host-memory blockers remain:
-
[P1 correctness / supported-build rejection] Host-limited rotation is still circular and can be rejected before a valid capacity is derived. Both create paths compute
stagingBytesfromstagingRowBound(srcRowCount, requestedCapacity, rowsFit)and then callHostRowsFitting(hostPerRow, stagingBytes)beforeplanCapacity(ivfpq_create_gpu.go:521-560,cagra_create_gpu.go:465-515). Thus the staging bound cannot include the host-derived capacity, even though nativestaging_bound_rows()ultimately caps the arena by that final per-sub-index capacity. Concrete IVF-PQ case: 1 GiB measured host availability gives an 805,306,368-byte budget; f32 dim=768 -> int8, 1,000,000 source rows, no explicit capacity, VRAM fit >=1,000,000, and a sufficiently large train limit charges 3,072,000,000 staging bytes and fails atHostRowsFitting. But a 209,279-row rotated sub-index fits the same budget: 642,905,088 bytes of raw staging plus 162,240,504 bytes of int8 vectors and int64 IDs = 805,145,592 bytes, and it remains above the default 1,024-list minimum. Please solve staging and host capacity as one bounded/fixed-point equation (or equivalent conservative formulation) and add injected-host-availability regressions for valid host-limited rotation in both IVF-PQ and CAGRA. -
[P1 unhappy path / OOM] Geometric staging growth claims only the net capacity delta, while
std::vector::resizemust allocate and materialize the complete replacement buffer before the old buffer can be released. Inindex_base.hpp:1541-1560,growthisnew capacity - old capacity, but a reallocating resize transiently holds old + new. Current availability already reflects the resident old buffer; admitting only the delta therefore does not bound the next allocation peak. Example: if the intended final arena is S=75% of initial free F and the old arena S/2 is resident, current free is 0.625F. The net S/2 claim passes the 75%-of-current budget (0.46875F), but resize then needs a new S=0.75F allocation while only 0.625F is free and can be cgroup-OOM-killed before the old arena is freed. Claim the complete replacement allocation(s), or grow through a non-overlapping strategy, and test the real geometric reallocation boundary; the newTwoConcurrentStagingGrowthsCannotBothPasstest exercises the ledger directly and does not cover this vector replacement peak.
git diff --check is clean and exact-head ordinary CI is green. CUDA/GPU execution is not available on this host, so the CUDA path was reviewed statically; these two counterexamples follow directly from the exact-head sizing and allocation rules.
Three changes, all about refusing the right thing at the right time. A HARD CEILING on quantizer_train_limit (kMaxQuantizerTrainLimit = 1,000,000). The only bound was the device budget, which on a large card is millions of rows, so `quantizer_train_limit = 100000000` was taken literally -- 300 GB of raw base rows at dim 768 f32, discovered only when the capacity model refused the whole build. Enforced in quantizer_staging_rows, the one function both the Go planner and the native index resolve the sample through, so it binds at every entry point rather than only at the DDL parameter. CREATE INDEX REJECTS an over-large setting instead of clamping. The native clamp is the right backstop, but silently handing a statement less than it asked for leaves the operator believing they have a sample they do not have. Go reads the ceiling from its C++ owner rather than restating it. THE AGGREGATE VRAM GATE now runs after each sub-index is packed, not only at end(). PerDeviceDemand is monotone -- a sub-index only adds bytes to a device -- so a running total already over the ceiling guarantees the finished one is, and everything packed after that point is work thrown away. This gate suits the incremental form better than its load-path twin: its ceiling is total VRAM and does not move between checks. A partial refusal is worded as "at least N, after K sub-index(es)" so a fix is not sized from a figure that is still rising. SHARDED never reaches the incremental path -- planCapacity refuses to combine it with a split, so it is always one sub-index. Tested in both device layouts, including gpu_multi_simulation: with [0,0,0,0] every rank aliases onto one physical card, which therefore holds ALL the shards and is judged on their sum. Attributing per rank there would admit an index that only fits because the simulation claims cards that do not exist. Mutation-checked. Also: the host budget refusal now names max_index_capacity alongside quantizer_train_limit, since the sample is capped by a sub-index's capacity and an operator told only about the train limit would not find the other remedy. The design doc gains a reference-hardware envelope (512 GB RAM, L40S 48 GB, 64 CPU; ~10.7:1 host-to-VRAM). It exists so counterexamples have to be reachable from a real deployment: the largest arena the ceiling permits is 2.86 GB, 0.75% of that host budget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
std::vector does not grow in place: a growing resize allocates the replacement, copies, and only then frees the old buffer, so the transient peak is old + new. Availability already reflects the old buffer -- its pages are faulted -- so what still needs admitting is the WHOLE new buffer. Claiming new-capacity-minus-old under-admits by exactly the old buffer's size, which is the part a peak has to cover. With an arena at S/2 resident and a target of S = 75% of initial free, the delta passes the budget while the allocation itself needs S against less free than that, and the resize-based faulting makes reaching it deterministic rather than incidental. The regression exercises a REAL reallocation, which the two-reservation ledger test does not. That distinction turned out to matter: the two formulas are IDENTICAL on the first growth, where the old capacity is zero, so an earlier attempt at this test could not fail no matter what the claim was sized from. Only a second growth separates them. At dim 256 f32 the first lands at kStagingReserveFloorRows = 4096 rows (4.19 MB) and the second at 8194 (full 8.39 MB, delta 4.20 MB), so ~6 MiB of headroom refuses one and admits the other. Mutation-checked: restoring the delta fails it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arena is capped by the final per-sub-index capacity (native
staging_bound_rows), but capacity is what the host budget is being asked to
produce -- so sizing the arena first and subtracting it is circular whenever
the HOST is the binding constraint. It charges rows no sub-index could contain
and refuses rotations that fit: at 1 GiB of availability, an 805,306,368-byte
budget with a 1M-row train limit charges 3.07 GB of staging and errors, though
209,279-row sub-indexes fit with 776 bytes to spare.
Bounding the charge by requested capacity and the VRAM fit (the previous fix)
narrowed it but could not close it, because the host-derived capacity is the
one term that does not exist yet.
No fixed point is needed. The arena is min(stageLimitRows, capacity), so
capacity has two branches and exactly one is self-consistent:
capacity <= stageLimitRows -> budget / (perRow + perTrainRow)
capacity > stageLimitRows -> (budget - stageLimitRows*perTrainRow) / perRow
HostRowsFittingStaged solves it directly. Against the reported case it returns
209,279 -- the reviewer's figure -- and at the 100k default it returns 641,889,
which is why that case never failed.
The solve also makes a failure mode unreachable rather than guarded: reaching
the second branch means stageLimitRows < budget/(perRow + perTrainRow), hence
stageLimitRows*perTrainRow < budget, so the sample can never consume the budget
on its own however large the limit. An error branch written for that case was
dead and is gone.
stagingRowBound and cuvs.QuantizerStagingBytes lose their last callers and are
removed; the solver needs the ROWS the arena stops growing at, which
cuvs.QuantizerStagingRows now exposes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both P1s fixed. Head is f514993. 1 — host-limited rotation circularity. You were right, and my previous fix only narrowed it. Bounding the staging charge by Fixed by not solving it sequentially. Substituting the runtime rule into the budget constraint — budget ≥ capacity × perRow + min(stage_limit, capacity) × perTrainRow — the min splits it into two branches, each linear in capacity and each directly solvable, so no term is unknown at evaluation Against your repro it returns 209,279 — your figure — and the test asserts it's the largest that fits (one more row doesn't). At One thing that fell out: I'd written an error branch for "the arena alone exceeds the budget" and a subtest for it failed — 2 — replacement allocation peak. Confirmed and fixed: stage_rows_locked claims the full data_rows × dimension × sizeof(B) (and Your point that the ledger test doesn't exercise reallocation was correct, and there's a sharper reason: the two formulas are Also since the last review: the aggregate VRAM gate now runs incrementally after each sub-index is packed rather than only in Validation: 204/204 C++ tests, full GPU suite green, go vet ./... clean in both default and -tags gpu builds, gofmt clean. 1M docs/design/gpu_vector_index_memory_admission.md carries the supported scope, the reference-hardware envelope (512 GB RAM / L40S |
Self-review found the aggregate gate taking a count AND a completeness flag, and the create path passing len(demand) -- the number of DEVICES -- as the number of sub-indexes. Harmless only because it also passed complete=true, which makes the count unused; had the flag ever flipped, the refusal would have quoted a figure it never measured, and nothing would have caught it. builtSoFar replaces both: > 0 means still rotating and carries the count, <= 0 means complete. The two can no longer disagree because there is only one. Also scopes the "cannot hold one row" refusal so it names the quantizer sample only when one is actually staged -- storage wider than a byte stages nothing, and pointing at quantizer_train_limit there sends the operator to a knob that is not involved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Target 6fd3ce2..f514993 (three commits, 09:57–10:37) · 18 files, +573/−143 · no callable review workflow, so §1–§5 ran Findings 1 — Wrong argument at the final gate · non-blocking · FIXED (454ba36) 2 — Refusal named a knob that isn't involved · non-blocking · FIXED 3 — Stray file in the tree · FIXED Verified clean The solver. Both branches are self-consistent — branch B provably yields rows > stageLimitRows, since small > stageLimitRows Closure (§3). Go's stageRows and C++'s staging_row_limit() resolve through the same quantizer_staging_rows with the same Unhappy paths (Q1–Q3). The new mid-build refusal is the one genuinely new failure path: free() runs on both success and failure Validation 204/204 C++ · full GPU suite green · T below timer resolution → N=100, B=30s on all three new tests · owning packages raced once Exit gate: passes. Zero open blockers. |
main brings 62 commits (465 files). No conflicts, and no overlap with the GPU vector-index subsystem: nothing in cgo/cuvs, pkg/cuvs, pkg/vectorindex or pkg/common/system is touched. The only changes that reach this branch's build are parallelism flags in the root Makefile (NATIVE_BUILD_JOBS) and dependency ordering in thirdparties/Makefile. table_function.go gains two unrelated cases (change_watermark, table_changes) alongside the existing registrations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
merged with main. |
aptend
left a comment
aptend
left a comment
There was a problem hiding this comment.
Re-review of exact head 6ea8a338d37efbb244fedb44d36a68efd88463d2. The previous host-capacity circularity and native delta-only staging claim are individually addressed, but the two fixes do not compose: the Go plan only fits the final resident arena, while native geometric growth admits a full replacement buffer against the availability left after the old arena and permanent capacity are resident. This leaves one blocking issue inline.\n\nValidation: go test ./pkg/common/system ./pkg/vectorindex/memory ./pkg/sql/colexec/table_function -count=1 -timeout=600s passed; focused -race host-budget tests passed. A temporary deterministic model of the native growth sequence reproduced the refusal at the default 100k staging limit (then removed). GPU execution was unavailable on this host.
XuPeng-SH
left a comment
XuPeng-SH
left a comment
There was a problem hiding this comment.
Deep re-reviewed exact head 6ea8a33, including the delta after my prior review at 6532ec7 and the complete memory ownership/admission path.
The two prior findings are individually addressed: HostRowsFittingStaged now solves host-limited capacity together with the capped staging arena, and stage_rows_locked now claims the full replacement buffers rather than only the net capacity delta. They do not compose, however.
[P1 correctness/unhappy path] The Go planner proves only that final resident capacity plus the final raw-vector staging arena fits 75% of the initial availability. Native geometric growth happens after the capacity and old staging arena are resident, then asks host_memory_governor to fit the full replacement allocation inside 75% of the now-smaller current availability. This second application of the 75% headroom deterministically rejects plans returned by Go. The planner also omits staged IdT bytes: perTrainRow is dim*baseElemBytes, while stage_rows_locked grows both staging_data_ and staging_ids_ and the production IVF-PQ path supplies IDs.
Exact default-setting counterexample:
- measured available = 1,073,741,824 B; Go budget = 805,306,368 B
- f32 dim=768 -> int8, no INCLUDE; resident perRow = 768 + 8 = 776 B
- stage limit = 100,000; HostRowsFittingStaged returns capacity 641,889
- resident capacity = 498,105,864 B
- immediately before the 65,566 -> 100,000 geometric growth, old raw+ID staging = 201,943,280 B
- measured availability is therefore 373,692,680 B; native 75% budget = 280,269,450 B
- the full 100,000-row replacement is 308,000,000 B, so reserve refuses it by 27,730,550 B
Even the final-state equation is slightly over budget because 100,000 staged IDs add 800,000 B that the Go solver does not charge: actual final resident bytes are 806,105,864 B versus the 805,306,368 B plan budget.
Please establish one invariant shared by planning and native admission that covers permanent capacity, staged data+IDs, every old+new geometric transient, and concurrent claims without applying incompatible headroom rules. Valid options include planning against the real native growth sequence, a correctly scoped build reservation, or a non-overlapping growth strategy; simply relaxing native admission would restore the OOM risk this PR is meant to remove. Add a deterministic production-path regression that materializes the capacity, advances the real staging arenas through the 65,566 -> 100,000 boundary under injected 1 GiB availability, and proves every returned plan is executable for both IVF-PQ and CAGRA.
Q1: refusal unwinds without a leaked ledger claim. Q2: the shared governor closes concurrent decided-but-not-allocated races, but plan/execution policy mismatch remains. Q3: no unbounded retained control state was added; the blocker is a deterministic supported-build refusal, and loosening it incorrectly would become cgroup OOM.
Exact-head ordinary CI and diff check are green. Local test binaries built, but execution on this macOS host is unavailable because libmo.dylib is absent; GPU execution was reviewed statically. The arithmetic follows directly from the exact-head production formulas.
…size HostRowsFittingStaged proved only that capacity plus the FINAL arena fits the initial 75% budget. That is not the figure the native side claims. stage_rows_locked grows the arena geometrically, and a growing resize does not extend in place: it allocates the new buffer, copies, and only then frees the old one. It therefore claims the whole replacement while the old buffer is still resident -- and host_memory_governor::reserve applies 75% to the availability left at that moment, with the permanent capacity already faulted in. It also stages ids beside the vectors whenever the caller supplies them, which the production IVF-PQ and CAGRA build paths always do (one row per call, with idBuf), while the callers passed only raw-vector bytes. Both gaps admit plans the governor then refuses deterministically. At 1 GiB of measured availability, f32 dim=768 -> int8, no INCLUDE columns and the 100k default train limit, the planner admitted 641,889 rows; the 65,566 -> 100,000 growth then claimed 308,000,000 bytes against a 280,269,510-byte budget. A sweep of 280 default-setting configurations found 26 such plans. Charge 2 * (rawRowBytes + stagedIdBytes) per staged row instead. The factor is sufficient because the superseded buffer can never exceed the bound: with C = budget - 2*L*s, the availability left at the final growth is at least 0.25*A + L*s, so the native budget is at least 0.1875*A + 0.75*L*s, which covers the L*s claim whenever L*s <= 0.75*A -- and admitting any rows already requires 2*L*s < 0.75*A. The same sweep refuses none after the change. That argument uses only "at most one superseded buffer is resident, and the claim is its replacement", never the growth schedule, so the factor survives a change to the doubling ratio or the reserve floor (verified across factors 1.5x-8x and floors 1-65536). stage_rows_locked now carries a pointer back to the planner naming what would actually invalidate it. The joint solve is unchanged in shape, and the property that made it worth having survives: a huge train limit still cannot starve capacity, because the arena is bounded by the capacity it shares the budget with. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
fixed |
…rowth The prior commit made the planner charge the staging arena's replacement peak, but the only regression covering it was a Go MODEL of the native rule. A change to stage_rows_locked would not have failed it. Walk the REAL growth rule instead, and submit each growth to the REAL admission rule. staging_grow_rows moves to namespace scope so a test can call the same function stage_rows_locked calls, rather than restating the schedule; only the availability reading is injected, at the value it would hold with capacity and the superseded buffer already resident. The case is the reported one: 1 GiB measured availability, f32 dim=768 -> int8, no INCLUDE columns, the 100k default train limit. The test asserts the steady state fits ids included (charging raw bytes alone left it 799,496 bytes over), that every growth in the sequence is admitted, and that the sequence is still 4096 -> 8194 -> 16390 -> 32782 -> 65566 -> 100000, so a schedule change surfaces here instead of silently moving which transient the planner must cover. Transcribing the pre-fix charge into the test fails it at the 65,566 -> 100,000 growth, which is the defect this covers. One walk covers IVF-PQ and CAGRA deliberately: staging_data_, staging_ids_ and stage_rows_locked live in the shared index_base, so both algorithms reach the rule through the same code and a second walk would re-run the same arithmetic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
fixed |
|
Two things worth saying back to them
|
aptend
left a comment
aptend
left a comment
There was a problem hiding this comment.
Re-reviewed exact head ec19bf7. I read the complete review/comment/thread history, compared the increment since the last reviewed head, and rechecked the full diff. The prior host-rotation and replacement-allocation blockers are closed. CPU unit/race tests and vet pass; current CI is green. One blocking plan/runtime consistency issue remains below.
| perTrainRow = dim * baseElemBytes(u.baseOid) | ||
| var serr error | ||
| // Probed on the PRIMARY gpu, which is where submit_main runs. | ||
| stageRows, serr = cuvs.QuantizerStagingRows(u.devices[0], perTrainRow, |
There was a problem hiding this comment.
[P1] Keep one staging ceiling for planning and execution
This snapshot sizes host capacity, but native staging_row_limit() calls quantizer_staging_rows() again on the first quantized row, and the resolved stageRows is never written into idxcfg or the native build params. When GPU free memory rises between the two calls (for example, an unrelated index is released), native chooses a larger host arena than the plan charged.
Concrete counterexample: 1 GiB host available, vecf32(768) -> int8, no INCLUDE. If this probe returns 10,000 rows, HostRowsFittingStaged admits capacity 958,384. If native later re-probes 100,000, the 32,782 -> 65,566 growth sees 743,705,984 B of capacity plus a 100,968,560 B old arena resident, leaving a 171,800,460 B 75% budget, but claims a 201,943,280 B replacement. AddRow deterministically fails although planning admitted it. CAGRA has the same split source of truth.
Persist/pass the exact resolved ceiling into native, or plan against a stable upper bound, and cover the low-planning-probe -> higher-native-probe transition.
7 participants
What type of PR is this?
Which issue(s) this PR fixes:
issue #27356
What this PR does / why we need it:
Makes IVF-PQ, IVF-Flat, and CAGRA index builds land correctly on a single L40S (45 GB VRAM) at wiki_all 88M scale, and pulls in the cuVS upgrade that closes a non-deterministic CAGRA build SIGSEGV.
AWS Instance Type: g6e.16xlarge
512GB Memory,
Single L40S GPU (45GB VRAM)
64 CPU
Summary
cuvs::neighbors::cagra::detail::graph::kern_prune(kernel-level bug duringgraph::optimize) that corrupted the CAGRA graph and eventually SIGSEGV'd insidecagra::serialize. Full extend / merge / deserialize migration to the 26.06+ typeddevice_padded_index<T, IdxT>API. Docker + Makefile pinned to CUDA 13.3.1 + cuvs 26.8.cuvs.RowsFittingFreeMem) is reachable from Go. Auto-clamped sub-index rotation gets a clean checkpoint after each sub-index.host_matrix_viewand let it stream device batches internally, instead of uploading the entire dataset as a device buffer. IVF-PQ especially: dropped the dataset term fromperRow, so a wiki_all 88M single-mode build fits an L40S without rotation./tmp), and build-artifact cleanup was made reliable for the rotation path.bench_wiki88,bench_hostview,bench_upload_costproject the 88M workload against a 20 GB reference card; new BVT cases exercise cagra and ivfpq multi-sub-index rotation.Verified end-to-end on this branch
cagra::serialize(cuvs bug)test_cuvs_worker(C++ unit tests)pkg/cuvs/...Go testsoptools/images/gpu/Dockerfile)CREATE INDEXOKKnown limitations / follow-ups (not blocking this PR)
vpq_f16_indexsearch path OOBs regardless ofsearch_algo— NVIDIA/cuvs#2489 filed with a self-contained repro. Once upstream ships a fix in a stable cuvs release, MO can move to a 2-sub-index-per-88M VPQ shape.conda env create -f optools/images/gpu/go_cuda-133_arch-x86_64.yamlrequires therapidsai-nightlychannel until the cuvs 26.8 stable release is picked up on the plainrapidsaichannel.Test plan
MO_CL_CUDA=1 make clean && MO_CL_CUDA=1 make -j8builds under cuvs 26.08 conda envtest_cuvs_worker— 162 / 162 passMO_CL_CUDA=1 .claude/skills/mo-dev/scripts/mo-cgo-test -count=1 ./pkg/cuvs/...— passbenchmark_cuvs— 24 index/dtype/batching combinations complete cleanly with recall ≥ 0.996bench_wiki88 88000000— 88M IVF-PQ projection completesCREATE INDEXon wiki_all — 63 min, tar 18.98 GB, recall 0.875 warmCREATE INDEXon wiki_all subset — 12 min, no crash, recall 0.886docker build -f optools/images/gpu/Dockerfile -t mo-gpu:cuda133 .→ 8.97 GB image;docker run --device nvidia.com/gpu=all ...reaches ISCP-Task Start; CAGRACREATE INDEXinside the container succeedsvector_cagra_multi_subindex,vector_ivfpq_multi_subindex