fix(cuda): restore DeepSeek-V2 output by reverting the regressed RMSNorm kernel

[inureyes]

Summary

Bumps the vendored MLX pin to the latest upstream (b7c3dd6d) and adds a CUDA RMSNorm overlay that fixes the DeepSeek-V2-Lite generation regression tracked in #824. The MLX pin stays at the latest commit; only the one regressed kernel file is overlaid back to its last-known-good version.

Root cause (pinned by a deterministic bisect)

Upstream MLX #3792 ("Fix CUDA RMSNorm small-row dispatch", commit a5a684d, first shipped in the 0.32.1 pin) regressed the small-axis CUDA RMSNorm. DeepSeek-V2's MLA kv_a_layernorm normalizes over kv_lora_rank (512), which with N_READS == 8 for half-precision activations lands on the multi-warp block_dim == 64 dispatch config that #3792 changed (groups_per_block 2 → 1). The later #3850 rewrite ("RMSNorm forward speed up") kept the same axis broken. The result was a deterministically-wrong normalizer that degenerated DeepSeek-V2 generation into repeated punctuation. Models whose norm axis is larger (qwen3 hidden_size 1024+, etc.) take a different dispatch path and were unaffected, which is why only DeepSeek-V2 broke.

MLX_USE_CUDA_GRAPHS=0 exposes the fault as deterministic; compute-sanitizer initcheck and memcheck are both clean, so it is a numerical kernel error, not a memory-safety bug. Bisect boundary: 1700b39 (commit before #3792) is coherent, a5a684d (#3792) is garbage. This corrects #817's earlier "memory-safety nondeterminism, not bisectable" reading (the graphs-on layer masked the determinism).

Fix

Overlay only mlx/backend/cuda/rms_norm.cu with the last-known-good upstream version (1700b39a, immediately before #3792), via the existing patches/mlx/backend/cuda/ overlay mechanism. Its helper dependencies (load_vector, BlockBroadcastReduce, dispatch_float_types, contiguous_copy_gpu) are unchanged between that commit and the pin, so it compiles as-is against the latest MLX. This trades the #3850 RMSNorm speedup for correctness on the affected small-axis path. The pin is updated in all three synchronized locations (mlx-cpp/CMakeLists.txt GIT_TAG, mlxcel-core/build.rs MLX_EXPECTED_COMMIT, .github/workflows/release.yml).

Verification (GB10)

• deepseek-v2-lite-4bit greedy, MLX_USE_CUDA_GRAPHS=0: 4/4 coherent (was 0/4 garbage before the fix).
• No regression: qwen3-0.6b-4bit, qwen2.5-0.5b-4bit (answers "Paris"), llama-3.2-1b-4bit all coherent and unchanged.

Scope / follow-ups

• The residual graphs-on nondeterminism for DeepSeek-V2 is the pre-existing CUDA-graph-capture layer documented in fix(models): dedicated root-cause debugging of the DeepSeek-V2-Lite greedy nondeterminism (follow-up to #817) #824 (present even before this regression, and at the healthy 2026-05-19 benchmark state); it is out of scope here.
• Report the small-axis RMSNorm regression upstream to ml-explore/mlx and drop this overlay once fixed there.

Closes #824

---

Update: second commit adds the CUDA-graph-capture fix (required for the server)

The RMSNorm overlay above fixes the deterministic forward, but DeepSeek-V2 also has the same CUDA-graph decode hazard as Gemma 4 (#688): with graph capture on, incremental decode degenerates into repeated tokens even though the identical forward is coherent with capture off. This is pre-existing (present before the RMSNorm regression) and dominates the mlxcel-server path (graph capture on by default), so the kernel fix alone left the server returning garbage.

Second commit extends the established MLX_USE_CUDA_GRAPHS=0 startup lever to the DeepSeek-V2 family (maybe_disable_cuda_graphs_for_gemma4 generalized to maybe_disable_cuda_graphs_for_model, now covering Gemma 4 / DeepSeek-V2 / DeepSeek-VL2). An explicit operator MLX_USE_CUDA_GRAPHS still overrides.

End-to-end deployment verification (GB10): mlxcel-server -m deepseek-v2-lite-4bit at default settings now answers coherently 5/5 ("The capital of France is Paris."), where it returned The!!!! garbage before. Startup log: "DeepSeek-V2 detected: disabling CUDA graph capture ... issue #824".

Both layers of #824 are now resolved: the RMSNorm numerical regression (overlay) and the pre-existing graph-capture hazard (startup lever). The upstream small-axis RMSNorm bug report to ml-explore/mlx remains a follow-up.