feat(rlix): lower whole-GPU residual gate default to 7.0 GiB; preload dual smoke

[zhenyulincs]

What

Lower the MILES_MAX_RESIDUAL_GPU_MEM_GB default from 13.0 to 7.0 at both call sites (hard gate in miles_pipeline.py, diagnostics mirror in miles_coordinator.py), switch run_smoke_dual.sh to MILES_TMS_HOOK_MODE=preload, and extend the gate's RuntimeError with the hook-mode coupling hint + rollback knobs. Wiring tests updated; docs rows appended (append-only).

Why — measured basis

2026-07-05 audit (4x RTX 5090 cu130; details in #17's description):

torch hook mode : tms pause releases only 2.76 GiB (init weights);
                  optimizer state (~5.7 GiB) never offloads;
                  whole-GPU residual 11.07 GiB at cycle 2  -> old default 13.0
preload mode    : everything pausable; residual stable 5.02 GiB -> default 7.0

Dependencies — status

1. fix(rlix): gate MILES wake on per-process residual GPU memory (R02-01) miles#5 — MERGED (2026-07-05). Demotes the per-engine /server_info accounting assert to diagnostics. Required: its ~9.3 GiB accounting value is NOT physical residency; a paired E2E without it false-failed at 7.0 exactly there (it passed at 13.0 only by numerical coincidence).
2. No other miles change needed. Preload on Blackwell is authorized via the guard's existing escape hatch — run_smoke_dual.sh in this PR exports MILES_TMS_ALLOW_PRELOAD_ON_BLACKWELL=1 (reaches actors via raylet env inheritance; exported before ray start). The planned guard-relaxation PR feat(miles): allow tms preload on Blackwell with cu13+ torch wheels miles#24 was closed as unnecessary; the cu12.9 segfault it would have gated on is confirmed gone on cu130 images (audit in fix(rlix): gate MILES wake on whole-GPU residual memory (R02-01) #17).

Compatibility / rollback

torch-hook deployments (pre-cu13 Blackwell): export MILES_TMS_HOOK_MODE=torch + export MILES_MAX_RESIDUAL_GPU_MEM_GB=13 (documented in the smoke scripts and appended docs rows; run_smoke_e2e.sh / run_smoke_inject_fault.sh ship with exactly that pinning). The gate's error message now names this remedy explicitly.

Tests

python -m pytest tests/test_miles_residual_threshold_wiring.py tests/test_env_utils.py -q
7 passed

E2E

Two dual-smoke validations on 4x RTX 5090 cu130, both EXIT_CODE=0: (a) hatch-authorized preload on unmodified miles (the route-A audit run, residual 5.02 GiB); (b) the 7.0 gate + miles#5 + preload (authorized via a since-dropped guard-relaxation build — runtime-equivalent to the hatch path, which only differs in how preload is authorized at actor init); hook_mode='preload' in actor logs; SGLang per-engine diagnostic 1.699-1.719 GiB; whole-GPU gate readings 3.45-3.47 GB (cycle 1) / 5.02-5.04 GB (cycle 2) vs threshold 7.00 GB, stable across cycles. Known teardown-time RolloutManager HTTP noise only.

Tail investigation update (2026-07-05, supersedes the earlier "lower to ~3.0" plan)

The ~1.6 GiB train-created tail was root-caused. Two findings change the roadmap:

1. The torch_memory_saver.disable() weight-sync-window suspicion is refuted for rlix mode. A mock confirmed the mechanism exists (allocations inside disable() do escape pause()), but rlix mode never executes that window: weight sync goes through run_sync_session (GPU→CPU copies only; update_weights() and its disable() wrap at miles actor.py:570 are not called — see rlix_train_loop.py:59).
2. The tail is non-torch-allocator process footprint, which cannot be offloaded. Under preload, pause releases 100% of torch-allocator memory (8.8 GiB released ≈ weights 4.68 + optimizer 4.08). The residual decomposes as: 1.23 GiB after init (CUDA context + NCCL base) → +0.5 GiB during the init bucket-cache window → +1.58 GiB at the first train step (CUDA context growth from lazy kernel/module loading for TE/flash-attn/fused ops, cuBLAS-class workspaces, NCCL slabs) → stable at 3.32 GiB across all later cycles. Loaded kernel modules cannot be unloaded, so ~3.3 GiB is the per-process floor for a Megatron actor that has trained at least once.

Consequence: gate-GPU floor = 3.3 (Megatron) + 1.7 (sleeping engine) ≈ 5.0 GiB — exactly the measured gate reading — so 7.0 is close to the practical floor for this topology and a later ~3.0 default is NOT achievable. Remaining shaving candidates are a few hundred MB at best (e.g. torch._C._cuda_clearCublasWorkspaces() in clear_memory(), auditing destroy_process_groups() for surviving default-group NCCL slabs) and are tracked separately; they would not change this PR's 7.0 default.

Codex review: APPROVE_WITH_NOTES, 0 blockers.

Refs: #17 (audit evidence), plans/m11-review.review-report/R02.md (R02-01).