perf(server): rotate the speculative slice slot across waiting requests

[inureyes]

Summary

The tick-slice path (issue #734, PR #745) held the worker's single speculative slot for a request's whole generation: a second speculative-eligible request arriving mid-slice fell back to classic decode permanently, so one long stream monopolized speculative acceleration (the MEDIUM QoS finding from PR #745's review). This PR implements the issue's option (a), grant-based slot rotation at round boundaries, with priority-ordered granting plus a skip-cap anti-starvation floor as a hardened slice of option (b).

Design

While a slice is in flight, up to 2 further tick-slice-eligible requests are parked in a grant backlog (BatchScheduler::speculative_slice_backlog, one FIFO ring holding both parked in-flight jobs and not-yet-started waiters); the cap bounds a waiter's worst-case pre-first-token wait to about cap x budget speculative slices, and anything beyond it, or any request the slice path cannot serve, keeps the pre-#746 classic-decode fallback. Once the active job has executed MLXCEL_MTP_SLICE_GRANT_ROUNDS slices (new env, default 8; slice 0 counts as the first slice of an admission grant, enforced even at budget 1 via an immediate post-install expiry check; 0 disables rotation and restores the legacy whole-generation hold including the classic fallback for concurrent requests; unparseable values fall back to the default) with the backlog non-empty, it parks at the next round boundary and the slot is granted to the next entry. Grant order is priority lane first and FIFO within a lane, bounded by a skip-cap escalation: every non-selected backlog entry counts the grant decisions that skipped it, and once any entry has been skipped MTP_SLICE_GRANT_SKIP_CAP (2) times the next grant must select the most-skipped entry regardless of lane. This restores the #745 "everyone progresses" floor that strict lane precedence would otherwise lose for parked jobs (which sit outside the active batch and receive no classic decode while parked), and matters because priorities are client-assignable via the unauthenticated X-Priority header; sustained higher-lane pressure now yields bounded delay instead of a stalled stream. The budget is resolved from the env once per grant (at admission install and at promotion) and cached in a scheduler field, so the uncontended per-round added cost is a counter increment plus one comparison, with no per-round env read or allocation. A parked job resumes from its saved MtpSessionState and its round runs in the promotion tick (promotion is bookkeeping); a promoted waiter is re-checked against the adaptive B=1 policy verdict and routed to classic prefill if the policy settled to decline while it was parked, and an accepted waiter's slice 0 is that tick's single action, so the #734 HOL bound of one model action per tick holds across rotations. The expiry condition requires a non-empty backlog, so an uncontended request never rotates and behaves exactly as under #734, and a job that overran its budget while uncontended parks at the first round boundary after contention appears, so a newcomer waits at most about one round for the first handoff. Operators should note the ring transiently peaks at cap+1 entries between a park and the next promotion, and peak live speculative KV residency is up to 3 sessions (1 active + 2 parked) versus 1 before this PR.

Drafter handoff decision: the park boundary routes the worker's one drafter handle through the existing WorkerDrafterSlot::return_drafter / take plumbing, the same path the end of a session uses, rather than a raw no-reset handle handoff. Correctness argument, verified in code: (a) Gemma4AssistantDraftModel does not override Drafter::reset, so the return's reset is the trait default no-op (drafter/mod.rs:630); it unloads no weights and unbinds nothing; (b) the drafter's bind state (captured target embedding + resolved LM head) is derived from the worker's one target model and the promotion-time re-bind recomputes it identically (and is required in the defensive case where a failed reset dropped the handle and ensure_loaded reloaded from disk); (c) step_session re-arms the drafter's shared KV from the session's own stored MtpVerifyOutput at the top of every round (set_shared_kv is a stateless overwrite consumed only by the following draft_block), so rounds of another session run in between cannot leak state into a parked session. The step_session resumability contract doc (generator.rs) is amended to state this precisely.

Acceptance criteria

Two long concurrent speculative-eligible requests both receive speculative rounds over the run (rotation harness test asserts both jobs' round counts match their isolated runs). The rotation behavior is pinned by a scheduler-protocol harness driving the real policy functions (slice_grant_expired, next_grant_index) and the real slice driver with one shared drafter handle, asserting the exact grant schedules [(A,2),(B,2),(A,2),(B,1)] at budget 2 and [(A,2),(B,1),(A,1),(B,1),(A,1),(B,1)] at budget 1. The single-slice invariant and the #734 HOL bound are preserved: at most one job is active, promotions run at most one model action per tick, and all pre-existing slice_takes_tick alternation tests pass unchanged. Env-gate semantics are unchanged (MLXCEL_MTP_TICK_SLICE untouched; the new budget env only adds rotation, and 0 restores pre-#746 behavior). Greedy parity is unchanged: the rotation tests assert each request's committed stream is byte-identical to its isolated single-request run, on top of the existing slice-vs-burst identity test.

Review hardening

Three review rounds are folded into this PR. Implementation review: the waiter-promotion arm now re-checks mtp_b1_should_run() so a waiter parked while the adaptive policy was profiling is routed to classic prefill if the policy settled to decline (previously the comments claimed this but no code enforced it), and an admission grant at budget 1 parks immediately after slice 0 instead of holding 2 slices. Security review: the skip-cap anti-starvation escalation described above (lane starvation of a parked job was client-triggerable and had no classic-progress floor), the once-per-grant cached budget read (the round tail previously took the process env lock and allocated per round on the uncontended path, contradicting the documented counter-only cost), and the documented cap+1 ring peak / 3-session KV residency bound.

Tests

cargo check --lib --tests clean; cargo fmt --check clean. make test-fast-cuda FILTER=batch::speculative (also re-run as cargo test --profile test-fast --features cuda --lib batch::speculative): 73 passed, 0 failed. New: pure-function tests for the budget env (default 8 / custom / 0 / garbage), grant expiry (binds only under contention, budget 0 unbounded), backlog cap margins (admits below 2, rejects at and beyond, rejects all under budget 0, pinning the classic fallback at the margin), grant order (priority lane first, FIFO within a lane, skip-cap escalation with most-skipped-then-earliest selection, bounded selection under an endless high-lane stream), promotion-tick alternation; rotation harness tests for slot sharing + stream parity, budget-1 strict per-slice interleave, high-lane starvation pressure with a bounded inter-grant gap for the normal-lane job, declined-waiter routing, uncontended no-rotation, cancelled parked job, and cancelled waiter. All pre-existing speculative slice/burst tests unchanged and passing.

generator.rs is touched only in the step_session resumability contract doc comment: clause (b) previously required the drafter be held strictly without Drafter::reset, which the slot rotation technically violates; the amended clause states the real requirement (no draft-relevant state destroyed) and records the verified no-op-reset fact that makes the rotation correct.

Docs

docs/CONTINUOUS_BATCHING.md gains a rotation paragraph in the tick-slice section covering the grant budget, skip-cap escalation, budget read cadence, and residency bounds; docs/environment-variables.md documents MLXCEL_MTP_SLICE_GRANT_ROUNDS.

Closes #746