fix(server): survive MLX decode-eval throws instead of aborting the worker

[inureyes]

Summary

The batch scheduler decode loop force-evaluated on-device tensors through the infallible mlxcel_core::eval / async_eval wrappers, which cxx compiles as noexcept shims, so any MLX C++ throw reached from decode (the graph-cache abort from #818, an allocation failure, an internal invariant) crossed the FFI boundary uncaught and called std::terminate, killing the whole server and dropping every in-flight request with no request-level error. This routes the decode/prefill evaluation through a fallible boundary so a backend throw fails just the affected request(s), not the process.

What changed

• src/lib/mlxcel-core/cpp/mlx_cxx_bridge.{h,cpp} + src/lib/mlxcel-core/src/lib.rs: add try_async_eval (body identical to async_eval, declared -> Result<()> so cxx catches any thrown MLX exception at the FFI boundary), mirroring the existing try_eval.
• src/server/batch/scheduler.rs: route every decode and chunked-prefill evaluation site through the fallible try_eval / try_async_eval variants and translate an Err into the existing abort_sequence path, so the worker keeps serving the other sequences. A bounded consecutive-failure guard (MAX_CONSECUTIVE_EVAL_FAILURES = 8, reset on any success) treats a persistently-failing backend as unrecoverable: past the threshold it logs a fatal line, aborts the remaining in-flight sequences, and stops the scheduler loop cleanly instead of spinning.

Scope and risk

Not every MLX throw is safely recoverable: the graph-cache throw happens at graph capture before any state mutation, so failing just that batch is clean, but an out-of-memory or internal-invariant throw can leave the allocator or device undefined. That is why the guard exists; the goal is a graceful request-scoped failure or a clean shutdown, not pretend-recovery. The success path is unchanged (try_* is the same MLX call plus a branch on Result).

Test plan

• try_async_eval_ok_matches_async_eval exercises the fallible boundary's success path (schedules a valid graph, returns Ok, reads back the expected value).
• Scheduler failure-counter logic covered by pure-function unit tests: reset on success, increment and saturate on failure, threshold inclusivity, guard tripping after consecutive failures, single-success reset.
• Full CUDA build (make release-cuda) passes; cargo check --lib --features cuda clean (no Rust warnings). Server decode path runs normally end to end (3 concurrent chat completions complete correctly through the fallible-eval loop, no regression).

Closes #822

[inureyes]

Implementation Review Summary

Intent

Route every batch-scheduler decode/prefill eval through the fallible try_eval / try_async_eval cxx boundary so an MLX C++ throw fails just the affected request(s) instead of std::terminate-ing the worker, guarded by a bounded consecutive-failure shutdown.

Findings Addressed

• Lookahead prime leaked a speculative KV position on a caught try_async_eval throw (HIGH) — fixed in eb3e8ff.

prime_lookahead_with_input returned None on the new caught-throw path without unwinding the speculative KV position lookahead_forward had already appended per sequence. The caller set decode_lookahead = None (and the steady path committed step n while dropping step n+1), so the next synchronous decode ran on a cache whose KV tail was one position longer than generated_tokens accounted for — a desync that corrupts the sequence(s) or forces a batch-wide re-throw, defeating the request-scoped recovery this change is meant to provide. It is a newly reachable path: the previous infallible async_eval could only hit the caches-vanished None (before any append). Fixed by trimming the one speculative position via the existing apply_lookahead_trim(seq_ids, lookahead_teardown_positions(false)) teardown before bailing, mirroring the stale/bootstrap fallbacks; the caches-vanished ? path is untouched (it returns before any append). All prime paths (bootstrap, stale, finishing re-prime, steady step n+1) funnel through this function, so one fix covers them.

Remaining Items (non-blocking, informational)

• Batched-fused decode readback (batched_fused_sample) and the primed-token readback (lookahead_tokens_to_host → array_evaluated_bytes) still materialize through infallible host-copy paths (LOW) — outside the enumerated eval/async_eval scope of fix(server): catch MLX C++ aborts at the decode FFI boundary so a backend throw fails the request, not the process #822 (the fix(server): speculative serving aborts the whole process via MLX CUDA graph-cache "Cache thrashing" throw (lifetime miss counter vs default capacity 400) #818 throw fires at graph capture, caught by the new try_* sites); the fused case is already documented in-code as a follow-up.

Verification

• All stated requirements implemented — all 10 enumerated eval sites converted (9 try_eval + 1 try_async_eval); no infallible eval/async_eval remains in the scheduler.
• No placeholder/mock code remaining.
• Integrated into project code flow — conversions live in the real decode/prefill functions reached via run() → decide_action(); shutdown_requested is honored at the loop head; guard drains both active_batch and chunked_prefill_seq without duplicate error events.
• Project conventions followed — try_*(&x).map_err(...) matches the whisper/audio fallible-eval pattern; reuses the existing abort_sequence / abort_sequence_with_error / apply_lookahead_trim helpers.
• Existing modules reused where applicable.
• No unintended structural changes — deferred-init prefill_eval borrow handling is sound; success path is unchanged (same MLX call plus a Result branch).
• Consecutive-failure guard correct — reset on success, saturating increment on failure, count >= 8 threshold; no off-by-one (unit-tested).
• Compiles clean — cargo check --lib --features cuda (no Rust warnings) after the fix.