Skip to content

fix(cache): treat empty paged length list as a benign no-op#826

Merged
inureyes merged 1 commit into
mainfrom
fix/issue-823-empty-paged-sync-noop
Jul 19, 2026
Merged

fix(cache): treat empty paged length list as a benign no-op#826
inureyes merged 1 commit into
mainfrom
fix/issue-823-empty-paged-sync-noop

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

During batched speculative decode the Failed to sync paged state for seq-N: CachePool: expected 32 paged layer lengths for seq-N, got 0 warning spammed the server log hundreds of times per burst. The underlying Err was benign: an empty per-layer length list is the model-owned "nothing to mirror yet" case, not a real length divergence. This makes the empty case a legitimate Ok(()) no-op so the spam disappears while a genuine mismatch still errors.

Root cause

The model-owned paged families (gemma3/gemma4/llama4/qwen3_5/qwen3_next) build the per-layer target_lens for CachePool::sync_paged_state_with_lengths from their per-sequence sequence_caches (src/models/gemma3.rs, llama4.rs, qwen3_5.rs). A just-admitted or speculative/draft row has empty sequence_caches before its first model-owned forward, so visible_lens is an empty vec and an EMPTY length list reaches the sync. The length-count guard in sync_paged_state_with_lengths treated 0 != state.layers.len() as a divergence and returned the "expected N ... got 0" error, which scheduler.rs swallowed into a warn! per tick per row. The model-owned paged family uses the paged backend for SHADOW block-table accounting only, so with no populated caches there is genuinely nothing to mirror on that tick.

What changed

  • src/lib/mlxcel-core/src/cache.rs: in sync_paged_state_with_lengths, add an early return Ok(()) when target_lens.is_empty(), before the target_lens.len() != state.layers.len() count check, with a comment explaining the model-owned "nothing to mirror yet" no-op and why it is distinct from a nonzero partial count. The nonzero-mismatch error path is unchanged; a real desync still returns Err.
  • The fix lives in the shared CachePool method so every model-owned family is covered by one change. The model-owned sync_sequence_storage impls, the scheduler.rs swallowed-Err warn! (which correctly fires only on a genuine Err now), and the paged block-table accounting are untouched.
  • Added a CachePool unit test sync_paged_state_with_lengths_empty_is_benign_noop: a 2-layer model-owned shadow state accepts an empty length list as an Ok no-op that leaves the layer lengths unchanged, while a nonzero-but-wrong count still returns the mismatch Err. Pure CachePool unit test, no model load.

Test plan

  • cargo test -p mlxcel-core --lib --features cuda sync_paged_state_with_lengths_empty_is_benign_noop (pass)
  • cargo clippy -p mlxcel-core --lib --features cuda -- -D warnings (clean)
  • cargo test -p mlxcel-core --lib --features cuda cache single-threaded: the new test and all sync/paged tests pass; the only 2 failures (cache::paged_detach::tests::detached_paged_set_accounts_real_pool_bytes, cache::paged_detach::tests::trim_detached_paged_to_enables_partial_prefix_adoption) are pre-existing on clean main (identical assertion failed: handle.offset >= target_tokens in paged_detach.rs, an unrelated module) and are not touched by this change.

Closes #823

During batched speculative decode the model-owned paged families (gemma3/gemma4/llama4/qwen3_5/qwen3_next) build the per-layer `target_lens` for `CachePool::sync_paged_state_with_lengths` from their per-sequence `sequence_caches`. A just-admitted or speculative/draft row has empty `sequence_caches` before its first model-owned forward, so an EMPTY length list reaches the sync. The length-count guard treated that as a divergence and returned `CachePool: expected 32 paged layer lengths for seq-N, got 0`, which the scheduler swallowed into a `warn!` that spammed the log hundreds of times per burst.

An empty length list is genuinely "nothing to mirror yet": the model-owned shadow paged state has no caches to populate on that tick, so the sync is a real no-op, not a length divergence. Add an early `Ok(())` for the empty case before the length-count check. A NONZERO count that differs from the layer count is still a real desync and still errors, unchanged. The swallowed-Err `warn!` at scheduler.rs is left as is; it now only fires for a genuine mismatch.

The fix lives in the shared `CachePool` method so every model-owned family is covered by one change; the model-owned `sync_sequence_storage` impls and the paged block-table accounting are untouched.

Add a `CachePool` unit test: a 2-layer model-owned shadow state accepts an empty length list as an Ok no-op that leaves the layer lengths unchanged, while a nonzero-but-wrong count still returns the mismatch Err.

Closes #823
@inureyes inureyes added status:review Under review type:bug Bug fixes, error corrections, or issue resolutions priority:medium Medium priority area:inference Generation, sampling, decoding (incl. speculative, DRY) status:done Completed and removed status:review Under review labels Jul 19, 2026
@inureyes
inureyes merged commit 58bec14 into main Jul 19, 2026
5 checks passed
@inureyes
inureyes deleted the fix/issue-823-empty-paged-sync-noop branch July 19, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:medium Medium priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(server): investigate the "Failed to sync paged state ... got 0" warning spam in batched speculative decode

1 participant