Skip to content

fix(qwen35): size mixed concurrent target-step graphs - #622

Draft
Graffioh wants to merge 1 commit into
Luce-Org:mainfrom
Graffioh:codex/ar-prefill-latency
Draft

fix(qwen35): size mixed concurrent target-step graphs#622
Graffioh wants to merge 1 commit into
Luce-Org:mainfrom
Graffioh:codex/ar-prefill-latency

Conversation

@Graffioh

@Graffioh Graffioh commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Problem

The Qwen target-step builder always allocates a 16,384-node graph. That is large enough for eight independent recurrent segments, but the default C16 continuous-serving path can legally schedule eight packed prefills plus compact decode in one traversal.

That ninth recurrent segment builds 16,741 nodes and aborts during graph construction. This reproduces on main with the normal continuous-decode scheduler; it is not caused by an experimental policy.

Fix

  • Keep the existing 16,384-node capacity through eight recurrent segments.
  • Add 2,048 nodes of capacity for each recurrent segment beyond eight.
  • Size the target-step graph from packed-prefill segment count plus compact decode.
  • Add a host regression test for the C16 mixed shape and larger future cohorts.

The observed C16 schedule now receives an 18,432-node graph. Common shapes at or below eight recurrent segments retain the existing capacity.

Scope

This PR is now only the graph-sizing correctness fix. The prefill-first scheduling experiment was removed after warmed three-repeat gfx1151 screening showed only a 1-3% C16 TTFT gain, a C8 TTFT regression, and materially worse overlap ITL. The existing invariant that every normal decode plan covers the complete live cohort remains unchanged.

Validation

  • Graph-sizing regression passed.
  • Scheduler plan: 62 checks passed.
  • Sequence-engine contract: 14 checks passed.
  • Complete server unit suite: 382/382 passed.
  • HIP server and focused targets rebuilt successfully.
  • git diff --check passed.

@Graffioh
Graffioh force-pushed the codex/ar-prefill-latency branch 2 times, most recently from 4742333 to 812eabc Compare August 18, 2026 10:42
@Graffioh
Graffioh force-pushed the codex/ar-prefill-latency branch from 30c47a7 to 5b7d306 Compare August 20, 2026 06:28
@Graffioh Graffioh changed the title perf(qwen35): improve concurrent AR prefill latency fix(qwen35): size mixed concurrent target-step graphs Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant