feat(speculation): add generic adaptive activation - #626
Draft
Graffioh wants to merge 42 commits into
Draft
Conversation
Wire the DSpark drafter heads (low-rank Markov bigram correction + confidence head) into the qwen35 spec-decode loop, so Qwen3.8-27B DSpark drafters (e.g. RadixArk/Qwen3.8-27B-DSpark) run with full head support: - spec loop: markov-corrected greedy chain (fused single-graph variant with non-fused fallback) replaces plain argmax projection when the drafter ships DSpark heads; DDTree candidate top-k gets the markov bias too. Env-gated: DFLASH_QWEN35_DSPARK, DFLASH_QWEN35_FUSED_DSPARK, DFLASH_QWEN35_DSPARK_TREE (all default on). - target capture layers now follow the drafter GGUF's dflash.target_layer_ids instead of the evenly-spaced derivation; the Qwen3.8 drafter is trained on layers 4/16/28/40/52, not 1/16/31/46/61. - draft loader: dflash.mask_token_id from the drafter GGUF wins over the family default (Qwen3.8 drafter uses 248077, default was 248070), and optional YaRN rope scaling keys are parsed into DraftWeights. - draft graph: rope calls honor the drafter's YaRN config (previously hardcoded plain NEOX rope). - Qwen35DFlashTarget exposes lm_head for the fused head path. - convert_dflash_to_gguf.py: handle single-file DSpark releases (markov/ confidence heads inline in model.safetensors), transformers>=5 nested rope_parameters and dflash_config.mask_token_id, and emit YaRN scaling metadata. The confidence-gate adaptive block length is not wired yet (q_len sizes the per-request step buffers); the chain runs with the gate off.
Verify/accept now run over v_len (the drafted chain's actual length) instead of the buffer-sizing q_len, so the DSpark confidence gate's adaptive block truncation is structurally supported. The gate itself stays off by default (DFLASH_QWEN35_DSPARK_CONFIDENCE_THRESHOLD=0): with the RadixArk Qwen3.8 drafter, any threshold in 0.1-0.5 truncates to the same short chain regardless of value, so the confidence scores coming out of the shared head path look mis-scaled and need a separate investigation before the gate can help. threshold=0 is bench-verified regression-free.
- ggml_ssm_conv_step: one kernel for the causal-conv decode/verify step (history window + silu(conv) + in-place history write-back + optional rollback window copy) replacing transpose/concat/ssm_conv/silu/cpy. - ggml_gated_delta_net_set_raw_gates: the GDN kernel applies sigmoid(beta) and softplus(alpha + dt_bias) * A itself. - ADD + RMS_NORM + MUL fusion (residual add materialized alongside the normalized output) in the CUDA/HIP graph evaluator. - legacy pool MAX_BUFFERS 256 -> 1024: LUCE_Q8_MEMO holds ~300 pooled buffers per evaluation; a full pool freed in-flight buffers with cudaFree and produced illegal memory accesses on long prefills.
Rename the RDNA small-tile macro to GGML_CUDA_MMQ_SMALL_TILE and apply it to IQ4_XS/Q4_K/Q5_K/Q6_K/Q8_0 in addition to the ROCmFPX formats. At spec-decode verify widths (N<=16) the 128-row tile leaves a 5120-row projection with only 40 blocks on a 64-CU gfx1201; 64x64/4-warp tiles measured +12-23% on those shapes (verify step 43.8 -> 39.7 ms on Qwen3.8-27B) at ~8% prefill cost.
- loader places attn_gate|attn_qkv and ssm_beta|ssm_alpha back to back and exposes zero-copy stacked aliases (L.wqkv_z, L.ssm_ba): one GEMV each instead of two (DFLASH_QWEN35_NO_STACK=1 disables). - FFN uses ggml_swiglu_split so the backend fuses gate/up/GLU into one vector kernel at decode. - DeltaNet block: single l2_norm over the q|k slab, ggml_ssm_conv_step, raw-gate gated_delta_net (in place, no state copy), no q/k head repeat (the kernel broadcasts). DFLASH_QWEN35_NO_FUSED_KERNELS=1 keeps the op-by-op graph for A/B. - DFLASH_KV_ROTATE=0 skips the FWHT K/Q rotation (precision-neutral with q8_0/f16 caches, two fewer launches per attention layer). Qwen3.8-27B IQ4_XS on R9700: plain decode 30.4 -> 33.8 tok/s with identical greedy output.
- Qwen35AdaptiveSpecPolicy: EMA of accepted draft tokens per step; below 0.8*(spec_step_ratio-1) the loop runs a burst of plain-decode steps (seed-only verify, no drafter/heads/snapshot/rollback, features still captured) and probes again afterwards. Env DFLASH_QWEN35_SPEC_STEP_RATIO (default 1.7, 0 disables) and DFLASH_QWEN35_AR_BURST (default 40). Low-acceptance prose 28.1 -> 32.4 tok/s, code/mixed unchanged. - Confidence gate now uses the fused Markov graph and truncates on the host; DFLASH_QWEN35_DSPARK_CONF_DEBUG=1 prints per-position scores. - spec-profile hooks for the chain path (project/snapshot/verify/ rollback/feature).
launch_fattn was told the vec kernel consumes D keys per step; it walks nthreads (128) per step, so a 256-key window at head_dim 256 ran as one block per head. Passing nthreads lets it use two blocks per head plus the combine pass: Qwen3.8-27B plain decode 34.3 -> 34.6 tok/s on R9700, identical output.
The first spec step after a plain-decode burst updates the acceptance EMA with alpha 0.5 so a stream that became predictable leaves plain decode immediately; step ratio and start value keep the measured best balance (45.7 / 31.8 / 40.4 tok/s code / prose / mixed).
The break-even acceptance now follows live EMAs of the spec-step and plain-step wall times (default 1.9 until both are measured), so it is right for any drafter block size (width-8 DSpark and width-16 DFlash measure ~1.8 on gfx1201).
DFlash 2 (z-lab/inco, e.g. z-lab/Qwen3.8-27B-DFlash2) is the DFlash backbone plus a grouped dynamic causal conv around attention and MLP in every layer and a candidate selector head (top-k lm_head candidates per block position, one path scored by a low-rank bigram form). - converter: maps attention_conv/mlp_conv (base kernels F32, kernel projections) and candidate_selector tensors, emits dflash2.* metadata, reads block_size from dflash_config, emits SWA pattern for drafters with causal sliding layers. - loader: DraftConvWeights per layer, DraftSelectorWeights, shape checks. - draft graph: conv prepare/finish (two taps over the block, per-element base + per-group dynamic coefficient) in both the stateless and the cached-KV builders. - selector chain: top-k via the target's GPU top-k (kMaxK 8 -> 16), one cached graph for hproj + codebook row gathers, host path search. - spec loop uses the selector before the DSpark/argmax paths. Qwen3.8-27B IQ4_XS on R9700, q8_0 drafter, greedy: 109.9 code / 50.7 prose / 111.8 mixed tok/s (DSpark drafter: 45.6 / 32.4 / 38.6); avg 5.9-6.0 accepted tokens per 8-token block on code, ~2.7 on prose.
With the 64-row/4-warp tile the mmq_x=32 instantiation runs at 180 GB/s on gfx1201 (17408x5120 IQ4_XS) against 443 GB/s at mmq_x=16 and 315 at 48, so N=17..32 batches (DDTree budgets, prefill remainders) took 2.4x longer than N=16 or N=40. Choose the next tile instead.
Add batched DFlash2 chain selection and a request-local benefit adapter feeding the sticky joint goodput gate, with explicit AR fallback and typed activation telemetry. Fix fractional gate pricing and executed-shape cost feedback, add variable positive chain depth, replay-journal prototypes, artifact conversion hardening, and fail-closed subset/refill analysis harnesses.
Graffioh
changed the title
Graffioh
changed the title
Graffioh
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Current behavior
--max-concurrencyis an upper slot capacity. Every round uses the current live decode count; the server does not assume all configured slots are occupied.(C - k) + depth * bucket(k)target rows in one launch. Verified attention K/V, target features, and DeltaNet transition journals are committed directly.DFLASH_CHAIN_DURABLE_REPLAY=1retains the previous two-pass executor as a compatibility escape hatch.What the benchmark measures
Refill means that when one request finishes, the client immediately submits another while queued work remains. Full-live tok/s counts only rounds where all
Crequests are decoding, so it measures saturated serving capacity. It is not a claim about idle, bursty, or draining traffic; finite refill tok/s additionally includes the underfilled terminal drain after the queue is exhausted.“Mixed” describes the request types in the cohort, not the load pattern. All full-live rows assume sustained queued work.
--max-concurrency=16remained fixed for every row, including C=1/C=4/C=8.Radeon AI PRO R9700
Radeon AI PRO R9700 (
gfx1201, ROCm 7.2.4), Qwen3.8-27B IQ4_XS target, DFlash2 Q8_0 draft, depth 8, 64 output tokens/request, three refill waves. C=1/C=4/C=8 use three fresh processes per arm at PR head7e876efa. C=16 is the earlier direct-executor diagnostic; its full-SPEC value has one repeat.SSSASASAAASSAAdaptive nearly matches full speculation at C=1. At C=16, full speculation over-expands the target batch, while the selected 6/16 route verifies 58 compact rows instead of 128 and beats both extremes.
R9700 adaptive speculative-round profile
Strix Halo
Radeon 8060S (
gfx1151, ROCm 7.2.4), built at the same PR head with HIP graphs enabled. Same models, depth, cache settings, fixed capacity 16, prompt cohorts, token count, three refill waves, and three fresh processes per arm.SSSASSSSASSSAThe C=4 preliminary route screen chose
SASS, while all three measured processes choseSSAS; the selected count stayed 3/4. At C=16, adaptive beat paired AR in every repeat (+4.0% to +9.6%) while forced full speculation was 16.9% slower than AR. This shows that speculation can help at high concurrency; forcing every lane speculative is the problem.Strix adaptive speculative-round profile
Pure-AR adaptive rounds are excluded here. Percentages pool measured direct-speculation rounds across repeats.
Across both GPUs, the remaining executor bottleneck is the single target verification pass, followed by batched drafting. Direct state promotion is no longer dominant.
Finite C=16 refill goodput can remain below AR after a finite source is exhausted because speculative requests finish at different times and leave an underfilled terminal drain. That matters for bursts, but is separate from saturated capacity.
Correctness and validation
gfx1201andgfx1151.git diff --checkpassed.The strict C=8/C=16 refill validator observes batch-shape-dependent greedy hash variation across waves, including pure AR cases. Those rows are performance/route diagnostics, not exact-output reproducibility evidence. No benchmark-result artifacts are included in this PR.
Remaining production policy limitation
Activation knowledge and execution scheduling are currently coupled: once an adaptive request selects AR or speculation, that execution mode remains sticky until the request ends.
The Strix C=16 runs also expose path-dependent online calibration. A warmup AR path updates the AR shape EMA while the counterfactual speculative estimate stays at its profile value; a warmup speculative path updates the opposite side. Near a gate boundary, that can move the next fresh cohort between
k=0andk=6, after which sticky request decisions preserve the early choice.The next production-oriented step is to retain request-level eligibility and acceptance knowledge, but choose the execution subset per round from current live concurrency, backlog, pending-prefill pressure, and drain state. Replan on arrivals/departures and material cost changes; use confidence-aware symmetric cost calibration plus hysteresis/minimum dwell to avoid boundary flapping. The one-pass compact executor remains useful underneath that scheduler.