perf(qwen36): add SpecLA factor-capture speculative decoding by Graffioh · Pull Request #606 · Luce-Org/lucebox

Graffioh · 2026-08-13T07:51:32Z

Current gain over `main`

Qwen3.6-27B Q4_K_M target, Q8_0 DFlash draft, DDTree budget 22, top-k 4, tau=6, ten HumanEval-style prompts, and 128 generated tokens:

Configuration	Mean decode throughput
Current `main`	25.14 tok/s
SpecLA	27.19 tok/s

SpecLA gain over current main: +8.2%.

Paper: https://arxiv.org/abs/2607.16673

CLI

The normal entry point is one flag:

dflash_server TARGET.gguf --draft DRAFT.gguf --specla

--specla enables speculative linear-attention verification when the loaded target and placement support it. SpecLA is the verification/state-management mode; it is not an alias for a particular proposal algorithm.

For the Qwen3.5/Qwen3.6 dense path implemented in this PR, the runtime currently:

selects DDTree as the compatible proposal adapter;
enables fast rollback;
uses DDTree budget 22;
uses confidence-pruning margin tau=6;
uses draft-tree top-k 4.

This separation is intentional. A future Qwen3.8 DSpark implementation can provide candidates and parent topology behind the same --specla flag without changing the user-facing CLI. That DSpark adapter and its target-specific recurrence/factor capture are not part of this PR.

Supported and permitted combinations

The path in this PR is supported for a monolithic Qwen3.5/Qwen3.6 dense target on one local device. It has been built and parity-tested on HIP.

The current Qwen adapter requires:

--draft DRAFT.gguf;
fast rollback, which --specla enables automatically;
a non-split, single-device target placement.

Setup-dependent overrides remain available:

Option	Constraint	Default with `--specla`	Purpose
`--ddtree-budget N`	positive integer	22	Maximum DDTree proposal nodes
`--ddtree-tau T`	finite, non-negative number	6	Cumulative log-probability pruning margin
`--specla-top-k K`	positive integer	4	Draft-tree expansion width
`--draft-swa N`	positive window when needed	checkpoint metadata	Override draft sliding-window metadata for older checkpoints

The environment variables DFLASH_SPECLA=1 and DFLASH_SPECLA_TOPK=K remain compatibility controls for non-CLI harnesses. DFLASH_SPECLA_CONDITIONAL_DRAFT=1 and DFLASH_SPECLA_FUSED_COMMIT=0 are advanced experiment/debug controls, not normal launch requirements.

Rejected or unavailable combinations

--specla --no-fast-rollback is rejected because SpecLA requires the rollback/state path.
--specla-top-k without --specla is rejected.
The test_dflash harness rejects --specla --seq-verify; sequential verification cannot exercise the SpecLA tree path.
The current Qwen adapter rejects a missing --draft.
Multi-device or layer-split SpecLA is not implemented by this PR.
--kvflash <tokens|auto> cannot use SpecLA factor migration. The server warns, reports SpecLA off, and falls back to ordinary DDTree verification.
Paged-attention autoregressive mode cannot be combined with the current draft/DDTree speculative path.
On an architecture or placement without a SpecLA adapter, the server prints a warning and uses that architecture’s normal decode path. An independently requested --ddtree mode remains independent.

Paper mapping

The paper requires scored speculative candidates and parent topology, not DDTree specifically. A chain is a degenerate tree. DDTree is simply the proposal adapter connected for Qwen3.6 today; DSpark can be connected later if it supplies the required candidate/topology contract and the target has a compatible SpecLA recurrence and factor-capture implementation.

Graffioh · 2026-08-14T13:13:26Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-08-14T13:13:30Z

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 33 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/qwen35/qwen35_dflash_target.cpp">

<violation number="1" location="server/src/qwen35/qwen35_dflash_target.cpp:609">
P2: The SpecLA bank-rotation / `specla_compact_fused` / `specla_flush_raw_fused` logic is copied nearly verbatim between this file and `server/test/test_dflash.cpp`. `rollback_to_tree` (walked_sibling compaction + `current_bank`/`old_pending_bank` rotation + `pending_count` update), `finish_speculative_state` (bank selection via `specla_pending_bank` + `specla_flush_raw_fused`), and `rollback_to_specla` (bank toggle + count) each have a hand-maintained mirror in the bench harness. These are the two most state-sensitive routines in the PR; any divergence between the production target and the harness (e.g. the `ne` bounds, channel mapping, or bank-toggle logic) silently changes acceptance behavior. Pull the shared commit/flush helpers into a common function (e.g. in `specla_commit_cuda.h/.cu`) and have both call sites invoke it.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Graffioh · 2026-08-17T06:07:08Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-08-17T06:07:13Z

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 36 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 36 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 37 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Resolves the conflicts with SpecLA (#606) and the packed concurrent prefill work (#595): - SSM_CONV op_params[0]: 1 stays the SpecLA heavy-light conv, the dflash fused conv step now uses 2; the CUDA dispatcher and supports_op handle both. - GDN raw-gate mode no longer uses src[8]/op_params[2] (compact-decode slot ids and the SpecLA marker): the loader builds one f32 [dt_bias | A] tensor per DeltaNet layer (TargetLayer::ssm_gate_ba, own small buffer) and ggml_gated_delta_net_set_raw_gates() attaches it as src[9] with op_params[10] = 1. - gated_delta_net.cu launchers carry both the active-slot arguments and the raw-gate parameters. - build_delta_net_block keeps main's token-axis segment structure; the stacked (z|qkv) and (beta|alpha) projections, the fused conv step, raw gates, single l2_norm over q|k and the head-broadcast shortcut apply on the plain single-sequence chain path only (ragged, compact-decode, SpecLA and chunked paths take main's materialized ops). - MMQ tiles: Q4_K keeps main's 128x64 (LUCEBOX_RDNA_MMQ_Y); IQ4_XS, Q5_K, Q6_K, Q8_0 and the ROCmFPX formats keep the 64x64 small tile. - Spec loop: DSpark Markov top-k stays available inside the non-conditional DDTree branch; SpecLA's conditional-draft path and draft-KV flag are preserved. Verified on lucebox8 (R9700): AR 34.6 tok/s, DFlash2 110/54/115 (code/prose/mixed), DSpark 48/32/40.

Graffioh force-pushed the codex/qwen35-specla-current-main branch from 27417ca to 5f38668 Compare August 13, 2026 10:55

cubic-dev-ai Bot reviewed Aug 14, 2026

View reviewed changes

Graffioh force-pushed the codex/qwen35-specla-current-main branch 2 times, most recently from e7dc8f0 to c775f9e Compare August 15, 2026 07:16

cubic-dev-ai Bot reviewed Aug 17, 2026

View reviewed changes

Graffioh force-pushed the codex/qwen35-specla-current-main branch from c775f9e to 7ec0e54 Compare August 17, 2026 06:45

Graffioh changed the title ~~perf(qwen35): add SpecLA factor-capture speculative decoding~~ perf(qwen36): add SpecLA factor-capture speculative decoding Aug 17, 2026

Graffioh force-pushed the codex/qwen35-specla-current-main branch from 7ec0e54 to 8d9c77a Compare August 17, 2026 18:57

Graffioh added 2 commits August 17, 2026 18:58

feat(specla): implement state-resident tree verification

259d788

test(specla): add parity coverage and benchmark controls

ebfa900

Graffioh force-pushed the codex/qwen35-specla-current-main branch from 8d9c77a to 3db9829 Compare August 17, 2026 19:08

Graffioh marked this pull request as ready for review August 18, 2026 06:51

cubic-dev-ai Bot reviewed Aug 18, 2026

View reviewed changes

Graffioh force-pushed the codex/qwen35-specla-current-main branch from 3db9829 to c8889ab Compare August 18, 2026 08:09

Graffioh marked this pull request as draft August 18, 2026 08:12

Graffioh force-pushed the codex/qwen35-specla-current-main branch from c8889ab to 5da798e Compare August 18, 2026 08:29

Graffioh marked this pull request as ready for review August 18, 2026 09:21

cubic-dev-ai Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread server/src/server/server_main.cpp

Comment thread server/test/test_dflash.cpp

Graffioh force-pushed the codex/qwen35-specla-current-main branch from 5da798e to 7824c6b Compare August 18, 2026 09:38

Graffioh added 3 commits August 18, 2026 09:40

feat(specla): add one-flag runtime mode

234a1c3

test(specla): harden CLI and bounds coverage

aa2dcc4

docs(specla): document paper mapping and qwen benchmarks

2782286

Graffioh force-pushed the codex/qwen35-specla-current-main branch from 7824c6b to 2782286 Compare August 18, 2026 09:41

davide221 merged commit ac22a3e into Luce-Org:main Aug 19, 2026
9 checks passed

davide221 mentioned this pull request Aug 20, 2026

qwen35: DSpark speculative decoding (Qwen3.8-27B drafters) #625

Draft

Uh oh!

Conversation

Graffioh commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current gain over main

CLI

Supported and permitted combinations

Rejected or unavailable combinations

Paper mapping

Uh oh!

Graffioh commented Aug 14, 2026

Uh oh!

cubic-dev-ai Bot commented Aug 14, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Graffioh commented Aug 17, 2026

Uh oh!

cubic-dev-ai Bot commented Aug 17, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Graffioh commented Aug 13, 2026 •

edited

Loading

Current gain over `main`

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading