[2026春季][T2-1-2] CearX

[CearX]

Summary

通过 Chunked Prefill、KV Cache 准入水位与请求/SSE 热路径优化，显著提升了单卡 8B 在高并发、长输出场景下的服务吞吐，同时保持小规模性能基本不变。

Motivation

T2-1-2 赛题需求多并发、长文本服务场景下的输出吞吐和总 token 吞吐，同时要求小规模性能无明显下降。

在单张 NVIDIA GeForce RTX 5090 32 GiB、Qwen3-8B BF16、TP=1 环境中，最终实现完成官方 8B 的 30 个测试点，所有请求均成功，输出长度严格命中，errors=0。在基线与最终实现共同有效的 27 个测试点上：

• output throughput 几何平均提升 13.56%；
• total-token throughput 几何平均提升 13.56%；
• 按并发数分组：C1 -0.09%、C4 +5.44%、C16 +19.81%、C64 +87.14%；
• 按输出长度分组：o256 +7.50%、o1024 +5.20%、o4096 +29.50%；
• c64/i32/o4096：361.93 -> 661.13 output tok/s，提升 82.67%；
• c16/i256/o4096：293.68 -> 515.95 output tok/s，提升 75.69%；
• c1/i32/o256：92.34 -> 91.77 output tok/s，变化 -0.62%，无明显小规模回退。

附件赛题报告包含完整 30 点最终矩阵与 27 点基线对比，包括所有负收益点。

Type of Change

• feat — new configurable service and benchmark entry points
• fix — exact LENGTH termination and OpenAI-compatible field handling
• perf — performance improvement
• refactor — code restructuring without behavior change
• test — reproducible official-style benchmark client
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

平台	模型	测试	结果
NVIDIA GeForce RTX 5090 32 GiB	Qwen3-8B BF16，TP=1	官方服务矩阵	30/30 点完成；输出长度精确；errors=0
NVIDIA GeForce RTX 5090 32 GiB	Qwen3-8B BF16，TP=1	基线对比	27 个共同点几何平均 +13.56%

Benchmark / Performance Impact

• 硬件：NVIDIA GeForce RTX 5090 32 GiB。
• 模型与数据类型：Qwen3-8B，BF16。
• 并行配置：TP=1，单卡。
• 服务接口：OpenAI /v1/chat/completions，streaming SSE。
• 负载：burst，request_rate=inf，num_prompts=max_concurrency。
• 生成配置：ignore_eos=true，客户端校验输出长度精确命中。
• 复现 seed：0。
• 矩阵：官方单卡 8B 共 30 个组合。
• 对比方法：只在基线与最终实现均有效的 27 个点上计算几何平均。

场景	基线 output tok/s	最终 output tok/s	变化
c64/i32/o256	1443.47	3092.98	+114.27%
c64/i32/o1024	1005.39	1683.44	+67.44%
c64/i32/o4096	361.93	661.13	+82.67%
c16/i256/o4096	293.68	515.95	+75.69%
c1/i32/o256	92.34	91.77	-0.62%

Notes for Reviewers

• chunked prefill 的中间 prompt chunk 只更新 KV 状态，不向客户端发送采样 token。
• KV 安全水位只延后新请求入场，不降低配置的最大 batch size。
• stream_interval 保证首 token 立即发送，之后只合并 SSE 事件，不丢失模型 token。
• 非流式请求使用 completion event 等待完成，最终文本只 decode 一次。
• LENGTH 终止路径保留最后一个采样 token。

PR 附件：

赛题报告.pdf
HONOR_CODE.md
REFERENCE.md

CI / ChatOps

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from main — the branch is rebased cleanly on top of the audited main.
• No fixup! / squash! / wip commits remain.
• Existing PR/branch/commit that followed the legacy issue format.

Scope and Design

• Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

• The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks (e.g. the `seqlens_k` tensor) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• No raw new/delete; RAII / smart pointers / existing allocators are used.
• Changed files are formatted by scripts/format.py.
• No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• Code is PEP 8 compliant.
• Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• Changed files are formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• For any platform that could not be tested, an explicit reason is given in the table and a reviewer with access has been tagged.
• Passed single request test (examples/test_infer.py), or specify the reason for skipping.
• Passed offline performance test (examples/bench.py), or specify the reason for skipping.
• Passed sanity test (test/bench/test_benchmark.py), or specify the reason for skipping.
• Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py), or specify the reason for using scripts/competition/official_client_bench.py: the competition requires the official burst service matrix and exact output-length validation.

Build, CI, and Tooling

• The project builds cleanly from a fresh directory on at least one affected platform.
• CI has been triggered manually (Actions -> CI on this branch), or /retest was requested.

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• Any user-visible breaking change is called out explicitly under "Motivation" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.