Repository navigation
feat(sglang): map reasoning_effort to per-level thinking budgets - #12687
pos-ei-don wants to merge 3 commits into
Conversation
The Go side already forwards the request's effort as Metadata["reasoning_effort"], but the sglang backend only read enable_thinking: every level got the same model-level thinking_budget, so low/medium/high produced identical reasoning. - New model option `effort_budgets:low=…;medium=…;high=…` picks the budget of the request's level; unlisted levels and requests without an effort keep `thinking_budget`. Parsing reuses _parse_thinking_budget. - A level listed in effort_budgets turns reasoning on for that request, so levels the Go side does not know cannot silently disable it. - ApplyReasoningEffort now treats `xhigh` and `max` as thinking levels. Before, they left DisableReasoning unset, so with a model default of "off" asking for more reasoning produced none. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
- warn on an effort_budgets entry without a token count (high=) - update comments: enable_thinking=false still wins over a listed level - document that precedence in text-generation.md - cover xhigh/max and the empty-entry warning in the tests Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
…ad warning - ApplyReasoningEffort keeps an operator's reasoning.disable for high, xhigh and max - sglang: _configured_budget() feeds the strict-thinking warning; tested with effort_budgets alone Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
|
@mudler Code-review sign-off at Reviewed against latest master Validation: |
|
Thanks. For the record, since the bot could not run the suites: at
|
Description
The Go side already forwards the request's effort as
Metadata["reasoning_effort"](core/backend/options.go), but the sglang backend only readsenable_thinking. Every level therefore gets the same model-levelthinking_budget, solow,mediumandhighproduce identical reasoning — the level only switches thinking on. Separately,ApplyReasoningEffortonly recognisesminimal|low|medium|high;xhighormaxleaveDisableReasoningunset, so withreasoning_default: offa request asking for more reasoning gets none (HTTP 200, no warning).This PR:
effort_budgets:low=…;medium=…;high=…._build_sampling_paramsuses the budget of the request's level; unlisted levels and requests without an effort keepthinking_budget. Parsing reuses_parse_thinking_budget(invalid entries are skipped with a warning,LoadModelnever fails over it). A level listed ineffort_budgetsalso turns reasoning on for that request. No behaviour change without the option.ApplyReasoningEfforttreatsxhighandmaxas thinking levels.effort_budgetsin the SGLang reasoning section;xhigh/maxin thereasoning_effortlists.Same idea as vLLM's proposed
--reasoning-effort-budgets(vllm-project/vllm#55432 / #55382) and Ollama's think levels (ollama/ollama#17566). Follow-up to #12193.Notes for Reviewers
Measured on a DGX Spark (GB10), Qwen3.8-35B-A3B NVFP4, sglang 0.5.21,
enable_strict_thinking, temperature 0, witheffort_budgets:low=256;medium=1024;high=4096;xhigh=16384;max=16384:Characters of reasoning per level:
Before the change, low/medium/high were byte-identical and
xhigh/maxproduced 0 characters of reasoning. On the false divisibility claim onlymaxstated the correct condition;medium/highproduced a wrong one.Tests: four new unit tests in
backend/python/sglang/test.py(parsing, per-level selection, fallback, map without default) plus one for the thinking toggle, run in the backend's own venv on the Spark: 29 tests, all new ones pass;test_build_prompt_forwards_enable_thinkingfails identically with and without this change on that machine's sglang build. Reverting the selection line makes the new tests fail. The Go spec incore/config/reasoning_effort_test.gofollows the existing cases but was not run locally.Signed commits