Skip to content

feat(sglang): map reasoning_effort to per-level thinking budgets - #12687

Open
pos-ei-don wants to merge 3 commits into
mudler:masterfrom
pos-ei-don:sglang-effort-budgets
Open

pos-ei-don wants to merge 3 commits into
mudler:masterfrom
pos-ei-don:sglang-effort-budgets

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Description

The Go side already forwards the request's effort as Metadata["reasoning_effort"] (core/backend/options.go), but the sglang backend only reads enable_thinking. Every level therefore gets the same model-level thinking_budget, so low, medium and high produce identical reasoning — the level only switches thinking on. Separately, ApplyReasoningEffort only recognises minimal|low|medium|high; xhigh or max leave DisableReasoning unset, so with reasoning_default: off a request asking for more reasoning gets none (HTTP 200, no warning).

This PR:

  • sglang backend: new model option effort_budgets:low=…;medium=…;high=…. _build_sampling_params uses the budget of the request's level; unlisted levels and requests without an effort keep thinking_budget. Parsing reuses _parse_thinking_budget (invalid entries are skipped with a warning, LoadModel never fails over it). A level listed in effort_budgets also turns reasoning on for that request. No behaviour change without the option.
  • Go: ApplyReasoningEffort treats xhigh and max as thinking levels.
  • Docs: effort_budgets in the SGLang reasoning section; xhigh/max in the reasoning_effort lists.

Same idea as vLLM's proposed --reasoning-effort-budgets (vllm-project/vllm#55432 / #55382) and Ollama's think levels (ollama/ollama#17566). Follow-up to #12193.

Notes for Reviewers

Measured on a DGX Spark (GB10), Qwen3.8-35B-A3B NVFP4, sglang 0.5.21, enable_strict_thinking, temperature 0, with effort_budgets:low=256;medium=1024;high=4096;xhigh=16384;max=16384:

Characters of reasoning per level:

prompt before (every level) low medium high max
Pell equation x²−7y²=1, x<10000 629 629 2,036 2,882 2,882
3×10 domino tiling — — 4,013 10,211 37,981
divisibility proof (false as stated) — — 1,939 10,055 25,168

Before the change, low/medium/high were byte-identical and xhigh/max produced 0 characters of reasoning. On the false divisibility claim only max stated the correct condition; medium/high produced a wrong one.

Tests: four new unit tests in backend/python/sglang/test.py (parsing, per-level selection, fallback, map without default) plus one for the thinking toggle, run in the backend's own venv on the Spark: 29 tests, all new ones pass; test_build_prompt_forwards_enable_thinking fails identically with and without this change on that machine's sglang build. Reverting the selection line makes the new tests fail. The Go spec in core/config/reasoning_effort_test.go follows the existing cases but was not run locally.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

The Go side already forwards the request's effort as
Metadata["reasoning_effort"], but the sglang backend only read
enable_thinking: every level got the same model-level thinking_budget,
so low/medium/high produced identical reasoning.

- New model option `effort_budgets:low=…;medium=…;high=…` picks the
  budget of the request's level; unlisted levels and requests without an
  effort keep `thinking_budget`. Parsing reuses _parse_thinking_budget.
- A level listed in effort_budgets turns reasoning on for that request,
  so levels the Go side does not know cannot silently disable it.
- ApplyReasoningEffort now treats `xhigh` and `max` as thinking levels.
  Before, they left DisableReasoning unset, so with a model default of
  "off" asking for more reasoning produced none.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
- warn on an effort_budgets entry without a token count (high=)
- update comments: enable_thinking=false still wins over a listed level
- document that precedence in text-generation.md
- cover xhigh/max and the empty-entry warning in the tests

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
…ad warning

- ApplyReasoningEffort keeps an operator's reasoning.disable for high, xhigh and max
- sglang: _configured_budget() feeds the strict-thinking warning; tested with
  effort_budgets alone

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

@mudler Code-review sign-off at 0bd723a65285a710c1355fad8776f8c43c0b8b26: no blocking source findings; good to merge once CI validates it.

Reviewed against latest master 91c3bf084bd4952cc51ce6e114ba71dea604ade2. Per-level budget selection preserves the model fallback, explicit thinking-disable metadata takes precedence, and xhigh/max retain the operator-disable safeguard. The change includes documentation and focused regression cases.

Validation: git diff --check passes. I attempted go test ./core/config -run Test -ginkgo.focus='ModelConfig.ApplyReasoningEffort' -count=1 -timeout=30s, but compilation fails on stale generated protobuf bindings (pb.Animate3DRequest, pb.MotionRequest, and related symbols); the same failure occurs on unchanged master. The tests did not execute. Python is unavailable in this environment, so I could not run the SGLang suite. Only DCO is currently reported as passing; this sign-off does not establish runtime or CI readiness.

@pos-ei-don

Copy link
Copy Markdown
Contributor Author

Thanks. For the record, since the bot could not run the suites: at 45a960c (adds two tests on top of the reviewed 0bd723a):

  • Go: lint and tests-linux pass in a fork run of this branch.
  • Python (backend/python/sglang/test.py) on a DGX Spark (GB10, sglang 0.5.21): 30/31 pass; the one failure, test_build_prompt_forwards_enable_thinking (SimpleNamespace lacks Images), fails identically on unchanged master.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants