Skip to content

fix: treat temperature=0 as greedy selection, untempered logprobs - #2197

Closed
EazyReal wants to merge 3 commits into
THUDM:mainfrom
EazyReal:fix/rollout-temp-zero-guard
Closed

fix: treat temperature=0 as greedy selection, untempered logprobs#2197
EazyReal wants to merge 3 commits into
THUDM:mainfrom
EazyReal:fix/rollout-temp-zero-guard

Conversation

@EazyReal

@EazyReal EazyReal commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Problem

get_responses and get_log_probs_and_entropy divide logits by rollout_temperature whenever it is not 1.0. Greedy rollout uses temperature=0, so that is a divide-by-zero.

SGLang treats T=0 as greedy token selection, but still returns finite untempered model log-probs. slime stores those as rollout_log_probs and builds PPO ratios from exp(log π_θ − log π_old). Forcing greedy log-probs to 0 would make an on-policy sample start at r = π_θ(a).

Value heads already pass apply_temperature=False.

Change

apply_rollout_temperature is the shared rule at both logit sites:

  • T in {0, 1}: leave logits unchanged
  • otherwise: logits / T

T=0 also skips top-p nucleus replay. SGLang remaps T=0 to top_k=1 and reports unmasked untempered log-probs.

Test plan

  • tests/test_rollout_temperature.py — recomputed logprobs match log_softmax at T=0/1 and log_softmax(logits/T) at T=0.5; get_responses does not scale at 0; greedy skips top-p replay
  • tests/test_value_temperature.py — value head still ignores rollout temperature

@EazyReal
EazyReal force-pushed the fix/rollout-temp-zero-guard branch from f28ff8a to 9f9bb9f Compare August 18, 2026 23:37
Keep the existing != 1.0 guards; dividing by 0 produced infs. Value
heads already pass apply_temperature=False.
@EazyReal
EazyReal force-pushed the fix/rollout-temp-zero-guard branch from 9f9bb9f to 8751e4d Compare August 21, 2026 00:50
SGLang greedy decoding still reports finite model log-probs, not a
delta at the argmax. Dividing logits by T=0 produced infs; using
literal log q=0 would also break on-policy PPO ratios. Share one
helper for that convention, skip top-p nucleus replay under greedy,
and test that recomputed log-probs match untempered log_softmax.

Co-authored-by: Cursor <cursoragent@cursor.com>
@EazyReal EazyReal changed the title fix: guard zero rollout temperature logprob scaling fix: treat temperature=0 as greedy selection, untempered logprobs Aug 21, 2026
Keep the helper and the logprob/top-p invariants; remove extra
validation, comments, and the cloned GPU e2e.

Co-authored-by: Cursor <cursoragent@cursor.com>
@EazyReal

Copy link
Copy Markdown
Contributor Author

reviewer @zhuzilin commented offline that rollout_temperature must be > 0 in RL to ensure meaningful exploration, so we do not need to consider this case.
opened and merged #2297 to ensure that.

@EazyReal EazyReal closed this Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant