fix(opd): score teacher logprobs at rollout temperature, not 0 - #2085
Conversation
945adff to
6c995f4
Compare
|
@zhuzilin could you review this one? OPD teacher scoring was using temperature=0 while rollout samples use --rollout-temperature; this makes the teacher logprobs match the actual sampled distribution instead of silently scoring a greedy distribution. |
382ef68 to
ca2ea7f
Compare
|
@zhuzilin refreshed on latest main and checks are green. This scores OPD teacher logprobs at rollout temperature instead of forcing temperature 0, which keeps teacher scoring aligned with generation. Could you review when you have bandwidth? |
f1dec0f to
a15928c
Compare
max_new_tokens=0 still only scores the existing sequence; temperature 0 was forcing a greedy distribution that does not match rollout. Co-authored-by: Cursor <cursoragent@cursor.com>
a15928c to
ca2aa80
Compare
|
@zhuzilin @EazyReal 这个pr是一个no-op操作,并且这个会引起混淆?默认teacher只需要用到 我新开了一个pr,#2319 ,辛苦review一下。非常感谢! 另外,sgl 的 SGLANG_RETURN_ORIGINAL_LOGPROB和temperature只影响output的logp,input的logp不受影响(opd在这,max_new_tokens=0) python -m sglang.launch_server \
--model /root/Qwen3-0.6B \
--reasoning-parser qwen3 \
--tool-call-parser qwen25temperature=1.0的payload curl -s http://127.0.0.1:30010/generate -H 'Content-Type: application/json' -d '{
"input_ids": [3838,374,279,6722,315,9625,30,576,6722,315,9625,374,12095],
"sampling_params": {"temperature": 1, "max_new_tokens": 0},
"return_logprob": true,
"logprob_start_len": 0
}' |
|
Thanks @kkyyxhll for catching the SGLang side — you are right that |



Problem
slime/rollout/on_policy_distillation.pyscores the teacher withmax_new_tokens: 0(logprobs of the tokens already in the sample, no generation). The payload hardcoded"temperature": 0.OPD's KL is student (rollout sampling) vs teacher. Teacher logprobs at temperature 0 are a greedy distribution, so the penalty does not match the rollout policy whenever
--rollout-temperatureis not 0.Change
max_new_tokens: 0is unchanged. Unauthenticated / temp-0 rollouts still send 0.Test plan
--rollout-temperature≠ 0: teacher/generatepayload should carry that temperature (not 0)