Skip to content

fix(rollout): make dynamic refills granular - #2218

Open
EazyReal wants to merge 1 commit into
THUDM:mainfrom
EazyReal:fix/granular-rollout-refills
Open

fix(rollout): make dynamic refills granular#2218
EazyReal wants to merge 1 commit into
THUDM:mainfrom
EazyReal:fix/granular-rollout-refills

Conversation

@EazyReal

Copy link
Copy Markdown
Contributor

Problem

Dynamic sampling currently requires over_sampling_batch_size to be at least rollout_batch_size, even though the rollout state machine supports any positive refill granularity. A single rejected prompt group can therefore launch a full replacement wave.

At rollout completion, non-partial generation tasks are also left running while SGLang is drained. Multi-turn custom generation can issue new requests during that drain, delaying or preventing it from reaching idle.

Change

  • accept any positive over_sampling_batch_size
  • cancel and await non-partial generation tasks before draining SGLang
  • preserve partial-rollout prefixes through server drain
  • make the complete abort lifecycle exception-safe while re-raising the original failure
  • align English and Chinese dynamic-sampling documentation
  • add CPU regressions for validation, cancellation ordering, partial preservation, and failure cleanup

The existing scheduler remains unchanged: rejected groups decrement accepted-plus-pending coverage, and the configured refill granularity restores it.

Validation

  • python tests/test_sglang_rollout_abort.py -q — 4 passed
  • focused Python 3.10 suite — 33 passed
  • python .github/workflows/generate_github_workflows.py
  • pre-commit run --all-files
  • git diff --check

No live GPU or distributed rollout was run; the affected lifecycle boundaries are covered with CPU and mocked regressions.

Allow any positive over_sampling_batch_size, and cancel unfinished
non-partial generation tasks before draining SGLang so one rejected
prompt group cannot launch a full replacement wave or keep the drain
loop alive. Abort failures cancel remaining groups before re-raising.

Co-authored-by: Cursor <cursoragent@cursor.com>
@EazyReal
EazyReal force-pushed the fix/granular-rollout-refills branch from b106461 to 1d1a758 Compare August 19, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant