Fix ValidationError on early client disconnect and ZeroDivisionError in draft metrics logging#433
Open
nakaken3013-code wants to merge 2 commits into
Conversation
_chat_stream_collector initialized `generation = {}` and only set the
"index" key inside the `async for` loop body. If the client disconnects
before the first token is produced (still in prefill), the loop never
runs, the empty dict is returned, and _compose_response constructs
ChatCompletionRespChoice(index=None), raising a pydantic ValidationError.
Initialize `generation = {"index": task_idx}` so a valid index is always
present even when no tokens were generated.
When draft_accept and draft_reject are both 0 (a request that produced no draft tokens), total_draft is 0 and `accept / total_draft` raises ZeroDivisionError inside log_metrics. Guard the division and report 0.0% in that case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two small, independent robustness fixes for edge cases we hit while load-testing the exllamav3 backend under concurrency. Each is a one-line change; happy to split into two PRs if you'd prefer.
1.
ValidationErrorwhen a client disconnects during prefillProblem
When a streaming chat-completion client disconnects before the first token is generated (i.e. the request is still in prefill), the server raises a pydantic
ValidationErrorinstead of ending the request cleanly.In
_chat_stream_collector(endpoints/OAI/utils/chat_completion.py),generationis initialized to{}and the"index"key is only assigned inside theasync for generation in new_generation:loop. If the disconnect happens during prefill, the generator yields nothing, the loop body never runs, and the empty{}is returned._compose_responsethen constructsChatCompletionRespChoice(index=None), which fails validation (indexmust be an int):Fix
Initialize
generation = {"index": task_idx}so a validindexis always present even when zero tokens are produced. The in-loop assignment is unchanged.Repro
Start a streaming chat completion with a large prompt, then abort the client (close the tab / Ctrl-C curl / Open WebUI stop button) while the server is still prefilling, before any token streams back. Before: traceback above. After: the request ends cleanly.
2.
ZeroDivisionErrorin draft accept-rate loggingProblem
log_metrics(common/gen_logging.py) computesaccept_rate = accept / total_draftwithtotal_draft = draft_accept + draft_reject. When both are0(a request that produced no draft tokens while speculative / MTP decoding is enabled),total_draft == 0and the division raisesZeroDivisionError, aborting the metrics log line.Fix
Reports
0.0%in that case instead of crashing.Both were observed with the exllamav3 backend, TP + MTP, under ~8-way concurrent load with clients that cancel mid-request.