Skip to content

Fix ValidationError on early client disconnect and ZeroDivisionError in draft metrics logging#433

Open
nakaken3013-code wants to merge 2 commits into
theroyallab:mainfrom
nakaken3013-code:fix/streaming-metrics-robustness
Open

Fix ValidationError on early client disconnect and ZeroDivisionError in draft metrics logging#433
nakaken3013-code wants to merge 2 commits into
theroyallab:mainfrom
nakaken3013-code:fix/streaming-metrics-robustness

Conversation

@nakaken3013-code

Copy link
Copy Markdown

Two small, independent robustness fixes for edge cases we hit while load-testing the exllamav3 backend under concurrency. Each is a one-line change; happy to split into two PRs if you'd prefer.


1. ValidationError when a client disconnects during prefill

Problem
When a streaming chat-completion client disconnects before the first token is generated (i.e. the request is still in prefill), the server raises a pydantic ValidationError instead of ending the request cleanly.

In _chat_stream_collector (endpoints/OAI/utils/chat_completion.py), generation is initialized to {} and the "index" key is only assigned inside the async for generation in new_generation: loop. If the disconnect happens during prefill, the generator yields nothing, the loop body never runs, and the empty {} is returned. _compose_response then constructs ChatCompletionRespChoice(index=None), which fails validation (index must be an int):

ValidationError: index
  Input should be a valid integer [type=int_type, input_value=None]

Fix
Initialize generation = {"index": task_idx} so a valid index is always present even when zero tokens are produced. The in-loop assignment is unchanged.

Repro
Start a streaming chat completion with a large prompt, then abort the client (close the tab / Ctrl-C curl / Open WebUI stop button) while the server is still prefilling, before any token streams back. Before: traceback above. After: the request ends cleanly.


2. ZeroDivisionError in draft accept-rate logging

Problem
log_metrics (common/gen_logging.py) computes accept_rate = accept / total_draft with total_draft = draft_accept + draft_reject. When both are 0 (a request that produced no draft tokens while speculative / MTP decoding is enabled), total_draft == 0 and the division raises ZeroDivisionError, aborting the metrics log line.

Fix

accept_rate = accept / total_draft if total_draft > 0 else 0.0

Reports 0.0% in that case instead of crashing.


Both were observed with the exllamav3 backend, TP + MTP, under ~8-way concurrent load with clients that cancel mid-request.

_chat_stream_collector initialized `generation = {}` and only set the
"index" key inside the `async for` loop body. If the client disconnects
before the first token is produced (still in prefill), the loop never
runs, the empty dict is returned, and _compose_response constructs
ChatCompletionRespChoice(index=None), raising a pydantic ValidationError.
Initialize `generation = {"index": task_idx}` so a valid index is always
present even when no tokens were generated.
When draft_accept and draft_reject are both 0 (a request that produced
no draft tokens), total_draft is 0 and `accept / total_draft` raises
ZeroDivisionError inside log_metrics. Guard the division and report
0.0% in that case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant