Reconnect game stream when it drops mid-game

[thilo11]

Type of pull request:

• Bug fix
• Feature
• Other

Description:

When the per-game HTTP stream (/api/bot/game/stream/{id}) ends with StopIteration but Lichess still lists the game as ongoing, re-raise so the existing @backoff.on_exception wrapper retries play_game (new stream, new engine) instead of treating it as a local game over.

Previously:

stopped = isinstance(e, StopIteration)
stay_in_game = not stopped and (move_attempted or game_is_active(li, game.id))

So any stream EOF exited the game, even with time on the clock. A dropped mid-game stream then looks like a clean “Game over” locally while Lichess keeps the game alive until the bot flags.

We hit this in production: bot played a move with ~30s remaining, the game stream died, no further search was started, and the game was lost on time.

Retrying play_game is enough: the new stream’s initial full game info is injected with prior_game is None, so a search starts immediately if it is our turn.

Related Issues:

Checklist:

• I have read and followed the contribution guidelines.
• I have added necessary documentation (if applicable).
• The changes pass all existing tests.

Screenshots/logs (if applicable):

[MarkZH]

Some initial suggestions:

• Put as much of the new code into named functions so it's easier to see the flow of the main game loop. This will also help with the "too many branches" and "too complex" errors from ruff.
• Instead of time.monotonic(), we have a Timer class that's built for this purpose: game_state_timer = Timer(seconds(10)) to start, game_state_timer.reset() to reset to 10 seconds, and game_state_timer.is_expired() to check if the timer has run out.
• In line 830, use the cast re_lines: Iterator[bytes] to fix the mypy error.

This is a very complex change, but it looks plausible.

[thilo11]

Thanks for the review — addressed in e09b1f2:

• Extracted reconnect / silence handling into named helpers (recover_from_game_stream_error, note_game_stream_silence, close_extra_game_stream) so play_game stays readable and within ruff complexity/branch limits
• Switched the 10s silence warning to Timer(seconds(10)) with reset() / is_expired()
• Annotated reopened streams as re_lines: Iterator[bytes] (and typed game_stream: Iterator[bytes]) for mypy
• After reconnect, re-inject the current gameState into the stream so a search runs immediately when it is our turn

Locally: ruff check, mypy --strict, and pytest (53 passed, 1 skipped) are green.

[AttackingOrDefending]

What do you think about doing

if stopped and game_is_active(li, game.id):
    raise

which would simplify this PR? I think it should accomplish everything this does apart from reusing the same engine instance, but I think it is probably better, as it keeps the code simpler and the rest of the code doesn't use same engine reconnect.

[thilo11]

Agreed — that's a better fit for this codebase.

play_game is already wrapped in @backoff.on_exception(backoff.expo, BaseException, max_time=600), so re-raising StopIteration retries the whole function: new game stream, new engine, and prior_game is None so a search starts immediately if it is our turn. That covers the production failure (stream EOF after a move, clock still running) without a second stream state machine.

Pushed in b870003. The only extra is a warning log so this path is visible at the default log level (backoff_handler is debug).

What we give up vs the previous approach, which I think is the right tradeoff here:

• The engine process is torn down and restarted (hash/NNUE not reused), matching the rest of the code
• Retries go through the existing exponential backoff instead of reconnecting the iterator in place
• Connection errors still follow the existing stay_in_game path until the iterator EOFs, at which point this raise kicks in