Skip to content

fix: stop the broker before the Run handler returns - #386

Open
RiSKeD wants to merge 1 commit into
mainfrom
fix/agent-panic-on-module-error
Open

fix: stop the broker before the Run handler returns#386
RiSKeD wants to merge 1 commit into
mainfrom
fix/agent-panic-on-module-error

Conversation

@RiSKeD

@RiSKeD RiSKeD commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

The bug

dutagent died with a panic whenever a module printed something and then failed:

panic: Write called after Handler finished
	panic: Header called after Handler finished
connectrpc.com/connect.flushResponseWriter → net/http.(*http2responseWriter).Flush
	.../internal/dutagent/session.toClientWorker → worker.go:41

session.Print is a rendezvous: it returns as soon as a worker has taken the
message, not when the message is on the wire. The worker is then still inside
stream.Send while the module already returns its error, the FSM finishes and
Run returns. connect invalidates the response writer once the handler is gone,
so that send does not fail - it panics. net/http recovers a panic in the
handler itself, but not in a goroutine the handler left behind, so this took the
whole agent down, including every other run in flight. Restart=always hides it
as a climbing NRestarts and a dropped connection for concurrent clients.

The flash module hits this window on every failed run: it forwards the flash
tool's output and then returns the tool's exit error, so a dpcmd exiting
non-zero was enough to kill the agent.

The fix

  • session.Broker.Stop cancels the workers and waits for them to return,
    signalled by a new stopped channel closed after wg.Wait(). errCh keeps
    its existing contract; only Stop reads stopped, so the two never steal each
    other's values.
  • Run owns the broker now: it creates it and stops it in a deferred cleanup,
    registered after the auto-lock release so it runs before it - the device is
    handed on only once the workers are gone. Ownership sits where the lifetime
    ends, and no state function has to remember anything.
    A state function could not carry this guarantee: it does not cover a panic
    unwinding past the FSM, fsm.Run skips the remaining states once ctx is done,
    and every future early return in the handler would reopen the hole.
  • A worker stuck in a send past the stop timeout (5 s) is abandoned - a handler
    that never returns is the worse outcome. For that leftover case the workers now
    recover from a panic and report it on errCh, so the run fails instead of the
    process, and a panicking worker no longer looks like a broker that finished
    cleanly.

Tests

internal/dutagent/session: Stop waits for an in-flight send, gives up on a
stuck worker at its own timeout, is a no-op when the broker was never started or
is stopped twice, and a worker panic is recovered and surfaced as an error.

cmds/dutagent: the seam with the real broker - a module prints, then fails, and
no send is in flight once the broker is stopped. Plus the cancelled-RPC case in
TestWaitModules.

Every new test was checked against a deliberately broken version of the code:

Stop as a no-op:        Stop returned with 1 send(s) in flight
                        Stop returned after 80ns, it did not wait out its 100ms timeout
panic not reported:     expected the panic to be reported once, got []
ctx.Done branch gutted: expected error code canceled, got nil
recover removed:        the test binary dies with the original panic

Verification

go test -race ./..., golangci-lint run ./... and go vet ./... are clean.

Ran against real hardware (a Raspberry Pi agent driving an Alderlake-P board):
the same failing flash read that used to kill the agent now returns
flash operation failed: flash tool exited with code 1 to the client, the worker
stops first, and NRestarts stays at 0.

Left open, deliberately

  • session.Print still returns before its send completes. The window is closed
    by the deferred stop, but the asymmetry remains.
  • The module goroutine runs on the RPC context, so on a broker failure it keeps
    running after Run returned. That is pre-existing and worth its own issue, not
    this fix.
  • Nothing drives Run itself in tests - connect.BidiStream has no exported
    constructor - so the deferred stop is wired but only covered indirectly. The
    same is true of the pre-existing auto-lock release. An httptest-based test
    like internal/rpc/version_external_test.go could close both.

@RiSKeD
RiSKeD force-pushed the fix/agent-panic-on-module-error branch 2 times, most recently from 0dda788 to 5e8722c Compare September 1, 2026 09:14
The Run RPC returned while the broker's workers could still be inside a
stream send. connect invalidates the response writer once the handler is
gone, and a send running past that point does not fail, it panics with
"Write called after Handler finished". net/http recovers a panic in the
handler itself, but not in a goroutine the handler left behind, so the
panic took the whole agent down along with every other run in flight.

A module that prints right before it fails hits that window on every
failed run: the flash module forwards the flash tool's output and then
returns the tool's exit error, so a dpcmd exiting non-zero killed the
agent.

Give Broker a Stop that cancels the workers and waits for them, and let
Run own the broker: it creates it and stops it in a deferred cleanup,
next to the auto-lock release and before it, so the device is only handed
on once the workers are gone. A state function cannot carry this
guarantee - it does not cover a panic unwinding past the FSM, fsm.Run
skips the remaining states once ctx is done, and every future early
return would have to remember it.

A worker stuck in a send past the stop timeout is abandoned, since a
handler that never returns is the worse outcome. The workers now recover
from a panic and report it on the error channel, so that leftover case
fails the run instead of the process, and never looks like a broker that
finished cleanly.

Signed-off-by: Fabian Wienand <fabian.wienand@blindspot.software>
@RiSKeD
RiSKeD force-pushed the fix/agent-panic-on-module-error branch from 5e8722c to 492b5dd Compare September 1, 2026 12:31
@RiSKeD RiSKeD changed the title fix: stop broker workers before abandoning a run fix: stop the broker before the Run handler returns Sep 1, 2026
@RiSKeD
RiSKeD requested a review from jenstopp September 1, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant