fix: stop the broker before the Run handler returns - #386
Open
RiSKeD wants to merge 1 commit into
Open
Conversation
RiSKeD
force-pushed
the
fix/agent-panic-on-module-error
branch
2 times, most recently
from
September 1, 2026 09:14
0dda788 to
5e8722c
Compare
The Run RPC returned while the broker's workers could still be inside a stream send. connect invalidates the response writer once the handler is gone, and a send running past that point does not fail, it panics with "Write called after Handler finished". net/http recovers a panic in the handler itself, but not in a goroutine the handler left behind, so the panic took the whole agent down along with every other run in flight. A module that prints right before it fails hits that window on every failed run: the flash module forwards the flash tool's output and then returns the tool's exit error, so a dpcmd exiting non-zero killed the agent. Give Broker a Stop that cancels the workers and waits for them, and let Run own the broker: it creates it and stops it in a deferred cleanup, next to the auto-lock release and before it, so the device is only handed on once the workers are gone. A state function cannot carry this guarantee - it does not cover a panic unwinding past the FSM, fsm.Run skips the remaining states once ctx is done, and every future early return would have to remember it. A worker stuck in a send past the stop timeout is abandoned, since a handler that never returns is the worse outcome. The workers now recover from a panic and report it on the error channel, so that leftover case fails the run instead of the process, and never looks like a broker that finished cleanly. Signed-off-by: Fabian Wienand <fabian.wienand@blindspot.software>
RiSKeD
force-pushed
the
fix/agent-panic-on-module-error
branch
from
September 1, 2026 12:31
5e8722c to
492b5dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
dutagentdied with a panic whenever a module printed something and then failed:session.Printis a rendezvous: it returns as soon as a worker has taken themessage, not when the message is on the wire. The worker is then still inside
stream.Sendwhile the module already returns its error, the FSM finishes andRunreturns. connect invalidates the response writer once the handler is gone,so that send does not fail - it panics.
net/httprecovers a panic in thehandler itself, but not in a goroutine the handler left behind, so this took the
whole agent down, including every other run in flight.
Restart=alwayshides itas a climbing
NRestartsand a dropped connection for concurrent clients.The flash module hits this window on every failed run: it forwards the flash
tool's output and then returns the tool's exit error, so a
dpcmdexitingnon-zero was enough to kill the agent.
The fix
session.Broker.Stopcancels the workers and waits for them to return,signalled by a new
stoppedchannel closed afterwg.Wait().errChkeepsits existing contract; only
Stopreadsstopped, so the two never steal eachother's values.
Runowns the broker now: it creates it and stops it in a deferred cleanup,registered after the auto-lock release so it runs before it - the device is
handed on only once the workers are gone. Ownership sits where the lifetime
ends, and no state function has to remember anything.
A state function could not carry this guarantee: it does not cover a panic
unwinding past the FSM,
fsm.Runskips the remaining states once ctx is done,and every future early return in the handler would reopen the hole.
that never returns is the worse outcome. For that leftover case the workers now
recover from a panic and report it on
errCh, so the run fails instead of theprocess, and a panicking worker no longer looks like a broker that finished
cleanly.
Tests
internal/dutagent/session:Stopwaits for an in-flight send, gives up on astuck worker at its own timeout, is a no-op when the broker was never started or
is stopped twice, and a worker panic is recovered and surfaced as an error.
cmds/dutagent: the seam with the real broker - a module prints, then fails, andno send is in flight once the broker is stopped. Plus the cancelled-RPC case in
TestWaitModules.Every new test was checked against a deliberately broken version of the code:
Verification
go test -race ./...,golangci-lint run ./...andgo vet ./...are clean.Ran against real hardware (a Raspberry Pi agent driving an Alderlake-P board):
the same failing
flash readthat used to kill the agent now returnsflash operation failed: flash tool exited with code 1to the client, the workerstops first, and
NRestartsstays at 0.Left open, deliberately
session.Printstill returns before its send completes. The window is closedby the deferred stop, but the asymmetry remains.
running after
Runreturned. That is pre-existing and worth its own issue, notthis fix.
Runitself in tests -connect.BidiStreamhas no exportedconstructor - so the deferred stop is wired but only covered indirectly. The
same is true of the pre-existing auto-lock release. An
httptest-based testlike
internal/rpc/version_external_test.gocould close both.