fix(freshopencode): shared-daemon death self-heal, observable discards, and client 409 recovery - #818
Merged
Merged
Conversation
…ape, ApiError tests)
…tted interfaces, self-exit e2e)
…enced attach and refetch
…o end Verification spec for the Task-5 user story: a freshopencode pane whose snapshot GET first answers the typed 409 RESTORE_UNAVAILABLE recovers via one generation-fenced freshAgent.attach and a 200 refetch, with no dismiss-only dead-end. Follows the model-picker routed-fetch/suppression mechanics, so it is cloud-legal by construction (no CLOUD_SKIP_SPECS entry). The pre-fix dead-end was already proven red by Task 5's unit tests (commit 24ef078 red history), so no e2e red was needed.
…l lane) Local-lane e2e proof of the server-side daemon-death self-heal chain (plan Task 7; the 2026-09-20 incident class): a real RustServer + a real fake opencode serve daemon whose death is a scripted fixture SELF-exit (FAKE_OPENCODE_SELF_EXIT_MARKER marker-file poll; the spec never signals any PID). Proves (1) live pane, (2) unrequested daemon self-exit, (3) the OPENCODE_DAEMON_LOST 'Agent error:' banner with exactly one typed edge frame, (4) the backoff-guarded respawn (second managed serve launch, new pid), (5) the revival idle-snapshot push + transcript refetch served by the respawned daemon + dismissible banner + working follow-up turn, and (6) NO freshAgent.turn.complete during the death window (with a positive-control chime on the follow-up turn, so the absence is not vacuous). TDD red history is carried by the Task 3+4 Rust unit tests (commits 900742d, cc9c5f8). Registered in CLOUD_SKIP_SPECS (same provider-lifecycle-timing class as freshopencode-restart-recovery); cloud PR coverage is carried by the cloud-legal freshopencode-snapshot-409-recovery.spec.ts.
ep2-r2 fresheyes Major (cross-generation event contamination): taking the running entry released the generation lock while the old RunningServe — including its active SSE handle and event sink — stayed alive in the local variable until lose_daemon returned. A successor cold-started in that window registered B-era emitters while A's generation-less sink could still dispatch into the same shared map: a buffered/late A-era session.idle satisfied B's await_idle, the unit-level precursor of a false freshAgent.turn.complete clearing busy for a still-running B turn (violating the no-chime-on-daemon-loss contract). The dispatch-era gate: every daemon's sink captures the current era at its connect (inside the cold-start running critical section), every running-entry take — both lose_daemon arms and shutdown — retires the era under that same lock before it releases, and the sink drops any event whose era has been retired. The gate lives at the sink, the one point every dispatched event must pass: dropping the EventStreamHandle only aborts the transport's reader task, so an already-in-flight dispatch can still land past the handle drop — the era check is the airtight fence, transport-agnostic by construction. RED (observed on HEAD): a_lost_daemons_late_events_never_satisfy_the_ successors_await_idle dispatches a late session.idle through daemon A's REAL recorded connect sink — after A's watcher-arm loss, with B installed and B's await_idle in flight for the same durable session — and on HEAD the await resolved Ok. GREEN: the await stays pending and a genuine B-era idle through B's own sink still satisfies it (10/10 deterministic). The prior successor-emitter test dispatched through the generation-less dispatch_event seam; this one carries A's identity through the real path.
ep2-r2 fresheyes Major (the transitional-window dead bridge): the fork-child and resume constructions install the generation-fenced bridge BEFORE the session is published in `sessions` and committed `Live` (materialization remains `Starting`). A daemon loss landing inside that window killed the fresh bridge — the loss sweep claims its just-subscribed sender, the fanned Lost closes its channel — while every recovery trigger missed it: the successor's one-shot `Started` revival pass cannot see an unpublished session and deliberately skips transitional ownership states, and the re-warm's already-running fast path emits no second `Started`. The operation still answered success: a pane with a permanently dead event bridge. The post-commit rescue tail: after the Live commit (the only state the revival pass rescues) and before the success reply/return, both constructions re-run the SAME fenced restart_session_bridge_guarded tail the revival pass uses — a bridge still alive against the current daemon is the quiet no-op (happy path byte-identical: no restart, no push), a bridge stamped to the lost generation (or exited/absent) is restarted against the successor and the client gets exactly one recovery snapshot. The position-independent health re-check covers BOTH window variants — the unpublished leg a revival pass can never see, and the published-Starting leg a pass must skip — because it judges at tail time, not where in the window the loss landed. No adopt-guard: this is the operation finishing its own just-committed registration, the same exposure the bridge install itself carries; the re-lookup discipline (skip a map-removed session) mirrors the revival pass's. A respawn failure warns and still returns success (the session is registered; the next Started or fenced attach retries). RED (observed on HEAD): transitional_fork_child_survives_a_daemon_loss_in_its_starting_window parks the fork between the child's publication and its Live commit, drives the real watcher-arm loss (both bridges drain and exit), lets B's one-shot Started revival rescue the parent while proving it skipped the Starting child (sorted-id visit order), releases the park, and on HEAD the fork answered FreshAgentForked over the dead bridge. GREEN: live bridge + exactly one recovery snapshot + one typed loss edge, no chime anywhere (10/10 deterministic). The send-materialization site needs no tail: it holds the per-session lock through its commit, so any in-window revival pass parks on that lock and rescues post-commit by construction.
ep2-r3 fresheyes Major (the check-to-dispatch TOCTOU): a sink that passed the era check could be preempted before dispatch_event_on acquired the emitter mutex and resume only after its daemon's loss was taken (era retired, emitters swept) and the successor registered a replacement sender — the stale event was then delivered into the successor's registration, and a stale session.idle would falsely satisfy the successor's await_idle (the false turn-complete precursor). The authoritative era verification now rides UNDER the session_emitters mutex, held across check, sender-selection, and send (dispatch_event_on_era): every take retires the era before its sweep claims the same mutex, so a re-verifying callback either sees the retired era (dropped before any entry is selected or created) or holds the lock ahead of the sweep (every selectable entry was registered in its own era). The sink's load-compare remains only as the cheap fast path. The generation-less dispatch_event pub seam stays deliberately ungated (documented test-only lane). RED first: a_preempted_era_checked_event_never_reaches_the_ successors_registration pins the exact interleaving through a cfg(test)-only dispatch park — an A-era event that PASSED the check is held while the loss retires the era and B registers an in-flight await_idle; released, it satisfied the successor's await on HEAD and is dropped after the fix. A genuine B-era idle through B's own sink still satisfies it.
…surrects retired sessions
ep2-r3 fresheyes Major: the post-commit rescue cloned the session Arc
from the map, released the map lock, and took the session lock without
rechecking killed, map membership, or ownership — a kill/handoff
completing in that window (killed set, every map key removed, the
coordinator's stop committed) left the rescue holding a retired
session whose absent bridge it then restarted: a detached fresh-agent
bridge broadcasting for the killed session, an idle-snapshot push over
it, and the operation answering success over the retirement.
The rescue now rides the ESTABLISHED ownership-coordinator-gated
restart discipline (the revival pass's and fenced attach's gates)
around the same shared restart_session_bridge_guarded tail: a fresh
map re-lookup (a miss is the typed refusal, never a silent success),
the canonical ownership observation + arm_adopt_guard window (revive
only Live{FreshAgent}; every transition, terminal owner, and vacant
key refuses; the guard held across the restart), and the killed check
under the session lock. A refusal never installs and never pushes —
the mover's teardown owns the cleanup, and the operation surfaces its
typed ownership-changed error instead: the fork answers its
stale-commit error frame (never FreshAgentForked), the resume returns
its ResumeOpencodeError::Manager ownership-changed refusal. The
under-ticket handoff continuation (the runner's own lifecycle window)
skips only the coordinator observation; the killed check and map
re-lookup remain its gates. A bounded respawn failure still warns and
the operation still succeeds.
RED first, both legs forced deterministically through a cfg(test)-only
rescue stall (the rescue parked between its map-lookup clone and its
session lock): the fork leg answered FreshAgentForked over the killed
child on HEAD; the resume leg (unwired coordinator — the killed check
is the only gate) returned Ok over the killed session. Both refuse
typed after the fix; the ep2-r2 rescue contract (the transitional
child still recovers from a daemon loss in its window, exactly one
snapshot, no chime) is unchanged and green.
…d recovers daemon loss
…d ownership fence
…emon-death-recovery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the 2026-09-20 incident class where the single shared
opencode servedaemon died (or was killed by a stuck request) and every freshopencode pane went dark with no log, no recovery, and a dead-end snapshot 409. Four layers:compactandget_configswitch to captured-baseDiscardOnTimeout::No(serve.rs). Write lanes (prompt_asyncetc.) deliberately keepYesas a wedged-daemon recycler.discard_runningemits a structured WARNfreshagent.opencode.daemon_discardedwith its reason (the incident left zero log trace).freshAgent.error{code:"OPENCODE_DAEMON_LOST"}edge (no chime ever), the daemon respawns with exponential backoff, and bridges are revived level-triggered onStarted. Hardened across the review loop: stale request-timeouts can't kill a replacement daemon (captured-base fence), a lost daemon's emitters/events can never reach the successor era (atomic sweep + era-gated dispatch), transitional/handoff sessions recover at their commit points, and no rescue can resurrect a killed, handed-off, or replaced session (instance + committed-fence verification).RESTORE_UNAVAILABLEfor a fresh-agent owner no longer dead-ends: one generation-fenced attach carrying the 409's ownownerGeneration(monotonic refusal-fold, canonical session keying) + refetch, once per pane identity, with the reveal-dirty lane preserved.Verification
Production deploy is a separate step (requires explicit approval per repo rules).