diff --git a/AGENTS.md b/AGENTS.md index 7df547d65..60f47f54e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -265,7 +265,7 @@ stderr note). Provisioning, rotation, and revocation live in **Rename scope contract:** pane/tab labels are layout-local; only an explicit session rename writes a durable session title (terminal renames are terminal-scoped). The four name scopes and the reset-to-provider-title flow live in [docs/development/rename-scope-contract.md](docs/development/rename-scope-contract.md). -**Agent Status Indicators:** Blue/busy status is derived from provider activity slices through `resolvePaneActivity`; green/needs-attention and the idle sound flow through `recordTurnComplete` and `useTurnCompletionNotifications`. Turn-complete (green/sound) is server-authoritative everywhere: terminal CLIs via `terminal.turn.complete`, and fresh-agent panes (freshclaude/kilroy/freshcodex/freshopencode) via a discrete `freshAgent.turn.complete` edge emitted only on a positive completion — freshclaude/kilroy on the SDK `result` with `subtype === 'success'`, freshopencode on the success-only `emitStatus(idle)` path, and freshcodex on `turn/completed` only when `params.turn.status === 'completed'` (the notification also fires on interrupt). The client folds it in via `applyFreshAgentCompletion` using the `at`-monotonic dedupe regime (wall-clock `at`, no per-session counter, so a resumed durable session can't swallow completions across a server restart). The waiting-for-approval edge is ALSO server-authoritative: the Claude/kilroy `SdkBridge` emits a discrete `freshAgent.turn.waiting` edge on the 0→≥1 pending permission/question transition (only Claude/kilroy raise approvals/questions), and the client folds it in via `applyFreshAgentWaiting` under a distinct `${provider}:${sessionId}#waiting` dedupe namespace so it can never poison (or be poisoned by) the turn-complete bucket. The fragile client-side busy→idle derivation AND the client-side waiting-edge hook (`useAgentSessionTurnCompletion`) were both removed — all green/sound edges are now server-emitted. freshcodex additionally self-heals a crashed/disconnected codex sidecar by consuming the runtime `onExit` hook in `subscribe()`, emitting `sdk.status:'exited'` to clear BLUE (no chime — a crash is not a positive completion). freshcodex also runs a wedged-sidecar deadman: after a bounded quiet window (default 10 min, env `FRESHELL_FRESHCODEX_QUIET_WINDOW_MS`) with a turn in flight and no sidecar events, the server stops asserting busy and marks the pane `stuck`, and the client shows an amber "Agent appears stuck" card (`role="alert"`) with "Restart sidecar" (kill + resume re-mint) and "Start new conversation" actions; the deadman never fabricates a turn-complete (no green/chime). `freshopencode` still runs on a shared long-lived `opencode serve` sidecar and uses server-pushed `session.idle`/`session.status` events to drive busy. Gemini and Kimi terminal modes are status-in... [truncated] Separately, the sidebar shows cross-device remote status rings around a session row's icon: a green ring means the session is open on another device, a blue ring means it is busy on another device (blue wins over green), and rings are suppressed entirely when the session is open on this device (derived from `tabs.sync` registry snapshots — producing clients stamp pane payloads with `sessionKeys`/`busySessionKeys`, consumers re-query remote snapshots on a 30s interval, and the server partitions same-device records into `sameDeviceOpen`, which never produces rings). +**Agent Status Indicators:** Blue/busy status is derived from provider activity slices through `resolvePaneActivity`; green/needs-attention and the idle sound flow through `recordTurnComplete` and `useTurnCompletionNotifications`. Turn-complete (green/sound) is server-authoritative everywhere: terminal CLIs via `terminal.turn.complete`, and fresh-agent panes (freshclaude/kilroy/freshcodex/freshopencode) via a discrete `freshAgent.turn.complete` edge emitted only on a positive completion — freshclaude/kilroy on the SDK `result` with `subtype === 'success'`, freshopencode on the success-only `emitStatus(idle)` path, and freshcodex on `turn/completed` only when `params.turn.status === 'completed'` (the notification also fires on interrupt). The client folds it in via `applyFreshAgentCompletion` using the `at`-monotonic dedupe regime (wall-clock `at`, no per-session counter, so a resumed durable session can't swallow completions across a server restart). The waiting-for-approval edge is ALSO server-authoritative: the Claude/kilroy `SdkBridge` emits a discrete `freshAgent.turn.waiting` edge on the 0→≥1 pending permission/question transition (only Claude/kilroy raise approvals/questions), and the client folds it in via `applyFreshAgentWaiting` under a distinct `${provider}:${sessionId}#waiting` dedupe namespace so it can never poison (or be poisoned by) the turn-complete bucket. The fragile client-side busy→idle derivation AND the client-side waiting-edge hook (`useAgentSessionTurnCompletion`) were both removed — all green/sound edges are now server-emitted. freshcodex additionally self-heals a crashed/disconnected codex sidecar by consuming the runtime `onExit` hook in `subscribe()`, emitting `sdk.status:'exited'` to clear BLUE (no chime — a crash is not a positive completion). freshcodex also runs a wedged-sidecar deadman: after a bounded quiet window (default 10 min, env `FRESHELL_FRESHCODEX_QUIET_WINDOW_MS`) with a turn in flight and no sidecar events, the server stops asserting busy and marks the pane `stuck`, and the client shows an amber "Agent appears stuck" card (`role="alert"`) with "Restart sidecar" (kill + resume re-mint) and "Start new conversation" actions; the deadman never fabricates a turn-complete (no green/chime). `freshopencode` still runs on a shared long-lived `opencode serve` sidecar and uses server-pushed `session.idle`/`session.status` events to drive busy, and the runtime self-heals a shared-daemon death (the 2026-09-20 incident class): daemon loss fans a typed `freshAgent.error{code:"OPENCODE_DAEMON_LOST"}` edge out to exactly one frame per materialized session (the client's generic `sessionError` banner + busy-clear; NO chime — a crash is never a positive completion), the manager respawns the daemon on a backoff ladder (fresh-incident reset, crash-loop escalation), every successful cold start drives a level-triggered revival pass that restarts dead session bridges and pushes `freshAgent.session.snapshot{status:"idle"}` (the client's transcript-refetch trigger) while respecting the ownership coordinator (retired, transitioned, or terminal-owned sessions are never revived), and a generation-fenced `freshAgent.attach` is itself a recovery verb that respawns the daemon before re-bridging. Client-side, a snapshot GET that still answers the typed 409 `RESTORE_UNAVAILABLE` for the pane's own stale fresh-agent claim (e.g. the pane loaded while the server was restarting) does not dead-end on the dismiss-only banner: the pane drives the documented recovery ONCE per pane identity — it refreshes the observed owner fence from the refusal's own `ownerGeneration` (preserving the record's epoch), sends one generation-fenced `freshAgent.attach`, and refetches through the reveal path when reveal-dirty (so the "Refreshing conversation" overlay can clear) or via `manual` otherwise; a suppressed attach restores the once-guard, and repeated 409s fall through to the honest error banner — terminal-owned refusals stay out of this path (their recovery door is the session-directory handoff). Gemini and Kimi terminal modes are status-in... [truncated] Separately, the sidebar shows cross-device remote status rings around a session row's icon: a green ring means the session is open on another device, a blue ring means it is busy on another device (blue wins over green), and rings are suppressed entirely when the session is open on this device (derived from `tabs.sync` registry snapshots — producing clients stamp pane payloads with `sessionKeys`/`busySessionKeys`, consumers re-query remote snapshots on a 30s interval, and the server partitions same-device records into `sameDeviceOpen`, which never produces rings). **Fresh-Agent Orchestration:** The Rust REST agent API (`/api/tabs`, `/api/panes/:id/split`, `/api/panes/:id/send-keys`, `/api/panes/:id/capture`, `/api/panes/:id/wait-for`) and the standalone Node MCP client accept `agent`/`model`/`effort` parameters where the Rust contract supports them. The Rust orchestration layer dispatches to the registered fresh-agent runtimes. On MCP `new-tab`, resume sugar (`resume`/`resumeSessionId`) is honored for `agent: "opencode"`; terminal-mode resume uses an explicit provider-matched `sessionRef` (raw Codex resume IDs are rejected because they are not sufficient restore identity). Unsupported legacy actions return a deterministic unavailable result instead of contacting a removed backend route. diff --git a/crates/freshell-freshagent/src/opencode_ws.rs b/crates/freshell-freshagent/src/opencode_ws.rs index 6c66c1834..d4cd94738 100644 --- a/crates/freshell-freshagent/src/opencode_ws.rs +++ b/crates/freshell-freshagent/src/opencode_ws.rs @@ -65,8 +65,8 @@ use tokio::sync::Mutex as TokioMutex; use freshell_codex::next_monotonic_turn_complete_at; use freshell_opencode::{ - normalize_opencode_effort, normalize_opencode_model, ChangedReason, OpencodeServeManager, - SdkProviderEvent, ServeError, SessionSignal, SnapshotStatus, + normalize_opencode_effort, normalize_opencode_model, ChangedReason, DaemonSignal, + OpencodeServeManager, SdkProviderEvent, ServeError, SessionSignal, SnapshotStatus, }; use freshell_protocol::{ ErrorCode, ErrorMsg, FreshAgentAttach, FreshAgentCompact, FreshAgentConfigure, @@ -159,6 +159,54 @@ pub struct FreshOpencodeState { /// replacement probe re-issues the daemon-side abort through exactly /// this record. Cleared by whichever path settles the acceptance. condemned_sessions: Arc>>, + /// Task 4 (opencode daemon-death recovery): the daemon-loss watcher's + /// arming cell — set-once per state (Arc-shared across every clone), so + /// the FIRST `handle_send`/`handle_attach`/`handle_compact` arms exactly + /// ONE runtime-level listener on the manager's `DaemonSignal` stream for + /// the process lifetime. The watcher makes a shared-daemon death + /// observable (the typed `OPENCODE_DAEMON_LOST` edge per materialized + /// session) and recoverable (the level-triggered bridge revival on every + /// `Started`), mirroring the freshcodex onExit self-heal. See + /// [`Self::ensure_daemon_loss_watcher`]. + daemon_loss_watcher: Arc>, + /// ep2-r3 test seam (`cfg(test)`-only): when armed for a durable id, the + /// post-commit rescue tail parks BETWEEN its map re-lookup clone and the + /// session-lock take — the exact window the ep2-r3 fresheyes Major pins + /// (a kill/handoff completing against the rescue's retained `Arc`). + /// Never compiled into production builds. See [`Self::arm_rescue_stall`]. + #[cfg(test)] + rescue_stall: Arc>>, + /// ep2-r5 test seam (`cfg(test)`-only): when armed for a durable id, the + /// post-commit rescue tail parks BEFORE its map re-lookup — the window + /// where a same-key kill+replace can complete while the rescue is in + /// flight, so the re-lookup then observes whichever session (the + /// original or a replacement) the mover left under the key. Never + /// compiled into production builds. See + /// [`Self::arm_rescue_lookup_stall`]. + #[cfg(test)] + rescue_lookup_stall: Arc>>, +} + +/// The armed rescue-stall gate's state (ep2-r3 test seam, `cfg(test)`-only). +#[cfg(test)] +struct RescueStallGate { + /// The durable id this gate intercepts (the rescue is per-durable). + key: String, + /// Fires when the rescue has CLONED the session `Arc` out of the map and + /// is about to proceed — the exact parked point the finding's interleave + /// needs. + entered_tx: std::sync::mpsc::Sender<()>, + /// The test's release: the rescue parks until this oneshot resolves. + release_rx: StdMutex>>, +} + +/// The handles [`FreshOpencodeState::arm_rescue_stall`] hands the test +/// (ep2-r3 test seam, `cfg(test)`-only): `entered` fires at the parked point; +/// `release` lets the rescue proceed. +#[cfg(test)] +pub(crate) struct RescueStallHandles { + pub(crate) entered: std::sync::mpsc::Receiver<()>, + pub(crate) release: tokio::sync::oneshot::Sender<()>, } /// The condemned opencode session's quiescence identity (b8ke focused @@ -356,6 +404,21 @@ struct OpencodeSession { /// `freshAgent.session.snapshot` / `freshAgent.session.changed` / `freshAgent.error` /// for the lifetime of the session. `None` until materialized; aborted on kill. serve_bridge: Option>, + /// Delta round 2 (fresheyes Major — the doomed-bridge revival race): the + /// `ownership_id` of the daemon GENERATION the current `serve_bridge` + /// task was spawned against — the daemon-generation fence. Written ONLY + /// by [`FreshOpencodeState::install_serve_bridge`], together with the + /// handle, so `serve_bridge` is `Some` ⟹ this is `Some`. Read by + /// [`FreshOpencodeState::restart_session_bridge_guarded`]'s liveness + /// check: the manager's loss path fans `Lost` and re-warms + /// INDEPENDENTLY of the bridge tasks, so a bridge still draining its + /// closed channel when the new daemon's `Started` revival pass runs is + /// ALIVE-but-DOOMED — bound to a superseded generation it must be + /// restarted against, never mistaken for health by task liveness alone. + /// Not cleared on kill/handoff teardown: a torn-down session has no + /// handle, and a stale stamp with no handle is dead by the + /// absent-handle arm alone. + serve_bridge_daemon: Option, /// Retire-on-kill (delta-review round 5): set by `handle_kill` inside its /// session-lock phase. A send that took this session's Arc just before the /// kill's map removal is Parking on this lock and would otherwise @@ -430,6 +493,7 @@ impl OpencodeSession { turn_errored: Arc::new(AtomicBool::new(false)), last_turn_complete_at: Arc::new(StdMutex::new(None)), serve_bridge: None, + serve_bridge_daemon: None, killed: Arc::new(AtomicBool::new(false)), close_pending: 0, provenance: None, @@ -551,6 +615,11 @@ impl FreshOpencodeState { fork_in_flight: crate::InFlightRegistry::new(), rollback_in_flight: crate::InFlightRegistry::new(), condemned_sessions: Arc::new(std::sync::Mutex::new(HashMap::new())), + daemon_loss_watcher: Arc::new(std::sync::OnceLock::new()), + #[cfg(test)] + rescue_stall: Arc::new(StdMutex::new(None)), + #[cfg(test)] + rescue_lookup_stall: Arc::new(StdMutex::new(None)), } } @@ -607,6 +676,77 @@ impl FreshOpencodeState { self.fresh_agent.ownership_snapshot(provider, session_id) } + /// Arm the `cfg(test)`-only rescue stall (ep2-r3): the post-commit + /// rescue tail for `durable` fires `entered` once it has CLONED the + /// session `Arc` out of the sessions map, then parks until the test + /// sends on `release` — the deterministic interleaving point for "a + /// kill/handoff completes between the rescue's map lookup and its + /// session-lock take". One-shot per arming. + #[cfg(test)] + fn arm_rescue_stall(&self, durable: &str) -> RescueStallHandles { + let (entered_tx, entered) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = tokio::sync::oneshot::channel(); + *self.rescue_stall.lock().expect("rescue stall mutex") = Some(RescueStallGate { + key: durable.to_string(), + entered_tx, + release_rx: StdMutex::new(Some(release_rx)), + }); + RescueStallHandles { + entered, + release: release_tx, + } + } + + /// Take the armed rescue stall for `durable`, if it matches (one-shot; + /// `cfg(test)`-only, ep2-r3). + #[cfg(test)] + fn take_rescue_stall(&self, durable: &str) -> Option { + let mut guard = self.rescue_stall.lock().expect("rescue stall mutex"); + match guard.as_ref() { + Some(g) if g.key == durable => guard.take(), + _ => None, + } + } + + /// Arm the `cfg(test)`-only PRE-LOOKUP rescue stall (ep2-r5): the + /// post-commit rescue tail for `durable` fires `entered` BEFORE its + /// map re-lookup, then parks until the test sends on `release` — the + /// deterministic interleaving point for "a kill and a same-key + /// replacement resume complete while the rescue is in flight" (the + /// re-lookup below then finds whichever session the mover left under + /// the key). One-shot per arming. + #[cfg(test)] + fn arm_rescue_lookup_stall(&self, durable: &str) -> RescueStallHandles { + let (entered_tx, entered) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = tokio::sync::oneshot::channel(); + *self + .rescue_lookup_stall + .lock() + .expect("rescue lookup stall mutex") = Some(RescueStallGate { + key: durable.to_string(), + entered_tx, + release_rx: StdMutex::new(Some(release_rx)), + }); + RescueStallHandles { + entered, + release: release_tx, + } + } + + /// Take the armed pre-lookup rescue stall for `durable`, if it matches + /// (one-shot; `cfg(test)`-only, ep2-r5). + #[cfg(test)] + fn take_rescue_lookup_stall(&self, durable: &str) -> Option { + let mut guard = self + .rescue_lookup_stall + .lock() + .expect("rescue lookup stall mutex"); + match guard.as_ref() { + Some(g) if g.key == durable => guard.take(), + _ => None, + } + } + /// kata b8ke Task 3: begin this lane's coordinator claim. See /// [`crate::ownership_lane::begin_lane_claim`]. fn begin_lane_claim_at( @@ -655,6 +795,28 @@ impl FreshOpencodeState { ) } + /// The post-commit rescue's committed fence (fresheyes ep2-r5): the + /// `(epoch, generation)` pair the lane's own claim committed `Live` + /// under — the registry's boot epoch and the ticket's generation + /// (the exact generation `commit_live` lands at, so a still-own + /// coordinator observation reports the SAME pair back). `None` when + /// the coordinator is unwired or the caller holds no ticket (the + /// under-ticket handoff continuation — the runner's single `Live` + /// commit is still ahead — or the unwired lane, where no + /// coordinator observation exists to fence against). + fn committed_lane_fence( + ownership: &Option>, + ticket: Option<&freshell_ownership::OperationTicket>, + ) -> Option { + match (ownership.as_ref(), ticket) { + (Some(registry), Some(ticket)) => Some(freshell_ownership::ObservedFence { + epoch: registry.boot_epoch(), + generation: ticket.generation(), + }), + _ => None, + } + } + /// The initiator label for a coordinator transition event: the /// connection's device id when the provenance carries one, else the lane /// label (diagnostic, not audit-grade). @@ -1429,6 +1591,10 @@ impl FreshOpencodeState { let request_id = msg.request_id.clone(); let session_id = msg.session_id.clone(); + // Task 4: arm the runtime's daemon-loss watcher (idempotent — the + // first opencode WS traffic arms the one process-level listener). + self.ensure_daemon_loss_watcher().await; + let session_arc = { let guard = self.sessions.lock().await; guard.get(&session_id).cloned() @@ -1852,11 +2018,8 @@ impl FreshOpencodeState { // PR-3: `bindServeStream(state)` (adapter.ts:349) -- start the persistent // serve-SSE bridge ONCE, right after materialization. A later send never // re-enters this branch (mirrors `if (state.unsubscribeServe ...) return`). - session.serve_bridge = Some(self.spawn_serve_bridge( - manager.clone(), - durable_id.clone(), - session.turn_errored.clone(), - )); + self.install_serve_bridge(&manager, &mut session, &durable_id) + .await; // kata b8ke Task 3: the materialization's registration is // complete — commit `Live{FreshAgent}` for the minted `ses_*` @@ -2516,7 +2679,7 @@ impl FreshOpencodeState { cwd: Option<&str>, operation_id: &str, generation: u64, - ) -> Result { + ) -> Result<(freshell_ownership::OwnerIdentity, OpencodeSessionHandle), (String, String)> { match self .resume_durable_session( session_id, @@ -2527,15 +2690,22 @@ impl FreshOpencodeState { ) .await { - Ok(_session) => Ok(freshell_ownership::OwnerIdentity { - kind: freshell_ownership::RuntimeOwnerKind::FreshAgent, - terminal_id: None, - live_session_key: Some(session_id.to_string()), - // OpenCode invariant: the shared serve daemon is not the - // per-session writer — never a kill target, never the pid. - pid: None, - ownership_id: None, - }), + // ep2-r5: the under-ticket resume hands its committed session + // INSTANCE out with the identity — the runner's post-commit + // rescue re-check verifies this exact `Arc` against the map + // (a same-key replacement can never satisfy it). + Ok(session_arc) => Ok(( + freshell_ownership::OwnerIdentity { + kind: freshell_ownership::RuntimeOwnerKind::FreshAgent, + terminal_id: None, + live_session_key: Some(session_id.to_string()), + // OpenCode invariant: the shared serve daemon is not the + // per-session writer — never a kill target, never the pid. + pid: None, + ownership_id: None, + }, + OpencodeSessionHandle::new(session_arc), + )), Err(ResumeOpencodeError::NotFound) => Err(( "opencode session not known to the shared serve".to_string(), session_id.to_string(), @@ -3558,6 +3728,10 @@ impl FreshOpencodeState { return; } }; + // Task 4: arm the runtime's daemon-loss watcher (idempotent — the + // fence parse above stays the FIRST interaction, so a typed refusal + // still mutates nothing). + self.ensure_daemon_loss_watcher().await; let session_arc = { let guard = self.sessions.lock().await; guard.get(&session_id).cloned() @@ -4350,15 +4524,16 @@ impl FreshOpencodeState { ); child_session.provenance = fork_provenance.clone(); child_session.real_session_id = Some(child.id.clone()); - child_session.serve_bridge = Some(self.spawn_serve_bridge( - manager, - child.id.clone(), - child_session.turn_errored.clone(), - )); + self.install_serve_bridge(&manager, &mut child_session, &child.id) + .await; + // ep2-r5: bind the child's session INSTANCE — the fork's rescue + // verifies this exact `Arc` against the map (a same-key + // replacement for the child can never satisfy it). + let child_session_arc = Arc::new(TokioMutex::new(child_session)); self.sessions .lock() .await - .insert(child.id.clone(), Arc::new(TokioMutex::new(child_session))); + .insert(child.id.clone(), child_session_arc.clone()); // P1.13: binding row for the child (the materialization record pattern, // `_pattern :600-626`) — AWAITED BEFORE the forked reply @@ -4474,6 +4649,46 @@ impl FreshOpencodeState { return; } + // fresheyes ep2-r2 Major (the transitional-window dead bridge): the + // commit landed, so the child is now visible to every future + // revival trigger — but the triggers that fired inside the + // construction window are already spent. The rescue tail re-checks + // the bridge HERE, before the success reply: an operation must + // never answer success over a bridge its own window got killed + // (daemon loss swept the just-subscribed sender; the one-shot + // `Started` revival skipped the unpublished/`Starting` child). + // + // ep2-r3: a REFUSAL (the child was killed/handed off between the + // commit and this tail — the mover's teardown owns the cleanup) + // answers the stale-commit ownership-changed error frame instead: + // never a successful forked reply over a retired child, and never + // a resurrected bridge for it either. + // ep2-r5: the refusal set now includes a same-key REPLACEMENT for + // the child — the rescue is bound to the fork's committed instance + // and the child's committed (epoch, generation) fence, so it can + // only ever verify the fork's own child. + if matches!( + self.rescue_transitional_bridge_after_commit( + &child.id, + false, + &OpencodeSessionHandle::new(child_session_arc), + Self::committed_lane_fence(&self.fresh_agent.ownership, own_ticket.as_ref()), + ) + .await, + TransitionalBridgeRescue::Refused + ) { + reply_sink(event_frame( + &msg.session_id, + json!({ + "type": "freshAgent.error", + "sessionId": msg.session_id, + "code": "INTERNAL_ERROR", + "message": "session ownership changed during fork; the child was torn down", + }), + )); + return; + } + reply_sink(ServerMessage::FreshAgentForked(FreshAgentForked { request_id: msg.request_id.clone(), parent_session_id: msg.session_id.clone(), @@ -5179,6 +5394,10 @@ impl FreshOpencodeState { return; } }; + // Task 4: arm the runtime's daemon-loss watcher (idempotent — the + // fence parse above stays the FIRST interaction, so a typed refusal + // still mutates nothing). + self.ensure_daemon_loss_watcher().await; let session_arc = { let guard = self.sessions.lock().await; guard.get(&msg.session_id).cloned() @@ -5454,23 +5673,32 @@ impl FreshOpencodeState { let (status_session_id, running, real_session_id) = { let mut session = session_arc.lock().await; - // Ensure the serve-SSE bridge is running (restart it if it died) -- only - // meaningful once a durable session exists; a not-yet-materialized session has - // never started a bridge (`bindServeStream` only fires from `materializeOrSend`). - if let Some(real_id) = session.real_session_id.clone() { - let bridge_dead = session - .serve_bridge - .as_ref() - .map(tokio::task::JoinHandle::is_finished) - .unwrap_or(true); - if bridge_dead { - let manager = self.fresh_agent.ensure_manager().await; - session.serve_bridge = Some(self.spawn_serve_bridge( - manager, - real_id, - session.turn_errored.clone(), - )); - } + // Ensure the serve-SSE bridge is running (restart it if it died) -- + // only meaningful once a durable session exists; a not-yet- + // materialized session has never started a bridge (`bindServeStream` + // only fires from `materializeOrSend`). Task 4: the tail is the + // shared `restart_session_bridge_guarded` helper, which FIRST + // `ensure_started()`s the shared daemon (LB-05 — a restart without + // a daemon re-bridges into nothing; the incident's 3 attach + // attempts recovered NOTHING because nothing respawned the daemon + // for a map-hit). On a bounded respawn failure the attach answers + // the TYPED error path, never a silent half-attached state. + if let Err(err) = self.restart_session_bridge_guarded(&mut session).await { + drop(session); + tracing::warn!(target: "freshell_freshagent::opencode", + session_id = %msg.session_id, error = %err, + "freshagent.opencode.attach_daemon_respawn_failed: the dead-bridge \ + restart could not bring the shared daemon back — the attach answers \ + the typed error instead of a silent half-attached state" + ); + self.emit_fresh_agent_error( + &msg.session_id, + "OPENCODE_ATTACH_RESUME_FAILED", + &format!( + "The opencode serve daemon could not be restarted for this session: {err}" + ), + ); + return; } let status_session_id = session @@ -5733,11 +5961,8 @@ impl FreshOpencodeState { sink.as_ref() .and_then(|s| s.load_provenance(PROVIDER, session_id)) }); - session.serve_bridge = Some(self.spawn_serve_bridge( - manager, - session_id.to_string(), - session.turn_errored.clone(), - )); + self.install_serve_bridge(&manager, &mut session, session_id) + .await; let session_arc = Arc::new(TokioMutex::new(session)); self.sessions @@ -5958,6 +6183,50 @@ impl FreshOpencodeState { } } + // fresheyes ep2-r2 Major (the transitional-window dead bridge): the + // resume installed its bridge BEFORE the map publication and the + // claim commits above, so a daemon loss inside that window killed + // the fresh bridge with every recovery trigger spent (an + // unpublished session is invisible to the revival pass; a plain + // resume holds no `Live` state for it to rescue either; the + // re-warm's already-running fast path emits no second `Started`). + // Every failure gate has passed — the rescue tail re-checks the + // bridge HERE, before the success return, so the resume never + // hands back a session whose bridge died in its own window. + // + // ep2-r3: a REFUSAL (the session was killed/handed off between the + // commit and this tail — the mover's teardown owns the cleanup) + // answers the stale-commit ownership-changed error instead: never + // a handed-back retired session, and never a resurrected bridge for + // it either. The under-ticket handoff continuation runs inside the + // RUNNER's own lifecycle window (`handoff.is_some()`) — the rescue's + // coordinator observation is skipped for exactly that leg (the + // runner performs the one `commit_live`); the killed check and the + // map re-lookup remain its gates. + // ep2-r5: the refusal set now includes a same-key REPLACEMENT — + // the rescue is bound to THIS resume's committed instance (the + // `session_arc` it returns below, so a replacement under the key + // can never be adopted and the retired `Arc` can never be handed + // back) and, for the own-commit leg, the resume's committed + // (epoch, generation) fence. + if matches!( + self.rescue_transitional_bridge_after_commit( + session_id, + handoff.is_some(), + &OpencodeSessionHandle::new(Arc::clone(&session_arc)), + Self::committed_lane_fence(&self.fresh_agent.ownership, own_ticket.as_ref()), + ) + .await, + TransitionalBridgeRescue::Refused + ) { + return Err(ResumeOpencodeError::Manager( + freshell_opencode::ServeError::Transport(format!( + "opencode session {session_id} ownership changed during resume; \ + the session was retired mid-operation" + )), + )); + } + Ok(session_arc) } @@ -6022,6 +6291,655 @@ impl FreshOpencodeState { } }) } + + /// Install a fresh serve-SSE bridge on `session` for `real_id`, stamped + /// with the CURRENT daemon's `ownership_id` — the daemon-generation + /// fence [`Self::restart_session_bridge_guarded`] compares against. + /// EVERY bridge install goes through here (materialization, the resume + /// and fork-child constructions, the guarded restart), so the + /// handle/stamp pair can never drift apart. + async fn install_serve_bridge( + &self, + manager: &OpencodeServeManager, + session: &mut OpencodeSession, + real_id: &str, + ) { + // Read the identity BEFORE spawning: the stamp names the daemon + // whose event stream the bridge is about to subscribe to. A daemon + // lost in between is caught by the fence itself (the stamp no longer + // matches the running daemon, or none runs) — the next revival pass + // or fenced attach restarts the bridge. + let daemon = manager.ownership_id().await; + session.serve_bridge = Some(self.spawn_serve_bridge( + manager.clone(), + real_id.to_string(), + session.turn_errored.clone(), + )); + session.serve_bridge_daemon = daemon; + } + + // ── Task 4 (opencode daemon-death recovery): the runtime-level self-heal ── + + /// Arm the runtime's daemon-loss watcher (2026-09-20 incident: the shared + /// `opencode serve` daemon died and NOTHING told the panes — no status + /// edge, no bridge revival; panes dead-ended on the snapshot 409). + /// Idempotent: the set-once [`OnceLock`] guarantees exactly ONE listener + /// per state no matter how many handlers call this. Armed from + /// [`Self::handle_send`] / [`Self::handle_attach`] / [`Self::handle_compact`] + /// — the opencode WS entry points — so the listener exists before any + /// session traffic can depend on it. The watcher task holds a FULL state + /// clone (LB-10), so `spawn_serve_bridge(&self, ..)` is callable directly. + /// + /// - `Lost{reason}`: WARN `freshagent.opencode.daemon_loss_observed`, then + /// fan the typed `OPENCODE_DAEMON_LOST` edge out to every MATERIALIZED + /// session ([`Self::fan_out_daemon_loss_edge`]). The manager's own + /// backoff-guarded re-warm (Task 3) is already scheduled at that point. + /// - `Started`: the LEVEL-TRIGGERED revival pass + /// ([`Self::revive_dead_bridges_if_daemon_running`]) — never dependent + /// on having observed `Lost` (LB-02: tokio broadcast does NOT replay + /// history to late subscribers). + /// - `Lagged`: continue (LB-08: never disarm on lag). + /// - `Closed`: the manager (and its signal sender) is gone — return. + /// + /// NO CHIME (the freshcodex onExit mirror's discipline): neither edge of + /// this watcher ever emits `freshAgent.turn.complete` — a crash is not a + /// positive completion. + async fn ensure_daemon_loss_watcher(&self) { + if self.daemon_loss_watcher.set(()).is_err() { + return; // already armed (the cell is Arc-shared across every clone) + } + let manager = self.fresh_agent.ensure_manager().await; + let state = self.clone(); + let mut signals = manager.subscribe_daemon_signals(); + tokio::spawn(async move { + // Arming-time LEVEL pass (LB-02): broadcast does NOT replay + // history — a daemon that re-warmed before this subscription + // must still get its dead bridges revived now. + state.revive_dead_bridges_if_daemon_running().await; + loop { + match signals.recv().await { + Ok(DaemonSignal::Lost { reason }) => { + tracing::warn!(target: "freshell_freshagent::opencode", + reason = reason, + "freshagent.opencode.daemon_loss_observed: the shared \ + opencode serve daemon was lost — fanning the typed edge \ + out to every materialized session; the manager's \ + backoff-guarded re-warm is already scheduled" + ); + state.fan_out_daemon_loss_edge().await; + } + Ok(DaemonSignal::Started) => { + // LEVEL-TRIGGERED revival (LB-02): revive whatever is + // dead right now — no `saw_loss` heuristic. + state.revive_dead_bridges_if_daemon_running().await; + } + Err(RecvError::Lagged(_)) => continue, + Err(RecvError::Closed) => return, + } + } + }); + } + + /// The daemon-loss fan-out (Task 4): exactly ONE typed edge per + /// MATERIALIZED session — + /// `freshAgent.event{provider:"opencode", sessionType:"freshopencode", + /// event:{type:"freshAgent.error", code:"OPENCODE_DAEMON_LOST", message}}` + /// — which the client folds through the EXISTING generic `sessionError` + /// path (the dismissible "Agent error:" banner + busy-clear). The + /// sessions map is keyed by BOTH the placeholder and the durable id + /// pointing at the SAME session (`remember()` mirror) — dedupe by + /// `real_session_id` (BTreeSet) so each materialized session gets exactly + /// ONE edge. The message names the re-warm so the banner reads as the + /// self-heal it is, and NO chime ever accompanies it. + async fn fan_out_daemon_loss_edge(&self) { + const CODE: &str = "OPENCODE_DAEMON_LOST"; + const MESSAGE: &str = + "The opencode serve daemon was lost unexpectedly - it is restarting automatically."; + for id in self.materialized_session_ids().await { + self.emit_fresh_agent_error(&id, CODE, MESSAGE); + } + } + + /// The distinct durable `ses_*` ids of every MATERIALIZED session in the + /// map (dual-key dedupe), for the loss fan-out and the revival pass + /// alike. LB-01: the sessions-map guard is NEVER held across a + /// per-session lock (the documented contract above — the reverse edge + /// deadlocked production) — the session `Arc`s are cloned out under ONE + /// short map lock, the guard drops, and each session is read outside it. + async fn materialized_session_ids(&self) -> Vec { + let arcs: Vec>> = { + let map = self.sessions.lock().await; + map.values().cloned().collect() + }; + let mut ids = std::collections::BTreeSet::new(); + for arc in arcs { + if let Some(id) = arc.lock().await.real_session_id.clone() { + ids.insert(id); + } + } + ids.into_iter().collect() + } + + /// The LEVEL-TRIGGERED revival pass (Task 4, LB-02): restart dead/absent + /// serve-SSE bridges for MATERIALIZED sessions while the shared daemon + /// runs, and push `freshAgent.session.snapshot{status:"idle"}` ONLY to + /// sessions whose bridge was actually restarted (the client treats that + /// push as snapshot-invalidating → transcript refetch). Called on watcher + /// arming and on every `DaemonSignal::Started` — never dependent on + /// having observed `Lost`. "Dead" is judged by the shared tail's + /// daemon-generation fence ([`Self::restart_session_bridge_guarded`]): + /// a bridge still bound to a LOST daemon generation restarts even + /// while its old task is draining (the doomed-bridge revival race, + /// delta round 2); a bridge already bound to the CURRENT daemon is + /// healthy and is neither restarted nor re-pushed. + /// + /// The ownership coordinator (plan-review round 3) is respected at every + /// step: (1) `base_url()` is None → return (daemon absent — nothing to + /// revive into; the next `Started` or a fenced attach drives revival); + /// (2) snapshot the map (clone the `Arc`s under one short lock, drop the + /// guard); (3) per candidate OUTSIDE the map guard: RE-LOOKUP the id at + /// revival time (a killed/handed-off session's keys are gone — never act + /// on the retained `Arc` alone), observe the CANONICAL ownership state + /// fresh, skip on ANY transition (Handoff/Starting/Stopping/Fenced) or a + /// terminal owner, revive only `Live{FreshAgent}` (this runtime's own + /// sessions) behind the SAME `arm_adopt_guard` the attach path uses, + /// then run the shared [`Self::restart_session_bridge_guarded`] tail. An + /// unwired coordinator applies no gate (the pre-wiring legacy: map + /// membership is the only authority). + async fn revive_dead_bridges_if_daemon_running(&self) { + let manager = self.fresh_agent.ensure_manager().await; + if manager.base_url().await.is_none() { + return; + } + for durable in self.materialized_session_ids().await { + // (3a) Re-lookup at revival time — the retained Arc alone is + // stale the moment a kill/handoff removes the keys. + let session_arc = { + let map = self.sessions.lock().await; + map.get(&durable).cloned() + }; + let Some(session_arc) = session_arc else { + continue; + }; + // (3b) The ownership gate — armed with the SAME adopt-guard + // machinery the attach path uses, so the coordinator's + // atomicity rides along instead of being re-implemented. + let mut adopt_guard = None; + if self.fresh_agent.ownership.is_some() { + let snap = self + .fresh_agent + .canonical_ownership_snapshot(PROVIDER, &durable); + match snap.state { + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent => + { + let expected = freshell_ownership::OwnerIdentity { + kind: freshell_ownership::RuntimeOwnerKind::FreshAgent, + terminal_id: None, + live_session_key: None, + pid: None, + ownership_id: None, + }; + match crate::ownership_lane::arm_adopt_guard( + &self.fresh_agent.ownership, + PROVIDER, + &durable, + &format!("daemon-revive-{}", uuid::Uuid::new_v4()), + &expected, + freshell_ownership::ObservedFence { + epoch: snap.epoch, + generation: snap.generation, + }, + "freshopencode/daemon-revival", + ) { + crate::ownership_lane::LaneAttachGuard::Armed(guard) => { + adopt_guard = Some(guard) + } + // The coordinator moved between the observe and + // the arm — a lifecycle owns the window; skip, + // never force. + crate::ownership_lane::LaneAttachGuard::Refused => continue, + crate::ownership_lane::LaneAttachGuard::Unwired => {} + } + } + // Any transition (Handoff/Starting/Stopping/Fenced), a + // terminal owner, a vacant key, any other kind: someone + // else's window — never revive into it. + _ => continue, + } + } + // (3c/3d) The shared guarded-restart tail, under the session + // lock; the snapshot push goes ONLY to actually-restarted + // bridges. `Ok(None)` (bridge alive against the CURRENT daemon / + // unmaterialized) is the quiet no-op. + let restarted = { + let mut session = session_arc.lock().await; + self.restart_session_bridge_guarded(&mut session).await + }; + match restarted { + Ok(Some(real_id)) => { + self.broadcast(&event_frame(&real_id, snapshot_event(&real_id, "idle"))); + } + Ok(None) => {} + Err(err) => { + tracing::warn!(target: "freshell_freshagent::opencode", + session_id = %durable, error = %err, + "freshagent.opencode.daemon_revival_restart_failed: the \ + bridge restart's bounded daemon respawn failed — the next \ + Started signal or a fenced attach retries" + ); + } + } + // The guard covered the restart; release the window. + drop(adopt_guard); + } + } + + /// The GUARD-HELD bridge-restart tail (the Task 4 refactor): the ONE + /// shared restart both [`Self::handle_attach`]'s dead-bridge arm and the + /// revival pass ([`Self::revive_dead_bridges_if_daemon_running`]) call. + /// Callers arm `ownership_lane::arm_adopt_guard` and hold it ACROSS this + /// call so a handoff beginning inside the window answers the typed + /// Blocked outcome. The caller holds the per-session lock here (NEVER + /// the sessions-map guard — LB-01). + /// + /// LB-05 (falsified → redesign): in the incident the fenced attach was + /// exercised 3× against the dead shared daemon and recovered nothing — + /// the tail only re-subscribed the bridge. A restart must first + /// `ensure_started()` the shared daemon (mirroring + /// `resume_durable_session`'s map-miss behavior); `ensure_started` is + /// single-flighted, so concurrent attach/send/compact callers cannot + /// spawn a second daemon. + /// + /// Delta round 2 (fresheyes Major — the doomed-bridge revival race): + /// task liveness alone is NOT bridge health. The manager's loss path + /// fans `Lost` and re-warms INDEPENDENTLY of the bridge tasks, so a + /// bridge still draining its closed channel when the new daemon's + /// `Started` revival pass (or a fenced attach) reaches this tail is + /// ALIVE-but-DOOMED. The daemon-generation fence: the bridge is + /// healthy ONLY while its task is unfinished AND its + /// [`OpencodeSession::serve_bridge_daemon`] stamp matches the CURRENT + /// daemon's `ownership_id` (an absent daemon orphans every bridge; an + /// unstamped handle is unreachable by construction — every install + /// stamps — and fails toward recovery). + /// + /// `Ok(Some(real_id))` — the bridge was (re)started; `Ok(None)` — nothing + /// to do (unmaterialized, or the bridge is alive against the current + /// daemon); `Err` — the BOUNDED respawn failed (the caller answers + /// typed, never a silent half-attached state). + async fn restart_session_bridge_guarded( + &self, + session: &mut OpencodeSession, + ) -> Result, ServeError> { + // Only meaningful once a durable session exists; a not-yet- + // materialized session has never started a bridge (`bindServeStream` + // only fires from `materializeOrSend`). + let Some(real_id) = session.real_session_id.clone() else { + return Ok(None); + }; + let manager = self.fresh_agent.ensure_manager().await; + let current_daemon = manager.ownership_id().await; + let bridge_bound_to_current_daemon = matches!( + (&session.serve_bridge_daemon, ¤t_daemon), + (Some(stamp), Some(current)) if stamp == current + ); + let bridge_dead = session + .serve_bridge + .as_ref() + .map(|handle| handle.is_finished() || !bridge_bound_to_current_daemon) + .unwrap_or(true); + if !bridge_dead { + return Ok(None); + } + manager.ensure_started().await?; + // The superseded/draining task is ABORTED, not merely detached: + // dropping a JoinHandle does not reap the task, and a doomed + // bridge must not outlive the replacement it lost to. + if let Some(old) = session.serve_bridge.take() { + old.abort(); + } + self.install_serve_bridge(&manager, session, &real_id).await; + Ok(Some(real_id)) + } + + /// The POST-COMMIT transitional-bridge rescue tail (fresheyes ep2-r2 + /// Major — the transitional-window dead bridge; HARDENED ep2-r3 — the + /// rescue can resurrect a killed/handed-off session): the fork-child + /// and resume constructions install the generation-fenced bridge + /// BEFORE the session is published in `sessions` and committed + /// `Live`, so a daemon loss landing inside that window kills the fresh + /// bridge (the loss sweep claims its just-subscribed sender; the fanned + /// `Lost` closes its channel) while every recovery trigger misses it — + /// the successor's one-shot `Started` revival pass cannot see an + /// unpublished session, deliberately skips transitional ownership + /// states, and the re-warm's already-running fast path emits no second + /// `Started`. The operation would otherwise answer success over a + /// permanently dead bridge. The tail closes the window from the + /// operation's own side: after the `Live` commit (the only state the + /// revival pass rescues), it re-runs the SAME fenced + /// [`Self::restart_session_bridge_guarded`] tail the revival pass + /// uses — a bridge still alive against the CURRENT daemon is the + /// quiet `Healthy` no-op (no push, no restart: the happy path is + /// byte-identical), a bridge stamped to the lost generation (or + /// exited, or absent) is restarted against the successor and the + /// client gets the one recovery snapshot. The position-independent + /// re-check covers BOTH window variants — the unpublished leg (a + /// revival pass can never see a session before its map publication) + /// and the published-`Starting` leg (a pass that sees it must skip + /// it) — because it judges the bridge's health at tail time, not + /// where in the window the loss landed. + /// + /// **ep2-r3 hardening — the rescue rides the ESTABLISHED + /// ownership-coordinator-gated restart discipline** (the same gates + /// the revival pass and the fenced attach use), never a bare + /// install on a retained `Arc`: (a) a FRESH map re-lookup — a miss + /// is the typed [`TransitionalBridgeRescue::Refused`] (the keys were + /// removed by a kill/handoff whose teardown already aborted the + /// bridge), never a silent success over the retired session; + /// (b) the CANONICAL ownership observation + `arm_adopt_guard` window + /// — revive only `Live{FreshAgent}`; every transition + /// (Handoff/Starting/Stopping/Fenced), a terminal owner, and a + /// vacant key refuse, and the guard is held ACROSS the restart so a + /// handoff beginning inside the window answers its typed Blocked + /// outcome; (c) the `killed` check UNDER the session lock — a + /// kill/close/handoff teardown that completed between the map clone + /// and the lock take is final, never resurrected (its detached + /// `JoinHandle` would keep broadcasting for the retired session). + /// `own_lifecycle_window` skips ONLY the observation (b): the + /// under-ticket handoff continuation resumes inside the RUNNER's own + /// held claim (the coordinator key is mid-transition by design and + /// the runner — not this lane — performs the one `commit_live`), so + /// the killed check and the map re-lookup remain its gates. Every + /// caller whose lifecycle window has ALREADY committed passes + /// `false` — the observation then sees the caller's own + /// `Live{FreshAgent}` and arms the adopt guard across the restart + /// (the revival pass's exact discipline): the lane's own-commit + /// tails (fork/resume) and, since ep2-r4, the handoff runner's + /// post-commit re-check for its freshopencode targets — the + /// runner's commit is the ONE `Live` commit an under-ticket + /// continuation ever gets, so the runner re-runs this seam after it + /// (a daemon loss in the runner's awaited flavor-write window kills + /// the just-rescued bridge with every other recovery trigger spent: + /// the successor's `Started` revival pass skips the transitional + /// owner, and the commit itself emits no new daemon signal). + /// + /// **ep2-r5 hardening — the rescue is bound to the identity the + /// CALLER committed, never to whichever session currently occupies + /// the durable id**: every caller passes its committed session + /// instance ([`OpencodeSessionHandle`]) and its committed ownership + /// fence (the `(epoch, generation)` pair observed at ITS commit). + /// The by-id map re-lookup must return the SAME instance + /// (`Arc::ptr_eq` against the caller's handle) — a same-key + /// REPLACEMENT (the original killed/removed and another resume's + /// brand-new session object under the same durable id, moved in + /// during the awaited post-commit work) is a different instance and + /// REFUSES, never gets adopted — and, when the coordinator gates + /// the call, the canonical observation must still hold the CALLER's + /// OWN committed `(epoch, generation)` era: a replacement's + /// same-kind `Live{FreshAgent}` at a NEWER generation refuses too + /// (the kind-only expected-owner check alone accepted it — the + /// finding's exact hole). On either mismatch the outcome is + /// [`TransitionalBridgeRescue::Refused`]: the operation surfaces its + /// typed ownership-changed error, never answers success carrying its + /// obsolete generation, and never restarts a bridge on the retired + /// instance (the resume caller would otherwise hand back its + /// removed-and-killed `Arc`). The under-ticket pre-commit leg + /// (`own_lifecycle_window = true`) passes no fence — the runner's + /// `Live` commit is still ahead — so the instance binding, the + /// killed check, and the map re-lookup remain its gates; the + /// post-commit callers (the fork/resume tails, the handoff runner's + /// re-check) always carry the committed fence. + /// + /// A [`TransitionalBridgeRescue::Refused`] never installs and never + /// pushes: the mover's teardown owns the cleanup (the kill/handoff + /// paths remove every key, set the killed flag, and abort the bridge + /// under the same locks this rescue re-verifies), and the OPERATION + /// surfaces its typed ownership-changed error — the stale-commit + /// path's shape — instead of answering success over the retired + /// session. A bounded respawn failure still WARNS and the operation + /// still succeeds (the session is registered and visible to the next + /// `Started` revival pass and fenced attaches — the revival pass's + /// failure handling, verbatim). + pub(crate) async fn rescue_transitional_bridge_after_commit( + &self, + durable: &str, + own_lifecycle_window: bool, + committed: &OpencodeSessionHandle, + committed_fence: Option, + ) -> TransitionalBridgeRescue { + // ep2-r5 test seam (`cfg(test)`-only): park BEFORE the map + // re-lookup — the window where a test can complete a same-key + // kill+replace while the rescue is in flight, so the re-lookup + // below then observes whichever session the mover left under the + // key. See [`Self::arm_rescue_lookup_stall`]. + #[cfg(test)] + if let Some(gate) = self.take_rescue_lookup_stall(durable) { + let _ = gate.entered_tx.send(()); + // Take the release receiver OUT of its Mutex before awaiting — + // the std guard is not Send and must not live across the await. + let release = gate + .release_rx + .lock() + .expect("rescue lookup stall latch mutex") + .take(); + if let Some(release) = release { + let _ = release.await; + } + } + // (a) The fresh map re-lookup — the retained-claim-alone rule (the + // revival pass's (3a)): a session whose map keys are gone was + // killed/handed off mid-operation and its teardown already aborted + // the bridge. Never act on the retained claim alone. + let session_arc = { + let map = self.sessions.lock().await; + map.get(durable).cloned() + }; + let Some(session_arc) = session_arc else { + return TransitionalBridgeRescue::Refused; + }; + // (a′) The committed-INSTANCE binding (ep2-r5): the re-lookup must + // return the SAME session instance the caller committed — + // `Arc::ptr_eq` against the caller's handle. A replacement resume + // under the same durable id is a brand-new object: adopting it + // would answer the operation's success over ITS OWN retired + // session while the replacement lives under the key (and the + // resume caller would hand back its removed-and-killed `Arc` for a + // later bridge restart). Refuse — the mover's teardown owns the + // original's cleanup and the replacement's own lifecycle owns the + // key now. + if !committed.ptr_eq(&session_arc) { + return TransitionalBridgeRescue::Refused; + } + // ep2-r3 test seam (`cfg(test)`-only): park the rescue in the exact + // window the fresheyes finding names — the session `Arc` is CLONED + // and the session lock is not yet taken — so a test can complete a + // real kill/handoff against the retained `Arc` before the rescue + // proceeds. See [`Self::arm_rescue_stall`]. + #[cfg(test)] + if let Some(gate) = self.take_rescue_stall(durable) { + let _ = gate.entered_tx.send(()); + // Take the release receiver OUT of its Mutex before awaiting — + // the std guard is not Send and must not live across the await. + let release = gate + .release_rx + .lock() + .expect("rescue release latch mutex") + .take(); + if let Some(release) = release { + let _ = release.await; + } + } + // (b) The ownership gate — the revival pass's (3b) shape: observe + // the CANONICAL state fresh, revive only Live{FreshAgent}, and arm + // the SAME adopt-guard the attach path uses so the coordinator's + // atomicity rides across the restart below. An unwired coordinator + // applies no gate (the pre-wiring legacy: the killed check and the + // map re-lookup are the authority). + let mut adopt_guard = None; + if !own_lifecycle_window && self.fresh_agent.ownership.is_some() { + let snap = self + .fresh_agent + .canonical_ownership_snapshot(PROVIDER, durable); + match snap.state { + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent => + { + // (b′) The committed-FENCE binding (ep2-r5): the + // coordinator must still hold the CALLER's OWN era — + // the observed `(epoch, generation)` must EQUAL the + // pair the caller committed, not merely be a + // same-kind Live. A replacement resume committed at a + // strictly newer per-key generation (generations are + // monotonic and never reset), so the equality refuses + // exactly the shape the kind-only check accepted + // before. + let Some(committed_fence) = committed_fence else { + // A post-commit rescue over a WIRED coordinator + // with no committed fence has no era to verify — + // fail closed, never restart into an unverified + // ownership window. Unreachable for the real + // callers (a wired registry grants every + // own-commit caller a ticket; the handoff runner + // always carries its committed pair). + return TransitionalBridgeRescue::Refused; + }; + if snap.epoch != committed_fence.epoch + || snap.generation != committed_fence.generation + { + return TransitionalBridgeRescue::Refused; + } + let expected = freshell_ownership::OwnerIdentity { + kind: freshell_ownership::RuntimeOwnerKind::FreshAgent, + terminal_id: None, + live_session_key: None, + pid: None, + ownership_id: None, + }; + match crate::ownership_lane::arm_adopt_guard( + &self.fresh_agent.ownership, + PROVIDER, + durable, + &format!("transitional-rescue-{}", uuid::Uuid::new_v4()), + &expected, + freshell_ownership::ObservedFence { + epoch: snap.epoch, + generation: snap.generation, + }, + "freshopencode/transitional-rescue", + ) { + crate::ownership_lane::LaneAttachGuard::Armed(guard) => { + adopt_guard = Some(guard) + } + // The coordinator moved between the observe and the + // arm — a lifecycle owns the window; skip, never + // force. + crate::ownership_lane::LaneAttachGuard::Refused => { + return TransitionalBridgeRescue::Refused + } + // Unreachable with a wired registry (checked above); + // kept total against the enum. + crate::ownership_lane::LaneAttachGuard::Unwired => {} + } + } + // Any transition (Handoff/Starting/Stopping/Fenced), a + // terminal owner, a vacant key, any other kind: someone + // else's window — never revive into it. + _ => return TransitionalBridgeRescue::Refused, + } + } + // (c) The session lock: the killed check, then the shared guarded + // restart tail (the per-session lock is held here — never the + // sessions-map guard; LB-01). The adopt-guard (b) covers the + // restart and releases at the return below. + let restarted = { + let mut session = session_arc.lock().await; + if session.killed.load(Ordering::SeqCst) { + // A kill/close/handoff teardown marked the session retired + // while this rescue held its pre-lock clone — never + // resurrect; the teardown owns the cleanup. + return TransitionalBridgeRescue::Refused; + } + self.restart_session_bridge_guarded(&mut session).await + }; + drop(adopt_guard); + match restarted { + Ok(Some(real_id)) => { + self.broadcast(&event_frame(&real_id, snapshot_event(&real_id, "idle"))); + TransitionalBridgeRescue::Restarted + } + Ok(None) => TransitionalBridgeRescue::Healthy, + Err(err) => { + tracing::warn!(target: "freshell_freshagent::opencode", + session_id = %durable, error = %err, + "freshagent.opencode.transitional_bridge_rescue_failed: the \ + post-commit bridge rescue's bounded daemon respawn failed — \ + the next Started signal or a fenced attach retries" + ); + TransitionalBridgeRescue::RespawnFailed + } + } + } +} + +/// The post-commit rescue's outcome (ep2-r3): the operation tail (the +/// fork's reply, the resume's return, the handoff runner's post-commit +/// re-check) consumes this to decide between its success path and its +/// typed ownership-changed refusal. See +/// [`FreshOpencodeState::rescue_transitional_bridge_after_commit`]. +pub(crate) enum TransitionalBridgeRescue { + /// The bridge was (re)started against the successor daemon and the + /// client got its one recovery snapshot (pushed by the rescue itself — + /// only actually-restarted bridges are ever pushed). + Restarted, + /// Nothing to do — the bridge is alive against the current daemon (or + /// the session is unmaterialized): the quiet no-op; the operation + /// succeeds unchanged. + Healthy, + /// The bounded daemon respawn failed — WARN-only; the operation still + /// succeeds (the registered session is visible to the next `Started` + /// revival pass and fenced attaches). + RespawnFailed, + /// The session was retired or transitioned between the commit and the + /// tail — its map keys are gone (a kill/handoff removed them), its + /// `killed` flag is set under the session lock, the map key now holds + /// a DIFFERENT session instance than the caller committed (a same-key + /// replacement), or the canonical ownership no longer holds the + /// caller's committed `(epoch, generation)` era (and is not this + /// operation's own lifecycle window): NEVER resurrect, never adopt + /// the replacement, never answer success carrying an obsolete + /// generation. The mover's teardown owns the cleanup; the operation + /// must surface its typed ownership-changed error instead of + /// answering success over the retired session. + Refused, +} + +/// An opaque handle to ONE live freshopencode session instance (fresheyes +/// ep2-r5): identity is the `Arc` pointer itself, so a same-key +/// REPLACEMENT — another resume's brand-new session object under the +/// SAME durable id — can never masquerade as the instance a caller +/// committed. The handoff runner carries this out of its under-ticket +/// resume (the Ok payload of +/// [`FreshOpencodeState::opencode_resume_for_handoff`] → `start_target`) +/// and hands it back to +/// [`FreshOpencodeState::rescue_transitional_bridge_after_commit`], +/// which verifies it with `Arc::ptr_eq` against the sessions map's +/// current entry. The wrapped [`OpencodeSession`] stays private to this +/// module; the runner only ever passes the handle through. +#[derive(Clone)] +pub(crate) struct OpencodeSessionHandle(Arc>); + +impl OpencodeSessionHandle { + /// Wrap the session instance a caller is committing (or has just + /// committed) under its durable id. + fn new(instance: Arc>) -> Self { + Self(instance) + } + + /// The committed-instance binding (ep2-r5): true iff `current` IS the + /// session instance this handle was minted from. + fn ptr_eq(&self, current: &Arc>) -> bool { + Arc::ptr_eq(&self.0, current) + } } /// ISO-8601 / RFC-3339 millis-Z timestamp (matches `new Date().toISOString()`) for error @@ -10394,11 +11312,26 @@ mod tests { }, ); - let owner = st + let (owner, committed_instance) = st .opencode_resume_for_handoff("ses_r27_oc", None, "handoff-op-r27", 9) .await .expect("the under-ticket target resume succeeds"); assert_eq!(owner.kind, freshell_ownership::RuntimeOwnerKind::FreshAgent); + // ep2-r5: the under-ticket resume carries its committed session + // INSTANCE out with the identity — the exact `Arc` registered + // under the durable id (the runner's post-commit rescue re-check + // binds to it). + let registered = st + .sessions + .lock() + .await + .get("ses_r27_oc") + .cloned() + .expect("the under-ticket resume registered the target session"); + assert!( + committed_instance.ptr_eq(®istered), + "the carried handle IS the registered session instance (ep2-r5)" + ); let bindings = fake.bindings.lock().unwrap(); let last = bindings @@ -12987,10 +13920,20 @@ mod tests { } /// Insert a directly-materialized session (no send drove it) with the given model. + /// + /// Task 4 fixture fidelity: a materialized session carries a LIVE + /// serve-SSE bridge in production (`bindServeStream` fires at + /// materialization), so the fixture spawns one through the same + /// [`FreshOpencodeState::spawn_serve_bridge`] the materialization path + /// uses — the daemon-loss watcher (now armed by `handle_compact`) + /// otherwise "revives" the bridgeless fixture at its arming-time level + /// pass and pushes idle-snapshot frames these tests never modeled. async fn insert_compact_session(st: &FreshOpencodeState, id: &str, model: Option<&str>) { let mut session = OpencodeSession::new(id.to_string(), None, model.map(str::to_string), None); session.real_session_id = Some(id.to_string()); + let manager = st.fresh_agent.ensure_manager().await; + st.install_serve_bridge(&manager, &mut session, id).await; st.sessions .lock() .await @@ -18081,4 +19024,2024 @@ mod tests { assert_eq!(last.settings.model.as_deref(), Some("prov/mdl-b")); assert_eq!(last.settings.effort.as_deref(), Some("low")); } + + // ── Task 4 (opencode daemon-death recovery): the runtime-level self-heal ── + // + // 2026-09-20 incident: the shared daemon died and NOTHING told the panes — + // no status edge, no respawn, no bridge revival; panes dead-ended on the + // snapshot 409. The runtime self-heal must make daemon loss observable and + // recoverable per session (mirroring the freshcodex onExit self-heal). + // (LB-02: revival is LEVEL-TRIGGERED — it runs on arming and on every + // `Started`, never dependent on having observed `Lost`.) + + /// A serve whose "exit" is test-controlled: `exited()` reports `Some(0)` + /// once the shared flag is set (the Task 3 selfheal fixture shape) — the + /// flag-driven daemon death both the manager's exit watcher and the + /// runtime's loss listener observe. + struct FlagExitProcess { + exited: Arc, + } + impl ServeProcess for FlagExitProcess { + fn exited(&self) -> Option { + self.exited.load(Ordering::SeqCst).then_some(0) + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) {} + } + + /// Every generation hands out a [`FlagExitProcess`] sharing the flag; + /// spawns counted (the respawn accounting the fenced-attach test asserts). + struct FlagExitSpawner { + exited: Arc, + spawns: Arc, + } + impl ProcessSpawner for FlagExitSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + self.spawns.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(FlagExitProcess { + exited: self.exited.clone(), + })) + } + } + + /// Task 4 harness: a state wired to a STARTED selfheal-config fake manager + /// (tiny watch + backoff knobs so the manager's own backoff-guarded re-warm + /// runs at test speed) whose daemon's death is flag-controlled, plus the + /// bus receiver every frame assertion reads. + async fn selfheal_state( + backoff_initial_ms: u64, + backoff_max_ms: u64, + ) -> ( + FreshOpencodeState, + tokio::sync::broadcast::Receiver, + Arc, + Arc, + OpencodeServeManager, + ) { + let (tx, rx) = tokio::sync::broadcast::channel::(256); + let fresh_agent = FreshAgentState::new(Arc::new("tok".to_string()), Arc::new(tx)); + let exited = Arc::new(AtomicBool::new(false)); + let spawns = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(FakeHttp { + next_session: AtomicUsize::new(0), + }), + ports: Arc::new(FakeAllocator), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + idle_poll_interval: Duration::from_millis(20), + daemon_watch_interval: Duration::from_millis(10), + re_warm_backoff_initial_ms: backoff_initial_ms, + re_warm_backoff_max_ms: backoff_max_ms, + ..ServeConfig::default() + }; + let manager = OpencodeServeManager::new(deps, config); + manager + .ensure_started() + .await + .expect("healthy fake serve starts"); + fresh_agent.set_manager_for_test(manager.clone()).await; + ( + FreshOpencodeState::new(fresh_agent), + rx, + exited, + spawns, + manager, + ) + } + + /// The session's durable `ses_*` id (the fixture must have materialized). + async fn real_session_id_of(st: &FreshOpencodeState, placeholder: &str) -> String { + let session_arc = { + let sessions = st.sessions.lock().await; + sessions.get(placeholder).expect("tracked").clone() + }; + let durable = session_arc + .lock() + .await + .real_session_id + .clone() + .expect("the session materialized"); + durable + } + + /// Is the session's serve-SSE bridge live? False for an unmapped id (a + /// killed/handed-off session) and for a mapped session whose bridge is + /// dead/absent — the exact predicate the revival pass restarts on. + async fn session_serve_bridge_alive(st: &FreshOpencodeState, id: &str) -> bool { + let Some(session_arc) = st.sessions.lock().await.get(id).cloned() else { + return false; + }; + let session = session_arc.lock().await; + session + .serve_bridge + .as_ref() + .map(|b| !b.is_finished()) + .unwrap_or(false) + } + + /// Bounded wait until the coordinator shows the durable id Live under a + /// FRESH-AGENT owner (the materialization commit the revival/attach gates + /// consult). + async fn await_freshagent_live( + registry: &Arc, + durable: &str, + ) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + loop { + if matches!( + registry.observe("opencode", durable).state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ) { + return; + } + assert!( + tokio::time::Instant::now() < deadline, + "the materialization's Live{{FreshAgent}} commit never landed for {durable}" + ); + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + + /// Model the committed-terminal-handoff shape: `begin_handoff` (granted) + /// then `commit_live` under a TERMINAL owner — the exact pair of calls a + /// finished terminal handoff leaves behind in the coordinator. + async fn commit_terminal_owner( + registry: &Arc, + durable: &str, + ) { + let operation_id = format!("handoff-{durable}"); + let freshell_ownership::BeginOutcome::Granted { generation } = registry.begin_handoff( + "opencode", + durable, + freshell_ownership::RuntimeOwnerKind::Terminal, + &operation_id, + None, + "test", + 0, + ) else { + panic!("the handoff begin must grant for {durable}") + }; + assert!( + matches!( + registry.commit_live( + "opencode", + durable, + &operation_id, + generation, + freshell_ownership::OwnerIdentity { + kind: freshell_ownership::RuntimeOwnerKind::Terminal, + terminal_id: Some(format!("t-{durable}")), + live_session_key: None, + pid: None, + ownership_id: None, + }, + ), + freshell_ownership::CommitOutcome::Committed + ), + "the terminal owner commit must land for {durable}" + ); + } + + /// Materialize one session through the REAL create+send path and settle + /// its turn locally (the existing IdleTimeout-shaped seam) so the only + /// frames after it are the machinery under test. Returns + /// (placeholder, durable). + async fn materialized_selfheal_session( + st: &FreshOpencodeState, + rx: &mut tokio::sync::broadcast::Receiver, + req: &str, + ) -> (String, String) { + st.handle_create(create_msg(req), None).await; + let placeholder = format!("freshopencode-{req}"); + st.handle_send(send_msg(&placeholder, "materialize")).await; + let durable = real_session_id_of(st, &placeholder).await; + st.settle_local_turn_task_for_test(&placeholder).await; + let _ = drain_frames(rx); + (placeholder, durable) + } + + /// Task 4 contract: the incident's silent half — a daemon that dies must + /// (a) fan a TYPED `OPENCODE_DAEMON_LOST` edge out to every MATERIALIZED + /// session (the dismissible banner + busy-clear the client folds through + /// the generic `sessionError` path), exactly ONE edge per session even + /// though the map is DUAL-KEYED (placeholder + durable id → the SAME + /// session), (b) NEVER chime (a crash is not a positive completion), and + /// (c) after the manager's backoff-guarded respawn, restart the dead + /// bridge and push the `status:"idle"` snapshot the client treats as a + /// transcript refetch. + #[tokio::test] + async fn daemon_loss_fans_out_a_typed_edge_then_revives_bridges_after_respawn() { + let (st, mut rx, exited, _spawns, manager) = selfheal_state(5, 50).await; + + let (_placeholder, durable) = + materialized_selfheal_session(&st, &mut rx, "req-daemon-loss").await; + // Fixture honesty: the session is dual-keyed — the dedupe contract's + // whole point (two map keys, ONE materialized session). + assert_eq!( + st.sessions.lock().await.len(), + 2, + "fixture: placeholder + durable keys both map the session" + ); + + exited.store(true, Ordering::SeqCst); // the daemon dies + + // THE TYPED EDGE (the incident's missing half). + let loss_frames = frames_until(&mut rx, |f| { + f["type"] == "freshAgent.event" + && f["event"]["type"] == "freshAgent.error" + && f["event"]["code"] == "OPENCODE_DAEMON_LOST" + }) + .await; + let edge = loss_frames.last().expect("the matching edge"); + assert_eq!(edge["provider"], "opencode"); + assert_eq!(edge["sessionType"], "freshopencode"); + assert_eq!(edge["sessionId"].as_str(), Some(durable.as_str())); + assert_eq!(edge["event"]["sessionId"].as_str(), Some(durable.as_str())); + assert_eq!( + edge["event"]["message"].as_str(), + Some( + "The opencode serve daemon was lost unexpectedly - it is restarting automatically." + ) + ); + + exited.store(false, Ordering::SeqCst); // the re-warm's respawn now succeeds + + // `DaemonSignal::Started` → the LEVEL-TRIGGERED revival: the dead + // bridge restarts and the client sees the snapshot-invalidating idle + // push (never a remembered-`Lost` heuristic). + let revive_frames = frames_until(&mut rx, |f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable.as_str()) + }) + .await; + + // NO chime ever accompanies a daemon loss, and the DUAL-KEYED session + // got exactly ONE edge — audit the whole post-loss window. + let mut all = loss_frames; + all.extend(revive_frames); + all.extend(drain_frames(&mut rx)); + let edges = all + .iter() + .filter(|f| { + f["type"] == "freshAgent.event" + && f["event"]["type"] == "freshAgent.error" + && f["event"]["code"] == "OPENCODE_DAEMON_LOST" + && f["sessionId"].as_str() == Some(durable.as_str()) + }) + .count(); + assert_eq!( + edges, 1, + "exactly ONE loss edge for the dual-keyed session: {all:?}" + ); + assert!( + all.iter() + .all(|f| f["event"]["type"] != "freshAgent.turn.complete"), + "a daemon loss is never a positive completion — no chime: {all:?}" + ); + + assert!( + session_serve_bridge_alive(&st, &durable).await, + "the bridge must be restarted after the respawn" + ); + assert!( + manager.base_url().await.is_some(), + "the manager's backoff-guarded re-warm respawned the daemon" + ); + } + + /// Delta round 2 (fresheyes Major — the doomed-bridge revival race): the + /// manager's loss path clears the running entry, fans `Lost`, and + /// re-warms INDEPENDENTLY of the per-session bridge tasks — a bridge + /// task still draining its closed channel when the new daemon's + /// `Started` revival pass runs was treated as healthy (task liveness + /// alone), so the pass answered `Ok(None)`: no restart, no recovery + /// snapshot, and the pane dead-ended on NO bridge. The interleaving is + /// forced to its extreme here: the session's bridge is a task that + /// never finishes on its own (the real bridge's draining window + /// between consuming `Lost` and observing channel-closed, held open + /// past any revival pass), the daemon generation moves A→B underneath + /// it, and the `Started`-driven revival pass must nevertheless + /// RESTART the bridge bound to the new daemon and push the client's + /// recovery snapshot. + #[tokio::test] + async fn revival_restarts_a_bridge_still_draining_from_a_lost_daemon_generation() { + let (st, mut rx, exited, _spawns, manager) = selfheal_state(5, 50).await; + + let (_placeholder, durable) = + materialized_selfheal_session(&st, &mut rx, "req-doomed-bridge").await; + + // THE DOOMED BRIDGE: swap the materialized session's live bridge for + // a stand-in task that never finishes on its own — a bridge + // provably ALIVE (so the pre-fix liveness check is fooled) yet + // doomed (its daemon is about to be lost). The drop-guard records + // the abort the restart owes the superseded task: a restart that + // merely dropped the handle would leak the still-draining task. + let doomed_aborted = Arc::new(AtomicBool::new(false)); + struct AbortWitness(Arc); + impl Drop for AbortWitness { + fn drop(&mut self) { + self.0.store(true, Ordering::SeqCst); + } + } + let witness = doomed_aborted.clone(); + { + let session_arc = st + .sessions + .lock() + .await + .get(&durable) + .cloned() + .expect("the materialized session is tracked under its durable id"); + let mut session = session_arc.lock().await; + if let Some(old) = session.serve_bridge.take() { + old.abort(); + } + session.serve_bridge = Some(tokio::spawn(async move { + let _abort_witness = AbortWitness(witness); + std::future::pending::<()>().await + })); + } + + exited.store(true, Ordering::SeqCst); // daemon generation A dies + + // The loss completes — the typed edge proves the watcher's Lost arm + // ran (lose_daemon cleared the running entry before the edge was + // fanned, and scheduled the re-warm). + let loss_frames = frames_until(&mut rx, |f| { + f["type"] == "freshAgent.event" + && f["event"]["code"] == "OPENCODE_DAEMON_LOST" + && f["sessionId"].as_str() == Some(durable.as_str()) + }) + .await; + + exited.store(false, Ordering::SeqCst); // the re-warm's respawn now succeeds + + // `DaemonSignal::Started` → the LEVEL-TRIGGERED revival pass runs + // while the doomed bridge task is provably STILL ALIVE. It must + // not be fooled by task liveness: the bridge is bound to the LOST + // daemon generation, so it restarts against the new daemon and + // pushes the snapshot-invalidating idle frame (the client's + // transcript refetch). On the pre-fix code this never arrives — + // the pass sees the unfinished task, answers Ok(None), and the + // session is left with NO bridge. + let revive_frames = frames_until(&mut rx, |f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable.as_str()) + }) + .await; + + // The restart ABORTED the superseded draining task (bounded wait — + // the abort drop is scheduler-side). + let deadline = tokio::time::Instant::now() + Duration::from_secs(2); + while !doomed_aborted.load(Ordering::SeqCst) { + assert!( + tokio::time::Instant::now() < deadline, + "the revival restart must abort the doomed draining bridge task" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + + // The session now holds a NEW live bridge, bound to the respawned + // daemon. + assert!( + session_serve_bridge_alive(&st, &durable).await, + "the doomed bridge was replaced by a live bridge bound to the new daemon" + ); + assert!( + manager.base_url().await.is_some(), + "the manager's backoff-guarded re-warm respawned the daemon" + ); + + // Exactly ONE recovery snapshot for the actually-restarted bridge + // (a later level-triggered pass must no-op against the now-current + // bridge — push only to actually-restarted bridges), and NO chime + // anywhere in the post-loss window. + tokio::time::sleep(Duration::from_millis(100)).await; + let mut all = loss_frames; + all.extend(revive_frames); + all.extend(drain_frames(&mut rx)); + let pushes = all + .iter() + .filter(|f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable.as_str()) + }) + .count(); + assert_eq!( + pushes, 1, + "exactly ONE recovery snapshot for the restarted bridge: {all:?}" + ); + assert!( + all.iter() + .all(|f| f["event"]["type"] != "freshAgent.turn.complete"), + "a daemon loss is never a positive completion — no chime: {all:?}" + ); + } + + /// Bounded wait until the session's serve-SSE bridge reaches the wanted + /// liveness — the ep2-r2 transitional-window fixture's observable (the + /// bridge's death by channel-close and its rescue are both scheduler-side + /// events, so the fixture polls instead of racing them). + async fn await_bridge_liveness( + st: &FreshOpencodeState, + id: &str, + want_alive: bool, + what: &str, + ) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + loop { + if session_serve_bridge_alive(st, id).await == want_alive { + return; + } + assert!( + tokio::time::Instant::now() < deadline, + "timed out waiting for the {id} bridge to be {}: {what}", + if want_alive { "alive" } else { "dead" } + ); + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + + /// ep2-r2 fresheyes Major (the transitional-window dead bridge): the + /// generation-fenced bridge is installed BEFORE the forked child is + /// published in `sessions` or committed `Live` (materialization remains + /// `Starting`), so a daemon loss that captures the fresh bridge's sender + /// inside that window leaves the child with a dead bridge and NO + /// recovery trigger: B's one-shot `Started` revival pass either cannot + /// see the child (unpublished) or deliberately skips its transitional + /// ownership state, and the re-warm's fast path (B already running) + /// emits no second `Started`. The operation still answers success — a + /// pane with a permanently dead event bridge. + /// + /// The interleaving, forced deterministically through the REAL paths: + /// the child's binding-row write parks the fork between its map + /// publication and its `Live` commit (`arm_binding_stall`); while + /// parked, daemon A dies through the REAL watcher arm (the take sweeps + /// the child's just-subscribed sender — the fanned `Lost` closes the + /// bridge's channel and the task exits), B re-warms, and B's one-shot + /// `Started` revival pass runs: it rescues the PARENT (`Live`) — and, + /// because the pass visits the sorted durable ids ("ses_child" before + /// "ses_parent"), the parent's rescue PROVES the pass already visited + /// and SKIPPED the `Starting` child. Releasing the park lets the + /// commit land, and the operation must not answer success over the + /// dead bridge: the post-commit rescue tail restarts it against B and + /// pushes the client's recovery snapshot (exactly one; no chime + /// anywhere). + #[tokio::test(flavor = "multi_thread")] + async fn transitional_fork_child_survives_a_daemon_loss_in_its_starting_window() { + // The selfheal manager shape (flag-controlled daemon death, + // test-speed watch/backoff knobs) over a FORK-serving HTTP fake. + let (tx, mut rx) = tokio::sync::broadcast::channel::(256); + let fresh_agent = FreshAgentState::new(Arc::new("tok".to_string()), Arc::new(tx)); + let exited = Arc::new(AtomicBool::new(false)); + let spawns = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(ForkFakeHttp::child_ok()), + ports: Arc::new(FakeAllocator), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + idle_poll_interval: Duration::from_millis(20), + daemon_watch_interval: Duration::from_millis(10), + re_warm_backoff_initial_ms: 5, + re_warm_backoff_max_ms: 50, + ..ServeConfig::default() + }; + let manager = OpencodeServeManager::new(deps, config); + manager + .ensure_started() + .await + .expect("healthy fake serve starts"); + fresh_agent.set_manager_for_test(manager.clone()).await; + let mut st = FreshOpencodeState::new(fresh_agent); + let registry = Arc::new(freshell_ownership::RuntimeOwnershipRegistry::new()); + st.set_ownership(Arc::clone(®istry)); + let fake = Arc::new(crate::identity_sink::FakeIdentitySink::default()); + st.set_identity_sink(fake.clone()); + insert_fork_parent(&st, "ses_parent", Some("/parent/cwd"), None, None).await; + seed_live_fresh_owner(®istry, "ses_parent"); + // Arm the runtime's daemon-loss watcher (production arms it at the + // first WS entry point; the fork lane does not, so arm it here). Its + // arming-time level pass bridges the bridgeless parent; drain that + // push so the frames below are only the machinery under test. + st.ensure_daemon_loss_watcher().await; + tokio::time::sleep(Duration::from_millis(50)).await; + let _ = drain_frames(&mut rx); + assert!( + session_serve_bridge_alive(&st, "ses_parent").await, + "fixture: the arming-time pass bridged the parent" + ); + + // Park the fork between the child's map publication and its `Live` + // commit: the child's binding-row write stalls behind the release. + let stall = fake.arm_binding_stall("opencode", "ses_child"); + let (sink, captured) = capturing_sink(); + + // The mid-fork mover: once the fork parks, drive the finding's + // exact interleaving through the REAL loss/re-warm/revival paths, + // prove the revival pass skipped the transitional child, then + // release the park. + let mover_state = st.clone(); + let mover_manager = manager.clone(); + let mover_exited = exited.clone(); + let mover = tokio::spawn(async move { + stall + .entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the fork parks between the child's publication and its Live commit"); + // Daemon A dies. The loss completes: the take cleared the + // running slot (the sweep ran inside the same critical + // section)... + mover_exited.store(true, Ordering::SeqCst); + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + while mover_manager.base_url().await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "A's loss must take the running entry within the budget" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + // ...and the fanned `Lost` + dropped swept senders closed BOTH + // bridges' channels: the parent's A-era bridge and the parked + // child's (the loss captured the child's just-subscribed + // sender — the finding's premise). + await_bridge_liveness( + &mover_state, + "ses_parent", + false, + "A's loss drains the parent's A-era bridge", + ) + .await; + await_bridge_liveness( + &mover_state, + "ses_child", + false, + "A's loss drains the parked child's bridge (its sender was swept)", + ) + .await; + mover_exited.store(false, Ordering::SeqCst); + // B re-warms and its one-shot `Started` revival pass runs: the + // PARENT (Live) is rescued. The pass visits "ses_child" FIRST + // (sorted ids), so the parent's rescue proves the pass already + // visited — and skipped — the transitional child. + await_bridge_liveness( + &mover_state, + "ses_parent", + true, + "B's Started revival rescues the Live parent", + ) + .await; + assert!( + !session_serve_bridge_alive(&mover_state, "ses_child").await, + "fixture: the Started revival deliberately skipped the Starting \ + child — the dead transitional bridge the tail must rescue" + ); + stall.release.send(()).expect("release the parked fork"); + }); + + let mut fork_request = fork_msg("ses_parent", "fork-req-trans", None); + ( + fork_request.observed_epoch, + fork_request.observed_generation, + ) = fenced_observed(®istry, "ses_parent"); + st.handle_fork(fork_request, None, sink).await; + mover.await.expect("the mover completed the interleaving"); + + // The operation SUCCEEDED — the finding's premise: a successful + // forked reply over the dead transitional bridge. + let captured = captured.lock().expect("captured mutex").clone(); + assert_eq!( + captured.len(), + 1, + "the fork answers exactly one frame: {captured:?}" + ); + assert!( + matches!(captured[0], ServerMessage::FreshAgentForked { .. }), + "the forked reply landed: {captured:?}" + ); + await_freshagent_live(®istry, "ses_child").await; + + // THE BRIDGE: the pane must not dead-end — the child ends up with a + // LIVE bridge bound to the successor daemon (RED on HEAD: the + // skipped revival left the dead A-era task in place and nothing + // ever restarted it). + assert!( + session_serve_bridge_alive(&st, "ses_child").await, + "the fork must not answer success over a dead bridge — the \ + transitional child's bridge was killed by the daemon loss and \ + no revival trigger ever came (ep2-r2)" + ); + + // THE RECOVERY SNAPSHOT: the client's transcript-refetch push for + // the actually-restarted bridge. + let revive_frames = frames_until(&mut rx, |f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some("ses_child") + }) + .await; + let mut all = revive_frames; + all.extend(drain_frames(&mut rx)); + let pushes = all + .iter() + .filter(|f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some("ses_child") + }) + .count(); + assert_eq!( + pushes, 1, + "exactly ONE recovery snapshot for the rescued child: {all:?}" + ); + let edges = all + .iter() + .filter(|f| { + f["type"] == "freshAgent.event" + && f["event"]["type"] == "freshAgent.error" + && f["event"]["code"] == "OPENCODE_DAEMON_LOST" + && f["sessionId"].as_str() == Some("ses_child") + }) + .count(); + assert_eq!( + edges, 1, + "exactly ONE typed loss edge for the child: {all:?}" + ); + assert!( + all.iter() + .all(|f| f["event"]["type"] != "freshAgent.turn.complete"), + "a daemon loss is never a positive completion — no chime: {all:?}" + ); + assert!( + manager.base_url().await.is_some(), + "fixture: the successor daemon B serves" + ); + let _ = spawns; + } + + /// ep2-r3 fresheyes Major (the rescue can resurrect a killed/handed-off + /// session): the post-commit rescue clones the session `Arc` out of the + /// map, releases the map lock, and takes the session lock WITHOUT + /// rechecking `killed`, map membership, or ownership — a kill/handoff + /// that completes in that window (killed flag set, every map key + /// removed, the coordinator's stop committed) leaves the rescue + /// holding a RETIRED session whose absent bridge it then "restarts": a + /// detached fresh-agent bridge for the killed session, an idle-snapshot + /// push over the retired session, and the operation answering SUCCESS + /// over it. + /// + /// The interleaving, forced deterministically through the `cfg(test)` + /// rescue stall: the fork child's rescue fires `entered` right after its + /// map-lookup clone and parks; while held, a REAL `freshAgent.kill` of + /// the child COMPLETES (killed flag, every map key, the coordinator's + /// Vacant commit — the ownership observation's refusal input); only + /// then is the rescue released. It must REFUSE — no bridge install, no + /// snapshot push — and the fork must answer its stale-commit + /// ownership-changed error frame, never `FreshAgentForked`. Pre-fix, + /// the released rescue installed a fresh bridge on the killed child + /// (its detached task kept broadcasting for the retired session) and + /// pushed the idle snapshot, and the fork answered success. + #[tokio::test(flavor = "multi_thread")] + async fn a_kill_landing_between_the_rescue_lookup_and_its_lock_never_resurrects_the_fork_child() + { + // The selfheal manager shape (the daemon never dies in this test — + // the KILL is the mover) over a FORK-serving HTTP fake. + let (tx, mut rx) = tokio::sync::broadcast::channel::(256); + let fresh_agent = FreshAgentState::new(Arc::new("tok".to_string()), Arc::new(tx)); + let exited = Arc::new(AtomicBool::new(false)); + let spawns = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(ForkFakeHttp::child_ok()), + ports: Arc::new(FakeAllocator), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + idle_poll_interval: Duration::from_millis(20), + daemon_watch_interval: Duration::from_millis(10), + re_warm_backoff_initial_ms: 60_000, + re_warm_backoff_max_ms: 120_000, + ..ServeConfig::default() + }; + let manager = OpencodeServeManager::new(deps, config); + manager + .ensure_started() + .await + .expect("healthy fake serve starts"); + fresh_agent.set_manager_for_test(manager.clone()).await; + let mut st = FreshOpencodeState::new(fresh_agent); + let registry = Arc::new(freshell_ownership::RuntimeOwnershipRegistry::new()); + st.set_ownership(Arc::clone(®istry)); + let fake = Arc::new(crate::identity_sink::FakeIdentitySink::default()); + st.set_identity_sink(fake.clone()); + insert_fork_parent(&st, "ses_parent", Some("/parent/cwd"), None, None).await; + seed_live_fresh_owner(®istry, "ses_parent"); + + // Park the child's post-commit rescue between its map-lookup clone + // and its session-lock take. + let stall = st.arm_rescue_stall("ses_child"); + let (sink, captured) = capturing_sink(); + let st2 = st.clone(); + let mut fork_request = fork_msg("ses_parent", "fork-ep2r3", None); + ( + fork_request.observed_epoch, + fork_request.observed_generation, + ) = fenced_observed(®istry, "ses_parent"); + let fork = tokio::spawn(async move { st2.handle_fork(fork_request, None, sink).await }); + + stall + .entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the rescue parks between its map-lookup clone and its session-lock take"); + + // The concurrent kill, driven through the REAL path to completion: + // the killed flag, EVERY map key aliasing the child, the bridge + // abort, and the coordinator's Stopping→Vacant commit. + st.handle_kill(FreshAgentKill { + provider: AgentProvider::Opencode, + session_id: "ses_child".to_string(), + session_type: SessionType::Freshopencode, + cwd: None, + observed_epoch: None, + observed_generation: None, + }) + .await; + assert!( + !st.has_live_session("ses_child").await, + "fixture: the kill removed every map key for the child" + ); + assert!( + !matches!( + st.ownership_snapshot("opencode", "ses_child").state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ), + "fixture: the coordinator no longer holds the child Live{{FreshAgent}}" + ); + + // Release the parked rescue: its `Arc` was cloned BEFORE the kill — + // only the re-checks (killed/map/ownership) can stop a resurrection. + stall.release.send(()).expect("release the parked rescue"); + fork.await.expect("the fork settles"); + + // THE operation outcome: the stale-commit ownership-changed error + // frame — never a successful FreshAgentForked over the retired + // child (pre-fix: the fork answered success here). + let captured = captured.lock().expect("captured mutex").clone(); + assert_eq!( + captured.len(), + 1, + "the fork answers exactly one frame: {captured:?}" + ); + match &captured[0] { + ServerMessage::FreshAgentForked { .. } => panic!( + "the fork answered success over a session the concurrent kill \ + retired — the rescue resurrected it (ep2-r3)" + ), + ServerMessage::FreshAgentEvent(event) => { + assert_eq!( + event.event["code"], "INTERNAL_ERROR", + "the refusal is the stale-commit error frame: {:?}", + captured[0] + ); + assert!( + event.event["message"] + .as_str() + .unwrap_or_default() + .contains("session ownership changed during fork"), + "the stale-commit ownership-changed error frame: {:?}", + captured[0] + ); + } + other => panic!("unexpected fork answer: {other:?}"), + } + + // NO bridge was installed for the retired child: a REAL dispatch + // through the manager's emitter map reaches no resurrected bridge — + // no freshAgent frame for the killed session (pre-fix: the detached + // bridge translated the dispatch and broadcast for it). + manager.dispatch_event( + freshell_opencode::events::parse_serve_event(&json!({ + "type": "session.idle", + "properties": { "sessionID": "ses_child" } + })) + .expect("parseable serve event"), + ); + tokio::time::sleep(Duration::from_millis(100)).await; + let all = drain_frames(&mut rx); + assert!( + all.iter().all( + |f| !(is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some("ses_child")) + ), + "no recovery-snapshot push for the retired child: {all:?}" + ); + assert!( + !all.iter() + .any(|f| f["type"] == "freshAgent.event" + && f["sessionId"].as_str() == Some("ses_child")), + "no resurrected bridge frame for the killed session: {all:?}" + ); + // The coordinator's post-kill state stands — nothing re-registered + // the child. + assert!( + !matches!( + st.ownership_snapshot("opencode", "ses_child").state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ), + "the kill's coordinator commit stands — the rescue never re-owns the child" + ); + let _ = (exited, spawns); + } + + /// The same interleaving on the RESUME leg, with the coordinator UNWIRED + /// — the killed-check floor: nothing but the session's `killed` flag (set + /// under the session lock by the kill's phase 3) and the map re-lookup + /// stand between the released rescue and a resurrection. The resume must + /// surface its typed ownership-changed error instead of handing back a + /// session the concurrent kill retired. + #[tokio::test(flavor = "multi_thread")] + async fn a_kill_landing_between_the_rescue_lookup_and_its_lock_fails_the_resume_typed() { + // The selfheal fixture answers the cold resume's GET /session/:id + // with a benign `{}` row; NO registry is wired (the unwired lane — + // the killed check is the only gate), and the daemon never dies. + let (st, mut rx, exited, spawns, manager) = selfheal_state(60_000, 120_000).await; + let fake = Arc::new(crate::identity_sink::FakeIdentitySink::default()); + st.set_identity_sink(fake.clone()); + + // Park the rebuilt session's post-commit rescue between its + // map-lookup clone and its session-lock take. + let stall = st.arm_rescue_stall("ses_resume"); + let st2 = st.clone(); + let resume = tokio::spawn(async move { + st2.resume_durable_session("ses_resume", None, None, None, None) + .await + }); + + stall + .entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the rescue parks between its map-lookup clone and its session-lock take"); + + // The concurrent kill, through the REAL path, to completion. + st.handle_kill(FreshAgentKill { + provider: AgentProvider::Opencode, + session_id: "ses_resume".to_string(), + session_type: SessionType::Freshopencode, + cwd: None, + observed_epoch: None, + observed_generation: None, + }) + .await; + assert!( + !st.has_live_session("ses_resume").await, + "fixture: the kill removed the rebuilt session's map key" + ); + + // Release the parked rescue and settle the resume. + stall.release.send(()).expect("release the parked rescue"); + let out = resume.await.expect("the resume settles"); + + // THE outcome: the typed ownership-changed refusal — never Ok over + // the retired session (pre-fix: the resume returned the killed + // session and the rescue had installed a fresh bridge on it). + match out { + Err(ResumeOpencodeError::Manager(freshell_opencode::ServeError::Transport(msg))) => { + assert!( + msg.contains("ownership changed during resume"), + "the typed refusal names the ownership change: {msg}" + ); + } + Err(ResumeOpencodeError::Manager(err)) => { + panic!("the refusal must be the ownership-changed Transport error, got {err:?}") + } + Err(_) => { + panic!("the refusal must be a Manager error (NotFound/Reserved are wrong here)") + } + Ok(_) => panic!( + "the resume returned success over a session the concurrent kill \ + retired (ep2-r3)" + ), + } + + // NO bridge install, NO snapshot push for the retired session. + manager.dispatch_event( + freshell_opencode::events::parse_serve_event(&json!({ + "type": "session.idle", + "properties": { "sessionID": "ses_resume" } + })) + .expect("parseable serve event"), + ); + tokio::time::sleep(Duration::from_millis(100)).await; + let all = drain_frames(&mut rx); + assert!( + all.iter().all( + |f| !(is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some("ses_resume")) + ), + "no recovery-snapshot push for the retired session: {all:?}" + ); + assert!( + !all.iter().any(|f| f["type"] == "freshAgent.event" + && f["sessionId"].as_str() == Some("ses_resume")), + "no resurrected bridge frame for the killed session: {all:?}" + ); + let _ = (exited, spawns); + } + + // ── fresheyes ep2-r4: the handoff runner's post-commit bridge re-check ── + + /// The ep2-r4 flavor-writer latch: the runner's OWN awaited flavor steps + /// are the deterministic interleaving points (existing injection seams — + /// no `cfg(test)` production gates). `stage` parks the runner INSIDE the + /// Handoff window (before the `Live` commit — the pre-commit + /// daemon-loss test's hold); the staged handle's `commit` parks the + /// runner AFTER the commit (the post-commit kill test's hold). An + /// unwired latch is a plain immediate `Ok` — the happy-path writer. + #[derive(Default)] + struct ParkingFlavorWriter { + stage_entered: StdMutex>>, + stage_release: StdMutex>>, + commit_entered: StdMutex>>, + commit_release: StdMutex>>, + } + + /// The staged flavor handle [`ParkingFlavorWriter`] hands the runner: + /// `commit()` fires `entered` and parks on `release` when armed. + struct ParkingStagedFlavor { + entered: Option>, + release: Option>, + } + + impl crate::session_handoff::StagedFlavor for ParkingStagedFlavor { + fn commit( + self: Box, + ) -> std::pin::Pin> + Send>> + { + let Self { entered, release } = *self; + Box::pin(async move { + if let Some(entered) = entered { + let _ = entered.send(()); + } + if let Some(release) = release { + let _ = release.await; + } + Ok(()) + }) + } + } + + impl crate::session_handoff::FlavorWrite for ParkingFlavorWriter { + fn stage( + &self, + _provider: &str, + _session_id: &str, + _flavor: &str, + ) -> crate::session_handoff::StagedFlavorFuture { + let stage_entered = self.stage_entered.lock().expect("writer latch").take(); + let stage_release = self.stage_release.lock().expect("writer latch").take(); + let commit_entered = self.commit_entered.lock().expect("writer latch").take(); + let commit_release = self.commit_release.lock().expect("writer latch").take(); + Box::pin(async move { + if let Some(entered) = stage_entered { + let _ = entered.send(()); + } + if let Some(release) = stage_release { + let _ = release.await; + } + Ok(Some(Box::new(ParkingStagedFlavor { + entered: commit_entered, + release: commit_release, + }) + as Box)) + }) + } + } + + /// The ep2-r4 fixture: the selfheal manager shape (flag-controlled + /// daemon death, test-speed watch/backoff knobs) wired into a FULL + /// [`crate::session_handoff::SessionHandoffRunner`] — the finding lives + /// in the RUNNER's commit path (`start_target`'s under-ticket + /// continuation → the awaited flavor write → the single `Live` + /// commit), so the tests drive the real runner over the real + /// shared-serve machinery. The runner holds clones of the same states + /// the lane uses (the broadcast bus, the coordinator, the shared + /// manager). + async fn handoff_selfheal_rig( + backoff_initial_ms: u64, + backoff_max_ms: u64, + writer: crate::session_handoff::FlavorWriter, + ) -> ( + FreshOpencodeState, + Arc, + Arc, + tokio::sync::broadcast::Receiver, + Arc, + OpencodeServeManager, + ) { + let (tx, rx) = tokio::sync::broadcast::channel::(256); + let broadcast_tx = Arc::new(tx); + let auth_token = Arc::new("handoff-ep2r4-tok".to_string()); + let registry = Arc::new(freshell_ownership::RuntimeOwnershipRegistry::new()); + let terminal_registry = + freshell_terminal::TerminalRegistry::new().with_ownership(Arc::clone(®istry)); + let fresh_agent = FreshAgentState::new(Arc::clone(&auth_token), Arc::clone(&broadcast_tx)) + .with_ownership(Arc::clone(®istry)) + .with_terminal_registry(terminal_registry.clone()); + let exited = Arc::new(AtomicBool::new(false)); + let spawns = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(FakeHttp { + next_session: AtomicUsize::new(0), + }), + ports: Arc::new(FakeAllocator), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + idle_poll_interval: Duration::from_millis(20), + daemon_watch_interval: Duration::from_millis(10), + re_warm_backoff_initial_ms: backoff_initial_ms, + re_warm_backoff_max_ms: backoff_max_ms, + ..ServeConfig::default() + }; + let manager = OpencodeServeManager::new(deps, config); + manager + .ensure_started() + .await + .expect("healthy fake serve starts"); + fresh_agent.set_manager_for_test(manager.clone()).await; + let mut st = FreshOpencodeState::new(fresh_agent.clone()); + st.set_ownership(Arc::clone(®istry)); + // Production arms the daemon-loss watcher at the first opencode WS + // entry point; the handoff lane does not, so arm it here (the fork + // fixture's precedent) — the typed loss edge and the `Started` + // revival pass are the real machinery the finding is about. + st.ensure_daemon_loss_watcher().await; + let mut fresh_claude = crate::FreshClaudeState::new(Arc::clone(&broadcast_tx)); + fresh_claude.set_ownership(Arc::clone(®istry)); + let mut fresh_codex = crate::FreshCodexState::new( + Arc::clone(&auth_token), + Arc::clone(&broadcast_tx), + json!({ "freshAgent": { "enabled": true } }), + ); + fresh_codex.set_ownership(Arc::clone(®istry)); + let runner = crate::session_handoff::SessionHandoffRunner::new( + Arc::clone(&auth_token), + Arc::clone(&broadcast_tx), + Arc::clone(®istry), + terminal_registry, + fresh_codex, + fresh_claude, + st.clone(), + fresh_agent.clone(), + // A FreshAgent-target handoff never consults the CLI specs + // (validate_handoff_target's terminal arm only). + Arc::new(Vec::new()), + ) + .with_flavor_writer(Some(writer)); + (st, Arc::new(runner), registry, rx, exited, manager) + } + + /// A `switch` handoff moving `durable` to the freshopencode lane (the + /// finding's target arm — the under-ticket continuation). + fn freshopencode_handoff_request(durable: &str) -> crate::session_handoff::HandoffRequest { + crate::session_handoff::HandoffRequest { + action: crate::session_handoff::HandoffAction::Switch, + provider: "opencode".to_string(), + session_id: durable.to_string(), + target_kind: freshell_ownership::RuntimeOwnerKind::FreshAgent, + session_type: Some("freshopencode".to_string()), + mode: None, + cwd: Some(std::env::temp_dir().to_string_lossy().to_string()), + tab_id: None, + pane_id: None, + observed_epoch: None, + observed_generation: None, + device_id: Some("test-device-ep2r4".to_string()), + } + } + + /// fresheyes ep2-r4 Major (daemon loss in the handoff pre-commit + /// window): the under-ticket continuation installs its + /// generation-fenced bridge and runs its transitional rescue BEFORE the + /// runner's single `Live` commit; between the two the runner still + /// awaits the staged flavor write while the coordinator key remains + /// `Handoff`. A daemon loss there kills the fresh bridge with every + /// recovery trigger spent — the successor's one-shot `Started` revival + /// pass correctly skips the transitional owner, the `Live` commit emits + /// no new daemon signal, and the finished rescue is never rerun — so + /// the handoff answered plain success over an A-generation dead bridge. + /// + /// The interleaving, forced deterministically through the REAL paths: + /// the runner parks inside the flavor `stage` (its own awaited step + /// inside the Handoff window); while parked, daemon A dies and B + /// re-warms through the REAL loss/re-warm machinery (the fixture + /// proves B serves while the bridge stays dead — the revival pass + /// skipped the Handoff owner); releasing the park lets the commit + /// land, and the handoff must NOT leave a dead A-generation bridge: + /// the commit's re-check recovers it against B (a B-stamped live + /// bridge + exactly one idle snapshot push) or surfaces the typed + /// loss — never plain success over a dead bridge. + #[tokio::test(flavor = "multi_thread")] + async fn handoff_commit_recovers_daemon_loss_in_the_pre_commit_window() { + let writer = Arc::new(ParkingFlavorWriter::default()); + let (stage_entered_tx, stage_entered) = std::sync::mpsc::channel::<()>(); + let (stage_release_tx, stage_release_rx) = tokio::sync::oneshot::channel::<()>(); + writer + .stage_entered + .lock() + .expect("writer latch") + .replace(stage_entered_tx); + writer + .stage_release + .lock() + .expect("writer latch") + .replace(stage_release_rx); + let (st, runner, registry, mut rx, exited, manager) = handoff_selfheal_rig( + 5, + 50, + writer.clone() as crate::session_handoff::FlavorWriter, + ) + .await; + + // The prior: a committed TERMINAL owner (the terminal→freshopencode + // shape; the handoff's prior stop is the idempotent AlreadyGone + // against the nonexistent terminal row). + let durable = "ses_ep2r4_loss"; + commit_terminal_owner(®istry, durable).await; + + // The mover drives the finding's interleaving while the runner is + // parked inside the Handoff window. + let mover_state = st.clone(); + let mover_manager = manager.clone(); + let mover_exited = exited.clone(); + let mover = tokio::spawn(async move { + stage_entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the runner parks in the awaited flavor stage (inside the Handoff window)"); + // The under-ticket rescue has COMPLETED (start_target returned + // before the runner reached the flavor write) — the loss below + // lands in the finding's exact window. + mover_exited.store(true, Ordering::SeqCst); // daemon A dies + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + while mover_manager.base_url().await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "A's loss must take the running entry within the budget" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + // ...the take's sweep closed the target session's A-era bridge + // channel — the just-installed bridge is dead. + await_bridge_liveness( + &mover_state, + durable, + false, + "A's loss kills the target's just-installed bridge", + ) + .await; + mover_exited.store(false, Ordering::SeqCst); // B re-warms + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + while mover_manager.base_url().await.is_none() { + assert!( + tokio::time::Instant::now() < deadline, + "B's re-warm must respawn the daemon within the budget" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + // B's one-shot `Started` revival pass has run — and correctly + // SKIPPED the transitional (Handoff) owner: the bridge stays + // dead while B serves, with no recovery trigger left (the + // finding's unrecovered state, observed by the fixture before + // the commit proceeds). + tokio::time::sleep(Duration::from_millis(50)).await; + assert!( + !session_serve_bridge_alive(&mover_state, durable).await, + "fixture: the Started revival pass skipped the Handoff owner — \ + the dead A-era bridge with no recovery trigger (ep2-r4)" + ); + stage_release_tx.send(()).expect("release the parked stage"); + }); + + let resp = tokio::time::timeout( + Duration::from_secs(20), + runner + .spawn_handoff(freshopencode_handoff_request(durable)) + .completion, + ) + .await + .expect("the handoff completes within the budget") + .expect("the runner task lives to answer"); + mover.await.expect("the mover completed the interleaving"); + + // The operation's answer: SUCCESS (the recovery leg — the happy + // contract, not the typed-loss leg). + assert_eq!( + resp["ok"], + json!(true), + "the handoff answers success: {resp:?}" + ); + await_freshagent_live(®istry, durable).await; + + // THE BRIDGE: the handoff must not leave a dead A-generation + // bridge. RED on HEAD: plain success over the dead bridge. + assert!( + session_serve_bridge_alive(&st, durable).await, + "the handoff must not answer success over a dead A-generation \ + bridge — the pre-commit loss left every recovery trigger spent \ + and the commit's re-check is the only one left (ep2-r4)" + ); + // The recovered bridge is stamped to the CURRENT (successor) daemon + // generation — not the dead A era. + { + let session_arc = st + .sessions + .lock() + .await + .get(durable) + .cloned() + .expect("the handoff target session is registered"); + let session = session_arc.lock().await; + assert_eq!( + session.serve_bridge_daemon, + manager.ownership_id().await, + "the recovered bridge is stamped to the successor daemon B" + ); + } + + // The frames: the commit's re-check pushed EXACTLY ONE recovery + // snapshot for the actually-restarted bridge (the revival pass + // pushed none — it skipped the Handoff owner), exactly ONE typed + // loss edge reached the materialized session, and NO chime + // anywhere (a daemon loss is never a positive completion). + let mut all = frames_until(&mut rx, |f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable) + }) + .await; + all.extend(drain_frames(&mut rx)); + let pushes = all + .iter() + .filter(|f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable) + }) + .count(); + assert_eq!( + pushes, 1, + "exactly ONE recovery snapshot for the restarted bridge: {all:?}" + ); + let edges = all + .iter() + .filter(|f| { + f["type"] == "freshAgent.event" + && f["event"]["type"] == "freshAgent.error" + && f["event"]["code"] == "OPENCODE_DAEMON_LOST" + && f["sessionId"].as_str() == Some(durable) + }) + .count(); + assert_eq!( + edges, 1, + "exactly ONE typed loss edge for the target session: {all:?}" + ); + assert!( + all.iter() + .all(|f| f["event"]["type"] != "freshAgent.turn.complete"), + "a daemon loss is never a positive completion — no chime: {all:?}" + ); + assert!( + manager.base_url().await.is_some(), + "fixture: the successor daemon B serves" + ); + } + + /// The ep2-r4 REFUSAL leg (the ep2-r3 discipline applied to the + /// runner's commit): a kill completing between the runner's `Live` + /// commit and its answer must surface the typed ownership-changed + /// failure — never plain success over the retired session. The + /// deterministic hold is the runner's own awaited STAGED FLAVOR + /// COMMIT (post-commit, pre-answer): the mover completes a REAL + /// `freshAgent.kill` while the runner is parked there; the commit's + /// re-check then refuses (the kill's teardown owns the cleanup — no + /// bridge install, no snapshot push, no re-own). RED on HEAD: the + /// runner answered ok:true with nothing re-checking after the commit. + #[tokio::test(flavor = "multi_thread")] + async fn a_kill_between_the_handoff_commit_and_its_answer_fails_the_handoff_typed() { + let writer = Arc::new(ParkingFlavorWriter::default()); + let (commit_entered_tx, commit_entered) = std::sync::mpsc::channel::<()>(); + let (commit_release_tx, commit_release_rx) = tokio::sync::oneshot::channel::<()>(); + writer + .commit_entered + .lock() + .expect("writer latch") + .replace(commit_entered_tx); + writer + .commit_release + .lock() + .expect("writer latch") + .replace(commit_release_rx); + // The daemon never dies here (the KILL is the mover) — a far re-warm + // budget keeps the background machinery out of the window. + let (st, runner, registry, mut rx, exited, manager) = handoff_selfheal_rig( + 60_000, + 120_000, + writer.clone() as crate::session_handoff::FlavorWriter, + ) + .await; + + let durable = "ses_ep2r4_refused"; + commit_terminal_owner(®istry, durable).await; + + let mover_state = st.clone(); + let mover_registry = registry.clone(); + let mover = tokio::spawn(async move { + commit_entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the runner parks in the staged flavor commit (after the Live commit)"); + // Fixture honesty: the handoff's Live{FreshAgent} commit has + // landed (the parked staged commit is strictly post-commit). + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + loop { + if matches!( + mover_registry.observe("opencode", durable).state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ) { + break; + } + assert!( + tokio::time::Instant::now() < deadline, + "the handoff's Live{{FreshAgent}} commit never landed" + ); + tokio::time::sleep(Duration::from_millis(10)).await; + } + // The concurrent kill, driven through the REAL path to + // completion — the mover's teardown owns the cleanup. + mover_state + .handle_kill(FreshAgentKill { + provider: AgentProvider::Opencode, + session_id: durable.to_string(), + session_type: SessionType::Freshopencode, + cwd: None, + observed_epoch: None, + observed_generation: None, + }) + .await; + assert!( + !mover_state.has_live_session(durable).await, + "fixture: the kill removed every map key for the target session" + ); + assert!( + !matches!( + mover_registry.observe("opencode", durable).state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ), + "fixture: the coordinator no longer holds the session Live{{FreshAgent}}" + ); + commit_release_tx + .send(()) + .expect("release the staged commit"); + }); + + let resp = tokio::time::timeout( + Duration::from_secs(20), + runner + .spawn_handoff(freshopencode_handoff_request(durable)) + .completion, + ) + .await + .expect("the handoff completes within the budget") + .expect("the runner task lives to answer"); + mover.await.expect("the mover completed the interleaving"); + + // THE outcome: the typed ownership-changed failure — never plain + // success over the retired session (RED on HEAD: ok:true). + assert_ne!( + resp["ok"], + json!(true), + "the handoff must not answer success over a session a concurrent \ + kill retired between the commit and the answer (ep2-r4): {resp:?}" + ); + assert_eq!( + resp["error"]["code"], + json!("STALE_GENERATION"), + "the refusal is the typed ownership-changed failure: {resp:?}" + ); + assert_eq!( + resp["error"]["retryable"], + json!(true), + "the refusal is the canonical retryable race outcome: {resp:?}" + ); + + // NO resurrection: the refusal never installs a bridge and never + // pushes a snapshot — the kill's teardown owns the cleanup. + assert!( + !st.has_live_session(durable).await, + "the refusal never resurrected the retired session" + ); + manager.dispatch_event( + freshell_opencode::events::parse_serve_event(&json!({ + "type": "session.idle", + "properties": { "sessionID": durable } + })) + .expect("parseable serve event"), + ); + tokio::time::sleep(Duration::from_millis(100)).await; + let all = drain_frames(&mut rx); + assert!( + all.iter().all( + |f| !(is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable)) + ), + "no recovery-snapshot push for the retired session: {all:?}" + ); + assert!( + !all.iter().any( + |f| f["type"] == "freshAgent.event" && f["sessionId"].as_str() == Some(durable) + ), + "no resurrected bridge frame for the killed session: {all:?}" + ); + assert!( + all.iter() + .all(|f| f["event"]["type"] != "freshAgent.turn.complete"), + "no chime anywhere: {all:?}" + ); + // The kill's coordinator commit stands — nothing re-owns the + // retired session. + assert!( + !matches!( + registry.observe("opencode", durable).state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ), + "the kill's coordinator commit stands — the re-check never re-owns the session" + ); + let _ = (exited, manager); + } + + // ── fresheyes ep2-r5: the rescue is bound to the committed session + // instance and ownership fence, never to a same-key replacement ── + + /// fresheyes ep2-r5 Major (the rescue adopts a same-key replacement + /// for the committed session): the post-commit rescue verified + /// whichever session currently occupies the durable id — not the + /// session INSTANCE and ownership GENERATION the caller committed. A + /// kill that removes the committed session followed by another + /// freshopencode resume under the SAME durable id leaves a brand-new + /// session object under the key, committed Live at a NEWER + /// generation; the rescue's by-id lookup obtained the replacement + /// `Arc`, the kind-only ownership check accepted the replacement's + /// era, and the rescue answered Healthy — after which the resume + /// handed back its own removed-and-killed `Arc` for the attach + /// caller to restart a bridge on. + /// + /// The interleaving, forced deterministically through the + /// `cfg(test)` PRE-LOOKUP rescue stall: the original resume's + /// post-commit rescue fires `entered` BEFORE its map re-lookup and + /// parks; while held, a REAL `freshAgent.kill` of the committed + /// session COMPLETES and a REAL replacement resume installs a + /// brand-new instance under the same key (asserted: a strictly newer + /// Live generation); only then is the rescue released — its + /// re-lookup now finds the REPLACEMENT. The resume must REFUSE — + /// the typed ownership-changed error, never `Ok` over the retired + /// instance — and the refusal must leave the replacement untouched + /// (its registration, bridge, and era stand; the mover's own + /// lifecycle owns the key now). Pre-fix, the rescue adopted the + /// same-kind replacement and the resume answered `Ok` with the + /// retired `Arc`. + #[tokio::test(flavor = "multi_thread")] + async fn a_replacement_resume_under_the_killed_key_fails_the_original_resume_typed() { + // The selfheal fixture answers the cold resumes' GET /session/:id; + // the daemon never dies (the KILL+REPLACE is the mover). The + // coordinator IS wired — the committed-fence leg is half the + // finding, and the committed-instance leg must hold either way. + let (mut st, _rx, exited, spawns, _manager) = selfheal_state(60_000, 120_000).await; + let registry = Arc::new(freshell_ownership::RuntimeOwnershipRegistry::new()); + st.set_ownership(Arc::clone(®istry)); + let fake = Arc::new(crate::identity_sink::FakeIdentitySink::default()); + st.set_identity_sink(fake.clone()); + + let durable = "ses_ep2r5_resume"; + // Park the ORIGINAL resume's post-commit rescue BEFORE its map + // re-lookup — the exact window the finding names (the mover + // completes while the rescue is in flight). + let stall = st.arm_rescue_lookup_stall(durable); + let st2 = st.clone(); + let original = tokio::spawn(async move { + st2.resume_durable_session(durable, None, None, None, None) + .await + }); + + stall + .entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the rescue parks before its map re-lookup"); + + // Fixture honesty: the parked rescue is strictly POST-COMMIT — + // the original resume's own Live commit has landed; record its + // era (the committed fence the rescue must still find). + let committed_generation = match registry.observe("opencode", durable) { + freshell_ownership::OwnershipSnapshot { + generation, + state: freshell_ownership::OwnershipState::Live { owner, .. }, + .. + } if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent => generation, + snap => panic!("the original resume committed Live: {snap:?}"), + }; + + // The mover, part 1: a REAL kill of the committed session, driven + // through the REAL path to completion (the durable close, every + // map key, the killed flag, the bridge abort, Stopping→Vacant). + st.handle_kill(FreshAgentKill { + provider: AgentProvider::Opencode, + session_id: durable.to_string(), + session_type: SessionType::Freshopencode, + cwd: None, + observed_epoch: None, + observed_generation: None, + }) + .await; + assert!( + !st.has_live_session(durable).await, + "fixture: the kill removed the committed session's map key" + ); + assert!( + !matches!( + registry.observe("opencode", durable).state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ), + "fixture: the kill vacated the coordinator key" + ); + + // The mover, part 2: a REAL replacement resume installs a + // BRAND-NEW instance under the SAME durable id and commits it + // Live at a strictly newer generation — the finding's same-key + // replacement. + let replacement = match st + .resume_durable_session(durable, None, None, None, None) + .await + { + Ok(arc) => arc, + Err(ResumeOpencodeError::Manager(err)) => panic!( + "fixture: the replacement resume commits a fresh instance under the vacated key: {err:?}" + ), + Err(ResumeOpencodeError::NotFound) => { + panic!("fixture: the replacement resume found no serve row") + } + Err(ResumeOpencodeError::Reserved) => { + panic!("fixture: the replacement resume hit a reserved key") + } + }; + assert!( + st.has_live_session(durable).await, + "fixture: the replacement session is registered" + ); + let replacement_generation = match registry.observe("opencode", durable) { + freshell_ownership::OwnershipSnapshot { + generation, + state: freshell_ownership::OwnershipState::Live { owner, .. }, + .. + } if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent => generation, + snap => panic!("the replacement resume committed Live: {snap:?}"), + }; + assert!( + replacement_generation > committed_generation, + "the replacement's era is a strictly newer generation than the committed fence" + ); + // The retired-vs-replacement identity is the RESCUE's job, not the + // fixture's; the replacement's own Arc is only fixture honesty. + let _ = replacement; + + // Release the parked rescue: its re-lookup now finds the + // REPLACEMENT under the key. Only the committed-instance binding + // and the committed-fence check can refuse it. + stall.release.send(()).expect("release the parked rescue"); + let out = original.await.expect("the original resume settles"); + + // THE outcome: the typed ownership-changed refusal — never Ok + // over the retired original instance (pre-fix: the rescue + // adopted the same-kind replacement, answered Healthy, and the + // resume returned its own removed-and-killed `Arc`). + match out { + Err(ResumeOpencodeError::Manager(freshell_opencode::ServeError::Transport(msg))) => { + assert!( + msg.contains("ownership changed during resume"), + "the typed refusal names the ownership change: {msg}" + ); + } + Err(ResumeOpencodeError::Manager(err)) => { + panic!("the refusal must be the ownership-changed Transport error, got {err:?}") + } + Err(_) => { + panic!("the refusal must be a Manager error (NotFound/Reserved are wrong here)") + } + Ok(_) => panic!( + "the resume returned success over a session a concurrent kill retired and \ + a same-key replacement superseded (ep2-r5)" + ), + } + + // The refusal never touched the replacement: its registration, + // live bridge, and committed era stand — the mover's lifecycle + // owns the key now, and the retired original's rescue installed + // nothing for it (the refusal returns before any restart; the + // kill's teardown owns the original's cleanup). + assert!( + st.has_live_session(durable).await, + "the replacement's registration stands" + ); + assert!( + session_serve_bridge_alive(&st, durable).await, + "the replacement's bridge stands — the refusal never uninstalled it" + ); + assert_eq!( + registry.observe("opencode", durable).generation, + replacement_generation, + "the refusal never re-owned or moved the replacement's era" + ); + let _ = (exited, spawns, fake); + } + + /// The ep2-r5 interleave on the HANDOFF runner's post-commit re-check + /// (the ep2-r4 tail's caller leg): the runner committed its + /// freshopencode target and parks in its own awaited STAGED FLAVOR + /// COMMIT (the ep2-r4 latch, strictly post-commit); while held, a + /// REAL kill retires the committed target and a REAL replacement + /// resume installs a brand-new instance under the SAME durable id + /// (Live at a strictly newer generation). The re-check's by-id + /// lookup then finds the replacement — the runner must surface the + /// typed ownership-changed failure, never answer success carrying + /// its obsolete generation, and never touch the replacement (the + /// mover's lifecycle owns the key). Pre-fix, the rescue adopted the + /// same-kind replacement and the handoff answered ok:true. + #[tokio::test(flavor = "multi_thread")] + async fn a_replacement_resume_under_the_killed_key_fails_the_handoff_typed() { + let writer = Arc::new(ParkingFlavorWriter::default()); + let (commit_entered_tx, commit_entered) = std::sync::mpsc::channel::<()>(); + let (commit_release_tx, commit_release_rx) = tokio::sync::oneshot::channel::<()>(); + writer + .commit_entered + .lock() + .expect("writer latch") + .replace(commit_entered_tx); + writer + .commit_release + .lock() + .expect("writer latch") + .replace(commit_release_rx); + // The daemon never dies here (the KILL+REPLACE is the mover) — a + // far re-warm budget keeps the background machinery out of the + // window. + let (st, runner, registry, _rx, exited, manager) = handoff_selfheal_rig( + 60_000, + 120_000, + writer.clone() as crate::session_handoff::FlavorWriter, + ) + .await; + + let durable = "ses_ep2r5_handoff"; + commit_terminal_owner(®istry, durable).await; + + let mover_state = st.clone(); + let mover_registry = registry.clone(); + let mover = tokio::spawn(async move { + commit_entered + .recv_timeout(std::time::Duration::from_secs(15)) + .expect("the runner parks in the staged flavor commit (after the Live commit)"); + // Fixture honesty: the handoff's Live{FreshAgent} commit has + // landed (the parked staged commit is strictly + // post-commit); record the committed era. + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + let committed_generation = loop { + match mover_registry.observe("opencode", durable) { + freshell_ownership::OwnershipSnapshot { + generation, + state: freshell_ownership::OwnershipState::Live { owner, .. }, + .. + } if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent => { + break generation + } + _ => { + assert!( + tokio::time::Instant::now() < deadline, + "the handoff's Live{{FreshAgent}} commit never landed" + ); + tokio::time::sleep(Duration::from_millis(10)).await; + } + } + }; + // The mover, part 1: a REAL kill of the committed target, to + // completion — the mover's teardown owns the cleanup. + mover_state + .handle_kill(FreshAgentKill { + provider: AgentProvider::Opencode, + session_id: durable.to_string(), + session_type: SessionType::Freshopencode, + cwd: None, + observed_epoch: None, + observed_generation: None, + }) + .await; + assert!( + !mover_state.has_live_session(durable).await, + "fixture: the kill removed every map key for the committed target" + ); + assert!( + !matches!( + mover_registry.observe("opencode", durable).state, + freshell_ownership::OwnershipState::Live { owner, .. } + if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent + ), + "fixture: the kill vacated the coordinator key" + ); + // The mover, part 2: a REAL replacement resume installs a + // BRAND-NEW instance under the SAME durable id, committed + // Live at a strictly newer generation. + let replacement = match mover_state + .resume_durable_session(durable, None, None, None, None) + .await + { + Ok(arc) => arc, + Err(ResumeOpencodeError::Manager(err)) => panic!( + "fixture: the replacement resume commits a fresh instance under the vacated key: {err:?}" + ), + Err(ResumeOpencodeError::NotFound) => { + panic!("fixture: the replacement resume found no serve row") + } + Err(ResumeOpencodeError::Reserved) => { + panic!("fixture: the replacement resume hit a reserved key") + } + }; + let replacement_generation = match mover_registry.observe("opencode", durable) { + freshell_ownership::OwnershipSnapshot { + generation, + state: freshell_ownership::OwnershipState::Live { owner, .. }, + .. + } if owner.kind == freshell_ownership::RuntimeOwnerKind::FreshAgent => generation, + snap => panic!("the replacement resume committed Live: {snap:?}"), + }; + assert!( + replacement_generation > committed_generation, + "the replacement's era is a strictly newer generation than the handoff's commit" + ); + let _ = replacement; + commit_release_tx + .send(()) + .expect("release the staged commit"); + (committed_generation, replacement_generation) + }); + + let resp = tokio::time::timeout( + Duration::from_secs(20), + runner + .spawn_handoff(freshopencode_handoff_request(durable)) + .completion, + ) + .await + .expect("the handoff completes within the budget") + .expect("the runner task lives to answer"); + let (committed_generation, replacement_generation) = + mover.await.expect("the mover completed the interleaving"); + assert!( + replacement_generation > committed_generation, + "fixture: the replacement superseded the handoff's committed era" + ); + + // THE outcome: the typed ownership-changed failure — never plain + // success carrying the runner's obsolete generation (pre-fix: + // the rescue adopted the same-key replacement and the handoff + // answered ok:true). + assert_ne!( + resp["ok"], + json!(true), + "the handoff must not answer success over a target a concurrent kill retired \ + and a same-key replacement superseded (ep2-r5): {resp:?}" + ); + assert_eq!( + resp["error"]["code"], + json!("STALE_GENERATION"), + "the refusal is the typed ownership-changed failure: {resp:?}" + ); + assert_eq!( + resp["error"]["retryable"], + json!(true), + "the refusal is the canonical retryable race outcome: {resp:?}" + ); + + // The refusal never touched the replacement: its registration, + // live bridge, and committed era stand — the mover's lifecycle + // owns the key now, and the retired target's re-check installed + // nothing (the refusal returns before any restart; the kill's + // teardown owns the original's cleanup). + assert!( + st.has_live_session(durable).await, + "the replacement's registration stands" + ); + assert!( + session_serve_bridge_alive(&st, durable).await, + "the replacement's bridge stands — the refusal never uninstalled it" + ); + assert_eq!( + registry.observe("opencode", durable).generation, + replacement_generation, + "the refusal never re-owned or moved the replacement's era" + ); + let _ = (exited, manager); + } + + /// LB-05 (falsified → redesign): in the incident, the pane's fenced + /// attach was exercised 3× against the dead shared daemon and recovered + /// NOTHING, because the attach tail only re-subscribed the bridge — + /// nothing respawns the daemon for a map-hit. The fenced attach must be + /// a REAL recovery verb: `ensure_started` BEFORE the bridge restart, so + /// the map-hit attach respawns the daemon and re-bridges (mirroring + /// `resume_durable_session`'s map-miss behavior). + #[tokio::test] + async fn map_hit_fenced_attach_respawns_the_daemon_and_rebridges() { + // Backoff far beyond the test window: the background re-warm must NOT + // be the respawn this test credits — the ATTACH's own + // `ensure_started` is the recovery under proof. + let (mut st, mut rx, exited, spawns, manager) = selfheal_state(60_000, 120_000).await; + let registry = Arc::new(freshell_ownership::RuntimeOwnershipRegistry::new()); + st.set_ownership(Arc::clone(®istry)); + + let (_placeholder, durable) = + materialized_selfheal_session(&st, &mut rx, "req-attach-recover").await; + await_freshagent_live(®istry, &durable).await; + // The observed runtime-owner pair the fenced attach carries. + let before = registry.observe("opencode", &durable); + + // The shared daemon dies; the session row PERSISTS (the map-hit + // shape). Bounded wait for the manager's watcher to clear the entry. + exited.store(true, Ordering::SeqCst); + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + loop { + if manager.base_url().await.is_none() { + break; + } + assert!( + tokio::time::Instant::now() < deadline, + "the daemon's running entry must clear after the exit" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + let _ = drain_frames(&mut rx); // the loss edge (already fanned out) etc. + let spawns_before = spawns.load(Ordering::SeqCst); + // The respawn can now succeed (the flag-driven fixture's death flag + // clears, exactly like the crash-loop fixture in Task 3's tests). + exited.store(false, Ordering::SeqCst); + + // THE MAP-HIT FENCED ATTACH — the incident's three wasted attempts, + // now the documented recovery verb. + st.handle_attach(FreshAgentAttach { + provider: AgentProvider::Opencode, + session_id: durable.clone(), + session_type: SessionType::Freshopencode, + cwd: None, + observed_epoch: Some(before.epoch), + observed_generation: Some(before.generation), + resume_session_id: None, + session_ref: None, + }) + .await; + + assert!( + spawns.load(Ordering::SeqCst) > spawns_before, + "a map-hit attach against a daemon-absent manager must respawn the \ + shared daemon (observed {} spawns; {} before the attach)", + spawns.load(Ordering::SeqCst), + spawns_before + ); + assert!( + session_serve_bridge_alive(&st, &durable).await, + "the attach must re-bridge the session" + ); + let frames = frames_until(&mut rx, |f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(durable.as_str()) + }) + .await; + assert!( + !frames.is_empty(), + "the attach tail's snapshot push must arrive: {frames:?}" + ); + } + + /// Plan-review round 3 (the ownership-coordinator gate): a session + /// killed/retired or handed to a terminal owner between the loss and the + /// respawn must NOT be revived — the revival pass re-looks-up the map at + /// revival time (a retired session's keys are gone) and observes the + /// CANONICAL ownership state fresh (a terminal owner or any lifecycle + /// transition owns the window; only Live{FreshAgent} sessions — this + /// runtime's own — revive). + #[tokio::test] + async fn revival_skips_sessions_handed_off_or_removed_after_the_loss() { + let (mut st, mut rx, exited, _spawns, _manager) = selfheal_state(5, 50).await; + let registry = Arc::new(freshell_ownership::RuntimeOwnershipRegistry::new()); + st.set_ownership(Arc::clone(®istry)); + + let (_p_keeps, keeps) = materialized_selfheal_session(&st, &mut rx, "req-keeps").await; + let (p_gone, gone) = materialized_selfheal_session(&st, &mut rx, "req-gone").await; + let (_p_term, term) = materialized_selfheal_session(&st, &mut rx, "req-term").await; + for durable in [&keeps, &gone, &term] { + await_freshagent_live(®istry, durable).await; + } + + // While the daemon is down (modeled pre-loss here), session B is + // retired from the map AND its key completes a terminal handoff (the + // concurrent-handoff shape); session C STAYS mapped but its key is + // terminal-owned (the committed-handoff window). + { + let mut map = st.sessions.lock().await; + map.remove(&p_gone); + map.remove(&gone); + } + commit_terminal_owner(®istry, &gone).await; + commit_terminal_owner(®istry, &term).await; + let _ = drain_frames(&mut rx); + + exited.store(true, Ordering::SeqCst); // the daemon dies (bridges die with it) + // Deterministic anchor: the fan-out reaches the LAST mapped session + // (BTreeSet order — keeps, then term), so term's edge proves the + // whole fan-out ran. + let loss_frames = frames_until(&mut rx, |f| { + f["type"] == "freshAgent.event" + && f["event"]["code"] == "OPENCODE_DAEMON_LOST" + && f["sessionId"].as_str() == Some(term.as_str()) + }) + .await; + + exited.store(false, Ordering::SeqCst); // the re-warm's respawn now succeeds + + // `Started` → the revival pass: keeps IS revived... + let revive_frames = frames_until(&mut rx, |f| { + is_event(f, "freshAgent.session.snapshot", Some("idle")) + && f["sessionId"].as_str() == Some(keeps.as_str()) + }) + .await; + + // ...and the pass has had every opportunity to (wrongly) touch the + // retired and terminal-owned sessions — settle, then audit the whole + // post-loss window. + tokio::time::sleep(Duration::from_millis(100)).await; + let mut all = loss_frames; + all.extend(revive_frames); + all.extend(drain_frames(&mut rx)); + + assert!( + session_serve_bridge_alive(&st, &keeps).await, + "the healthy fresh-agent session revives" + ); + assert!( + !session_serve_bridge_alive(&st, &gone).await, + "the retired session must NOT be revived" + ); + assert!( + !session_serve_bridge_alive(&st, &term).await, + "the terminal-owned session must NOT be revived" + ); + let gone_snapshots = all + .iter() + .filter(|f| { + is_event(f, "freshAgent.session.snapshot", None) + && f["sessionId"].as_str() == Some(gone.as_str()) + }) + .count(); + assert_eq!( + gone_snapshots, 0, + "no snapshot push for the retired session: {all:?}" + ); + let term_snapshots = all + .iter() + .filter(|f| { + is_event(f, "freshAgent.session.snapshot", None) + && f["sessionId"].as_str() == Some(term.as_str()) + }) + .count(); + assert_eq!( + term_snapshots, 0, + "no snapshot push for the terminal-owned session: {all:?}" + ); + } } diff --git a/crates/freshell-freshagent/src/session_handoff.rs b/crates/freshell-freshagent/src/session_handoff.rs index 5f98f9e93..7f556c5d1 100644 --- a/crates/freshell-freshagent/src/session_handoff.rs +++ b/crates/freshell-freshagent/src/session_handoff.rs @@ -1398,7 +1398,7 @@ impl SessionHandoffRunner { .start_target(&req, &operation_id, generation, target_spawn_watch) .await { - Ok(owner) => { + Ok((owner, opencode_committed)) => { // The cancellation-safety window: a spawned-but-uncommitted // target is reaped by the guard if the runner is aborted // before the commit (round-1 review). The precise identity @@ -1633,6 +1633,106 @@ impl SessionHandoffRunner { "ownership.handoff.done", TransitionLevel::Info, ); + // fresheyes ep2-r4 Major (daemon loss in the + // handoff pre-commit window): the under-ticket + // continuation (`start_target` → + // `opencode_resume_for_handoff`) installs its + // generation-fenced bridge and runs its own + // transitional rescue BEFORE this runner's single + // `Live` commit — and between the two, this + // runner can still await + // `current_flavor()`/`stage()` while the + // coordinator key remains `Handoff`. A daemon + // loss in that interval kills the fresh bridge + // with every recovery trigger spent: the + // successor's `Started` revival pass deliberately + // skips the transitional owner (the + // foreign-transition rule), this commit emits no + // new daemon signal, and the finished rescue is + // never rerun — the handoff would answer plain + // success over an A-generation dead bridge. The + // ESTABLISHED post-commit rescue pattern (the + // fork/resume tails) closes the window from the + // commit's own side: re-run the SAME + // ownership-coordinator-gated guarded-restart + // seam for the freshopencode target. + // `own_lifecycle_window = false`: this commit IS + // the window's end, so the seam observes the + // runner's own `Live{FreshAgent}` and arms the + // adopt guard across the restart (the revival + // pass's exact discipline). A bridge alive + // against the CURRENT daemon is the quiet no-op; + // a dead/lost-generation bridge is restarted + // against the successor and the client gets its + // one recovery snapshot; a REFUSAL — the session + // was killed or re-transitioned between the + // commit and this tail, the mover's teardown + // owning the cleanup — surfaces the typed + // ownership-changed failure, never success over + // the retired session. A bounded respawn failure + // stays WARN-only (the seam's ep2-r2 contract). + // + // fresheyes ep2-r5: the re-check is bound to the + // identity THIS runner committed — the under- + // ticket continuation's session instance (carried + // out of `start_target`) and this commit's own + // `(epoch, generation)` fence. A same-key + // REPLACEMENT (the target killed and another + // resume's brand-new session installed under the + // same durable id during the awaited post-commit + // work) can never satisfy the instance binding or + // the fence equality, so the runner surfaces the + // typed failure instead of answering success + // carrying its obsolete generation. + if req.target_kind == RuntimeOwnerKind::FreshAgent + && req.session_type.as_deref() == Some("freshopencode") + && match opencode_committed.as_ref() { + Some(committed) => matches!( + self.fresh_opencode + .rescue_transitional_bridge_after_commit( + &req.session_id, + false, + committed, + Some(ObservedFence { + epoch: self.ownership.boot_epoch(), + generation, + }), + ) + .await, + crate::opencode_ws::TransitionalBridgeRescue::Refused + ), + // Unreachable for a freshopencode target + // (the under-ticket resume hands its + // committed instance out on every Ok) — + // fail CLOSED: an instance the runner + // cannot verify must never answer + // success. + None => true, + } + { + tracing::warn!(target: "freshell_freshagent::opencode", + operation_id = %operation_id, + provider = %req.provider, session_id = %req.session_id, + generation, + "freshagent.opencode.handoff_bridge_recheck_refused: the \ + freshopencode target committed but the session was retired \ + or re-transitioned before the handoff answered — the \ + mover's teardown owns the cleanup; the handoff surfaces \ + the typed ownership-changed failure (ep2-r4)" + ); + let current = self + .ownership + .observe(&req.provider, &req.session_id) + .generation; + return typed_failure( + "STALE_GENERATION", + "ownership changed during handoff; the freshopencode target \ + committed but the session was retired or re-transitioned \ + before the handoff could answer — refresh and retry", + true, + current, + ); + } json!({ "ok": true, "operationId": operation_id, @@ -3840,13 +3940,26 @@ impl SessionHandoffRunner { /// that does not serve the canonical `(provider, req.session_id)` is a /// `TARGET_SPAWN_FAILED` — never accept a respawned-new-thread runtime /// as handoff success, never mint a new session id. + /// + /// ep2-r5: the Ok payload additionally carries the freshopencode + /// target's committed session INSTANCE + /// ([`crate::opencode_ws::OpencodeSessionHandle`]) — `None` for every + /// other target kind — so the runner's post-commit rescue re-check can + /// verify the exact `Arc` its under-ticket continuation installed + /// (a same-key replacement can never satisfy it). async fn start_target( &self, req: &HandoffRequest, operation_id: &str, generation: u64, target_spawn_watch: Option, - ) -> Result { + ) -> Result< + ( + OwnerIdentity, + Option, + ), + (String, String), + > { // b8ke ext r23 F2: the attempt marker — start_target was ENTERED // (the spawn/resume is being attempted on this call). The // spawn-failure tests assert this event so a regression that @@ -3964,55 +4077,55 @@ impl SessionHandoffRunner { if let Some(hooks) = self.test_hooks.as_ref() { hooks.record("TargetStarted"); } - Ok(owner) + Ok((owner, None)) } RuntimeOwnerKind::FreshAgent => match req.session_type.as_deref() { - Some("freshcodex") | None => { - self.fresh_codex - .resume_for_handoff( - &req.session_id, - req.cwd.as_deref(), - operation_id, - generation, - ) - .await - } - Some("freshopencode") => { - self.fresh_opencode - .opencode_resume_for_handoff( - &req.session_id, - req.cwd.as_deref(), - operation_id, - generation, - ) - .await - } + Some("freshcodex") | None => self + .fresh_codex + .resume_for_handoff( + &req.session_id, + req.cwd.as_deref(), + operation_id, + generation, + ) + .await + .map(|owner| (owner, None)), + Some("freshopencode") => self + .fresh_opencode + .opencode_resume_for_handoff( + &req.session_id, + req.cwd.as_deref(), + operation_id, + generation, + ) + .await + .map(|(owner, committed)| (owner, Some(committed))), // Round-2 review: BOTH claude-lane flavors — the flavor is a // param (claude.rs), so a kilroy session resumes as kilroy // (the created/owner frames keep the flavor) and a freshclaude // session as freshclaude. Never map by provider alone. - Some("freshclaude") => { - self.fresh_claude - .resume_for_handoff( - "freshclaude", - &req.session_id, - req.cwd.as_deref(), - operation_id, - generation, - ) - .await - } - Some("kilroy") => { - self.fresh_claude - .resume_for_handoff( - "kilroy", - &req.session_id, - req.cwd.as_deref(), - operation_id, - generation, - ) - .await - } + Some("freshclaude") => self + .fresh_claude + .resume_for_handoff( + "freshclaude", + &req.session_id, + req.cwd.as_deref(), + operation_id, + generation, + ) + .await + .map(|owner| (owner, None)), + Some("kilroy") => self + .fresh_claude + .resume_for_handoff( + "kilroy", + &req.session_id, + req.cwd.as_deref(), + operation_id, + generation, + ) + .await + .map(|owner| (owner, None)), Some(other) => Err(("unsupported sessionType".to_string(), other.to_string())), }, } diff --git a/crates/freshell-opencode/src/lib.rs b/crates/freshell-opencode/src/lib.rs index 0145dc8e9..0de7946a4 100644 --- a/crates/freshell-opencode/src/lib.rs +++ b/crates/freshell-opencode/src/lib.rs @@ -50,8 +50,8 @@ pub use model::{ FRESHOPENCODE_DEFAULT_EFFORT, }; pub use serve::{ - build_prompt_body, display_error_chain, is_healthy_response, CreatedSession, Endpoint, - EventSource, EventStreamHandle, ForkedSession, OpencodeServeManager, PortAllocator, + build_prompt_body, display_error_chain, is_healthy_response, CreatedSession, DaemonSignal, + Endpoint, EventSource, EventStreamHandle, ForkedSession, OpencodeServeManager, PortAllocator, ProcessSpawner, Route, ServeConfig, ServeDeps, ServeError, ServeHttp, ServeHttpError, ServeHttpRequest, ServeHttpResponse, ServeProcess, SessionSignal, SpawnRequest, OPENCODE_SIDECAR_OWNERSHIP_ENV, diff --git a/crates/freshell-opencode/src/serve.rs b/crates/freshell-opencode/src/serve.rs index 9a86e8860..445e92851 100644 --- a/crates/freshell-opencode/src/serve.rs +++ b/crates/freshell-opencode/src/serve.rs @@ -23,7 +23,7 @@ use std::collections::HashMap; use std::future::Future; use std::pin::Pin; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; @@ -581,6 +581,17 @@ pub struct ServeConfig { /// 600 s opencode turn budget that also bounds the compact's await-idle /// tail (`opencode_ws.rs`'s `DEFAULT_TURN_TIMEOUT`). pub compact_timeout: Duration, + /// Daemon exit-watcher poll cadence (`Task 3`): how often the running + /// daemon's `exited()` is consulted between request traffic. + pub daemon_watch_interval: Duration, + /// Re-warm backoff: the initial delay before the first respawn attempt + /// after a daemon loss, doubling per failed attempt, capped at + /// [`ServeConfig::re_warm_backoff_max_ms`]. + pub re_warm_backoff_initial_ms: u64, + /// Re-warm backoff ceiling: the escalation stops here (a crash-looping + /// daemon retries at this interval forever — it self-heals when e.g. + /// disk frees). + pub re_warm_backoff_max_ms: u64, } impl Default for ServeConfig { @@ -598,6 +609,9 @@ impl Default for ServeConfig { required_idle_status_polls: 2, request_timeout: Duration::from_millis(30_000), compact_timeout: Duration::from_millis(600_000), + daemon_watch_interval: Duration::from_millis(1_000), + re_warm_backoff_initial_ms: 2_000, + re_warm_backoff_max_ms: 60_000, } } } @@ -640,10 +654,62 @@ pub enum SessionSignal { const SESSION_CHANNEL_CAPACITY: usize = 256; +/// The daemon-level channel capacity for [`DaemonSignal`] broadcasts. +const DAEMON_CHANNEL_CAPACITY: usize = 16; + +/// A daemon-lifecycle edge broadcast by the manager (the client-facing +/// runtime's Task-4 revival design consumes this): `Lost` when the shared +/// daemon is gone (a requested discard with its reason, or an unrequested +/// process exit), `Started` on every successful COLD start (not on the +/// fast-path return of an already-running daemon). +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum DaemonSignal { + /// The running daemon is gone. `reason` is the loss class: `"process_exit"` + /// for an unrequested exit, the discard's reason (e.g. `"request_timeout"`) + /// for a requested kill. + Lost { reason: &'static str }, + /// A previously-lost (or never-started) daemon completed a cold start and + /// is healthy again. + Started, +} + +/// Which loss arm is running the shared exactly-once path +/// ([`OpencodeServeManager::lose_daemon`]). +enum LossArm<'a> { + /// The exit watcher observed an unrequested process exit. `ownership_id` + /// gates staleness (a watcher for a superseded daemon must not run the + /// loss path on its successor); `base_url` rides the crash WARN. + Watcher { + base_url: &'a str, + ownership_id: &'a str, + }, + /// A requested discard (kill). `dispatched_base` is the base URL the + /// timed-out request was actually sent to — the discard takes+kills + /// ONLY the daemon at that base; a STALE timeout (its daemon already + /// lost, a replacement owns the entry) no-ops with the same silence as + /// a `None` take, never killing the innocent replacement. The watcher + /// is aborted first (a requested kill never raises the crash event); + /// `reason` names the discard cause and rides both the WARN and the + /// `Lost` signal. + Discard { + reason: &'static str, + dispatched_base: &'a str, + }, +} + struct RunningServe { base_url: String, - process: Box, + /// The shared daemon handle. `Arc` (LB-06): the exit watcher keeps a clone + /// that outlives this entry — the watcher polls ITS Arc, the entry keeps + /// its own, nothing is moved out. + process: Arc, + /// THIS daemon's spawn identity: the stale-watcher gate (a watcher for a + /// superseded daemon must not run the loss path on its successor). + ownership_id: String, _event_handle: Box, + /// The exit watcher's abort handle — aborted on the requested-loss paths + /// (discard/shutdown) so a killed daemon never raises the crash event. + _exit_watch: Option, } struct Inner { @@ -652,6 +718,37 @@ struct Inner { shutdown: AtomicBool, running: tokio::sync::Mutex>>, session_emitters: Mutex>>, + daemon_signals: broadcast::Sender, + /// The re-warm backoff's attempt counter: incremented per re-warm attempt + /// and reset when a loss is a FRESH incident (see + /// [`OpencodeServeManager::schedule_re_warm`]) — each new incident starts + /// at the initial delay while a crash-looping daemon still escalates to + /// and retries at the capped interval. + re_warm_attempts: AtomicUsize, + /// The DISPATCH ERA (ep2-r2 fresheyes Major — cross-generation event + /// contamination; HARDENED ep2-r3 — the check-to-dispatch TOCTOU): a + /// monotonic counter retired (incremented) by every running-entry TAKE + /// under the `running` lock — the two `lose_daemon` arms and `shutdown`. + /// Each daemon's dispatch sink captures the era at its connect (inside + /// the cold-start critical section, so no take can interleave) and + /// [`dispatch_event_on_era`] re-verifies it UNDER the `session_emitters` + /// mutex, atomically with sender-selection + send: no event originating + /// from a daemon whose loss has been taken can dispatch into the + /// successor era, even when the sink's fast-path check passed before a + /// preemption straddled the take. + event_era: AtomicU64, + /// ep2-r3 test seam (`cfg(test)`-only): when armed, the dispatch sink + /// parks AFTER passing its era check and BEFORE dispatching — the exact + /// preempted-callback window the ep2-r3 fresheyes Major pins (a check + /// that is not atomic with dispatch can pass, be preempted, and then + /// deliver into a successor era's registration). Never compiled into + /// production builds. See + /// [`OpencodeServeManager::arm_dispatch_park_for_tests`]. + #[cfg(test)] + test_dispatch_park: Mutex>>, + /// When the running daemon completed its (healthy) cold start — the + /// fresh-incident clock for the re-warm backoff. + last_cold_start_at: Mutex>, } /// The opencode serve sidecar client. Cheap to clone (`Arc`-backed). @@ -669,6 +766,12 @@ impl OpencodeServeManager { shutdown: AtomicBool::new(false), running: tokio::sync::Mutex::new(None), session_emitters: Mutex::new(HashMap::new()), + daemon_signals: broadcast::Sender::new(DAEMON_CHANNEL_CAPACITY), + re_warm_attempts: AtomicUsize::new(0), + event_era: AtomicU64::new(0), + #[cfg(test)] + test_dispatch_park: Mutex::new(None), + last_cold_start_at: Mutex::new(None), }), } } @@ -687,6 +790,22 @@ impl OpencodeServeManager { .map(|r| r.base_url.clone()) } + /// The CURRENT running daemon's spawn identity: the `ownership_id` + /// minted at its cold start — unique per daemon GENERATION, stable + /// across `ensure_started` fast paths, `None` while no daemon runs. + /// The freshopencode runtime's daemon-generation fence (Task 4 delta + /// round 2): a session bridge stamped with any other id was spawned + /// against a SUPERSEDED daemon and is dead for revival purposes even + /// while its task is still draining. + pub async fn ownership_id(&self) -> Option { + self.inner + .running + .lock() + .await + .as_ref() + .map(|r| r.ownership_id.clone()) + } + /// Idempotent start: allocate a loopback port, spawn the ownership-tagged sidecar, /// wait (bounded) for health, then connect the SSE consumer. Concurrent callers are /// single-flighted by the `running` mutex (`ensureStarted`, `serve-manager.ts:181-194`). @@ -734,7 +853,7 @@ impl OpencodeServeManager { OPENCODE_CONFIG_CONTENT_ENV.to_string(), merged_opencode_config_content(inherited.as_deref()), )); - let process = self + let process: Arc = self .inner .deps .spawner @@ -742,18 +861,26 @@ impl OpencodeServeManager { command: self.config().command.clone(), hostname: endpoint.hostname.clone(), port: endpoint.port, - ownership_id, + ownership_id: ownership_id.clone(), env, pure: false, cwd: None, }) - .map_err(ServeError::Spawn)?; + .map_err(ServeError::Spawn)? + .into(); if let Err(e) = self.wait_for_health(&base_url, process.as_ref()).await { process.kill(); return Err(e); } + // Arm the exit watcher BEFORE storing the entry — the spawn is + // synchronous (the guard is never held across an await here) and the + // watcher's first action is a sleep, so by the time it first consults + // `exited()` the entry is stored; its loss path still re-verifies + // ownership, so a store-visibility race can only no-op, never mis-fire. + let watch = + self.spawn_exit_watch(base_url.clone(), Arc::clone(&process), ownership_id.clone()); let sink = self.make_dispatch_sink(); let handle = self .inner @@ -764,8 +891,20 @@ impl OpencodeServeManager { *guard = Some(Arc::new(RunningServe { base_url: base_url.clone(), process, + ownership_id, _event_handle: handle, + _exit_watch: Some(watch), })); + // The fresh-incident clock for the re-warm backoff: this daemon's + // healthy-service lifetime starts now. + *self + .inner + .last_cold_start_at + .lock() + .expect("cold-start clock mutex") = Some(Instant::now()); + // COLD-start edge only — the fast path above (already running) never + // re-broadcasts this. + let _ = self.inner.daemon_signals.send(DaemonSignal::Started); Ok(base_url) } @@ -833,22 +972,325 @@ impl OpencodeServeManager { }) } + /// The per-connection dispatch sink, ERA-GATED (ep2-r2 fresheyes Major — + /// cross-generation event contamination; HARDENED ep2-r3 — the + /// check-to-dispatch TOCTOU): the sink captures the CURRENT + /// [`Inner::event_era`] at its daemon's connect (called from + /// `ensure_started` inside the cold-start `running` critical section, + /// so no take can interleave with the capture) and hands it to + /// [`dispatch_event_on_era`], which re-verifies the era UNDER the + /// `session_emitters` mutex, atomically with selection+send. The + /// load-compare here is only the CHEAP FAST PATH (the common + /// already-retired case drops without touching the emitter map) — it + /// is NOT the fence: an event that passes it can still be preempted + /// before dispatch, and only the under-lock re-verification stops a + /// stale event from reaching a successor-era registration after its + /// era was retired, its emitters swept, and the successor registered a + /// replacement sender. Every daemon removal is a TAKE under that same + /// `running` lock (`lose_daemon`'s two arms, `shutdown`), each take + /// retires the era via [`Self::retire_event_era`] BEFORE its sweep + /// claims the emitter mutex — so by construction NO event originating + /// from a daemon whose loss has been taken can dispatch into the + /// successor era: a buffered/late `session.idle` from the lost + /// generation can never satisfy the successor's `await_idle` (the + /// false `freshAgent.turn.complete` the no-chime-on-loss contract + /// forbids), and no other late event can contaminate a successor-era + /// bridge. The authoritative gate lives at DISPATCH — under the + /// emitter mutex — because dropping the [`EventStreamHandle`] only + /// ABORTS the transport's reader task (`SseHandle`'s drop), and a + /// check that is not atomic with dispatch can be preempted past both. fn make_dispatch_sink(&self) -> EventSink { let weak = Arc::downgrade(&self.inner); + let era = self.inner.event_era.load(Ordering::Acquire); Arc::new(move |event: ParsedServeEvent| { - if let Some(inner) = weak.upgrade() { - dispatch_event_on(&inner, event); + let Some(inner) = weak.upgrade() else { + return; + }; + if inner.event_era.load(Ordering::Acquire) != era { + // A superseded generation's late/buffered event: dropped at + // the gate, never reaching the successor era's emitters. + return; + } + // ep2-r3 test seam (`cfg(test)`-only): hold the callback in the + // exact window the fresheyes finding names — the era check has + // PASSED and the dispatch has not yet run — so a test can + // retire the era and register a successor-era sender while this + // callback is parked. See + // [`Self::arm_dispatch_park_for_tests`]. + #[cfg(test)] + if let Some(park) = inner + .test_dispatch_park + .lock() + .expect("test dispatch park mutex") + .clone() + { + park(); } + dispatch_event_on_era(&inner, event, Some(era)); }) } + /// Arm or clear the `cfg(test)`-only dispatch park (ep2-r3): the armed + /// closure is invoked by every dispatch sink AFTER passing its era + /// check and BEFORE dispatching, so a test can hold an in-flight + /// callback while it retires the era (a daemon loss take) and + /// registers a successor-era sender — the preempted-callback + /// interleaving the era verification's dispatch-time atomicity must + /// survive. Production builds never see the seam. + #[cfg(test)] + pub(crate) fn arm_dispatch_park_for_tests(&self, park: Option>) { + *self + .inner + .test_dispatch_park + .lock() + .expect("test dispatch park mutex") = park; + } + + /// Retire the current dispatch era — the take-side half of the era gate + /// ([`Self::make_dispatch_sink`]). Called ONLY while the `running` lock + /// is held for a take (the two `lose_daemon` arms, `shutdown`): from + /// this moment no event from the daemon being taken can dispatch into + /// the session-emitter map, whatever its SSE connection still buffers — + /// a successor era's emitters can never be contaminated by the lost + /// generation, and a successor (which can only cold-start after the + /// lock releases) always connects a strictly newer era. + fn retire_event_era(&self) { + self.inner.event_era.fetch_add(1, Ordering::Release); + } + + /// Spawn the daemon exit watcher for one cold-started daemon (Task 3, + /// the shared-daemon adaptation of the freshcodex onExit self-heal): poll + /// the SHARED process Arc's `exited()` every `daemon_watch_interval`; on + /// `Some` run the staleness-gated loss path and end. The manager clone is + /// cheap (`Arc`-backed); the process Arc and ownership id move in with + /// the task — the running entry keeps its own Arc (LB-06: share, never + /// move the daemon out of the entry). + fn spawn_exit_watch( + &self, + base_url: String, + process: Arc, + ownership_id: String, + ) -> tokio::task::AbortHandle { + let manager = self.clone(); + let interval = self.config().daemon_watch_interval; + let handle = tokio::spawn(async move { + loop { + tokio::time::sleep(interval).await; + if process.exited().is_some() { + manager + .lose_daemon(LossArm::Watcher { + base_url: &base_url, + ownership_id: &ownership_id, + }) + .await; + return; + } + } + }); + handle.abort_handle() + } + + /// The shared exactly-once daemon-loss path: a daemon that died on its own + /// (the watcher arm) or was discarded (the requested-kill arm) must not + /// leave a poisoned running entry (the 2026-09-20 incident's silent + /// half). **Exactly-once (LB-07):** the running-entry take is the race + /// arbiter — a `None` take means the other arm already handled this loss, + /// and the whole path is a silent no-op: no log, no Lost, no re-warm. The + /// arm selects the pre-take gate and the structured WARN (a requested kill + /// never raises the crash event; a watcher's WARN names the dead daemon). + /// + /// **Sweep fencing (ep2-r1 fresheyes Major):** the session-emitter sweep + /// is part of the SAME `running` critical section as the take, BEFORE the + /// lock releases. A successor daemon cannot cold-start while this lock is + /// held (`ensure_started` needs it), so every sender claimed by the sweep + /// was necessarily registered against the daemon being lost (or against + /// no daemon at all — an in-flight `once_idle` whose request was going to + /// fail anyway); any registration that happens after the release belongs + /// to a SUCCESSOR and is never swept by this loss. Pre-fix, the sweep ran + /// after the lock release (kill/reap → `emit_lost_for_all`), so a fenced + /// attach that cold-started daemon B in that window had B's fresh bridge + /// sender wiped by A's late cleanup — B's bridge then drained its closed + /// channel and exited with no recovery trigger left (B's `Started` pass + /// had already seen a live B-stamped bridge, the `Lost` pass never + /// revives, and A's re-warm takes B's fast path silently) — the exact + /// dead-ended pane this recovery exists to heal. + /// + /// **Era retirement (ep2-r2 fresheyes Major):** the take also retires + /// the lost daemon's DISPATCH ERA inside the same critical section + /// ([`Self::retire_event_era`]) — the taken daemon's event sink drops + /// every event from this moment on, so a buffered/late event from the + /// lost generation can never dispatch into the successor era's + /// emitters (the cross-generation `session.idle` that would falsely + /// satisfy a successor's `await_idle` and produce a chime). See + /// [`Self::make_dispatch_sink`]. + async fn lose_daemon(&self, arm: LossArm<'_>) { + let (taken, lost_senders) = { + let mut running = self.inner.running.lock().await; + match arm { + LossArm::Watcher { + base_url: _, + ownership_id, + } => match running.as_ref() { + // Still OUR daemon: retire its dispatch era, take it + // (the loss is ours to handle) and sweep the shared + // session-emitter map while no successor can be + // starting. + Some(r) if r.ownership_id == ownership_id => { + self.retire_event_era(); + let senders = self.take_session_emitters(); + (running.take(), senders) + } + // Stale watcher — a newer daemon owns the entry: no-op. + _ => return, + }, + LossArm::Discard { + reason: _, + dispatched_base, + } => match running.as_ref() { + // The wedged request's OWN daemon: abort the watcher + // FIRST (inside the lock, before the take and the + // WARN/kill sequence) — the requested kill must never + // raise the crash event. After the take the watcher can + // never win its own take; if it already won, our take + // below is the silent no-op. + Some(r) if r.base_url == dispatched_base => { + if let Some(watch) = &r._exit_watch { + watch.abort(); + } + // The take retires the taken daemon's dispatch era + // under the same lock (the era-gate invariant). + self.retire_event_era(); + let senders = self.take_session_emitters(); + (running.take(), senders) + } + // Stale timeout — the request's daemon is already gone + // and a replacement owns the entry: no-op (no kill, no + // log, no Lost, no re-warm), the same silence as a + // `None` take. The gate is the base URL captured at + // dispatch — the exact address the wedged request was + // sent to — so a mismatch means the entry is a + // different (re-warmed) daemon the request never used. + _ => return, + }, + } + }; + let Some(running) = taken else { + return; + }; + let reason = match arm { + LossArm::Watcher { base_url, .. } => { + tracing::warn!( + reason = "process_exit", + base_url = %base_url, + "freshagent.opencode.daemon_crash_detected" + ); + "process_exit" + } + LossArm::Discard { reason, .. } => { + tracing::warn!(reason = reason, "freshagent.opencode.daemon_discarded"); + reason + } + }; + // The watcher arm's kill is reaper parity only (the process already + // exited); the discard arm's kill is the requested kill. Either way + // kill() reaps the /proc-scoped ownership tree. + running.process.kill(); + // The sweep already ran INSIDE the take critical section, so + // `lost_senders` is exactly the set of THIS daemon's emitters it + // claimed: the Lost edge goes only to the sessions the lost daemon + // served, and a successor's post-release registration is not in it + // and stays live. + for sender in lost_senders { + let _ = sender.send(SessionSignal::Lost); + } + let _ = self + .inner + .daemon_signals + .send(DaemonSignal::Lost { reason }); + self.schedule_re_warm(); + } + + /// Schedule the backoff-guarded respawn after a daemon loss: a RETRY loop + /// that sleeps `re_warm_backoff_initial_ms * 2^(attempts-1)` (capped at + /// `re_warm_backoff_max_ms`), then calls `ensure_started`. A FAILED + /// attempt logs and schedules the next (escalating) attempt — a transient + /// spawn/health failure (e.g. disk pressure) must not strand the daemon + /// permanently absent. A SUCCESSFUL attempt ends the loop; the fresh + /// daemon's own exit watcher is armed by `ensure_started`. Never spawns + /// (nor retries) once shutdown is set. + /// + /// **Fresh-incident gate** (the round-2 "no permanent 60 s first delay" + /// finding, reconciled with the behavior list's crash-loop escalation): + /// a daemon that OUTLIVED the whole backoff ladder makes this loss a NEW + /// incident — the attempt counter resets so its re-warm starts at the + /// initial delay. A daemon that died faster KEEPS the accumulated + /// escalation: a daemon dying immediately after every successful start + /// must climb the ladder (50→100→200→400 ms…), never respawn at the + /// floor every cycle. The counter therefore persists across re-warm + /// successes and resets only here, at the next loss, when the lost + /// daemon's healthy lifetime reached the ladder's cap. + fn schedule_re_warm(&self) { + if self.inner.shutdown.load(Ordering::SeqCst) { + return; + } + let fresh_incident = { + let last_cold_start = *self + .inner + .last_cold_start_at + .lock() + .expect("cold-start clock mutex"); + last_cold_start + .map(|started_at| { + started_at.elapsed() + >= Duration::from_millis(self.config().re_warm_backoff_max_ms) + }) + .unwrap_or(true) + }; + if fresh_incident { + self.inner.re_warm_attempts.store(0, Ordering::SeqCst); + } + let manager = self.clone(); + tokio::spawn(async move { + loop { + let attempts = manager + .inner + .re_warm_attempts + .fetch_add(1, Ordering::SeqCst) + + 1; + let delay_ms = (manager + .config() + .re_warm_backoff_initial_ms + .saturating_mul(1u64 << (attempts - 1).min(16))) + .min(manager.config().re_warm_backoff_max_ms); + tokio::time::sleep(Duration::from_millis(delay_ms)).await; + if manager.inner.shutdown.load(Ordering::SeqCst) { + return; + } + match manager.ensure_started().await { + Ok(_) => { + tracing::info!(attempt = attempts, "freshagent.opencode.daemon_re_warm"); + return; + } + Err(err) => { + tracing::warn!( + attempt = attempts, + error = %err, + "freshagent.opencode.daemon_re_warm" + ); + } + } + } + }); + } + async fn require_base(&self) -> Result { self.ensure_started().await } /// One JSON request/response through the transport, bounded by the config's - /// `request_timeout`. On a timeout the running sidecar is discarded - /// (`discardRunning('request_timeout')`, `serve-manager.ts:320-324`). + /// `request_timeout`. On a timeout the sidecar the request was dispatched + /// against is discarded (`discardRunning('request_timeout')`, + /// `serve-manager.ts:320-324`) — never a re-warmed replacement. /// `not_found_value` mirrors `json`'s 404 handling. async fn json_request( &self, @@ -923,7 +1365,11 @@ impl OpencodeServeManager { { Err(_) => { if discard_on_timeout == DiscardOnTimeout::Yes { - self.discard_running("request_timeout").await; + // Gate on the daemon THIS request was dispatched against: + // a stale timeout (its daemon already discarded, a + // re-warmed replacement installed) must never kill the + // replacement — the discard no-ops on a base mismatch. + self.discard_running("request_timeout", &base).await; } return Err(ServeError::RequestTimeout { method: method_str, @@ -1164,9 +1610,24 @@ impl OpencodeServeManager { /// compact path consumes only its `model` key (probed on 1.18.18: present, /// string-or-null) as the model-pair fallback when a session carries no splittable /// model of its own. + /// + /// A slow config read must never kill the shared daemon — the FR2 read rule + /// (b8ke): capture the base once (spawn-on-demand is preserved), then + /// transport over the captured base with `DiscardOnTimeout::No`. pub async fn get_config(&self, route: &Route) -> Result { + let base = self.require_base().await?; let path = with_route("/config", route); - self.json_request(HttpMethod::Get, &path, None, None).await + self.json_request_over_base( + HttpMethod::Get, + &path, + None, + None, + base, + DiscardOnTimeout::No, + &[], + None, + ) + .await } /// `POST /session/:id/summarize` — the compact RPC. VALIDATED opencode 1.18.18 @@ -1217,11 +1678,22 @@ impl OpencodeServeManager { if let Some(w) = accepted_witness { witnesses.push(w); } - self.json_request_maybe_witnessed( + // 2026-09-20 incident: the summarize POST used the discard-on-timeout + // lane, so a 600 s budget exceeded on a healthy-but-busy daemon KILLED + // the one shared daemon for every freshopencode session. Mirror the + // FR2 captured-base transport (`get_session_at`): a timed-out compact + // answers `RequestTimeout` and NEVER kills the shared daemon. The + // redo-destroy classification is unchanged — `RequestTimeout` stays + // outside `never_dispatched()` (a timed-out POST may have reached the + // daemon). + let base = self.require_base().await?; + self.json_request_over_base( HttpMethod::Post, &path, Some(json!({ "providerID": provider_id, "modelID": model_id })), None, + base, + DiscardOnTimeout::No, &witnesses, // The summarize handler runs the whole LLM turn before answering; // use its dedicated timeout rather than the generic request bound. @@ -1321,36 +1793,75 @@ impl OpencodeServeManager { self.emitter_for(session_id).subscribe() } - /// Feed one parsed SSE event into the per-session fan-out. This is the ingestion - /// point the [`EventSource`] sink calls (`dispatchEvent`, `serve-manager.ts:429-432`). + /// Subscribe to the daemon-level lifecycle stream ([`DaemonSignal`]). + /// + /// NOTE (LB-02a, source-verified at the locked tokio version): tokio + /// broadcast does NOT replay history to late subscribers — a receiver + /// created here starts at the channel's current TAIL and observes only + /// signals sent AFTER this call. Consumers (Task 4's bridge revival) must + /// therefore be LEVEL-TRIGGERED (query daemon state on receipt), never + /// event-history-dependent. + pub fn subscribe_daemon_signals(&self) -> broadcast::Receiver { + self.inner.daemon_signals.subscribe() + } + + /// Feed one parsed SSE event into the per-session fan-out. This is the + /// generation-less ingestion seam (`dispatchEvent`, + /// `serve-manager.ts:429-432`) — test-facing in practice. It is + /// deliberately UNGATED (no era): the era fence guards the REAL + /// per-connection sinks (see [`Self::make_dispatch_sink`] and + /// [`dispatch_event_on_era`]); a production caller must ingest through + /// a sink, not this seam. pub fn dispatch_event(&self, event: ParsedServeEvent) { dispatch_event_on(&self.inner, event); } + /// Collect every registered session-emitter sender and CLEAR the shared + /// map — the sweep half of [`Self::emit_lost_for_all`], split out so the + /// daemon-loss path can run it INSIDE the `running` critical section + /// that removes the dead daemon (see [`Self::lose_daemon`]): while that + /// lock is held no successor daemon can cold-start (`ensure_started` + /// needs it), so every sender claimed here belongs to the daemon being + /// lost, and anything registered later (a successor's bridge) is never + /// swept by this loss. + fn take_session_emitters(&self) -> Vec> { + let mut map = self + .inner + .session_emitters + .lock() + .expect("session emitters mutex"); + let senders = map.values().cloned().collect(); + map.clear(); + senders + } + /// Signal every subscriber that the sidecar was lost (`emitLostForAllSessions`, - /// `serve-manager.ts:126-132`). Exposed for the sidecar-loss liveness path/tests. + /// `serve-manager.ts:126-132`). Exposed for the sidecar-loss liveness path/tests + /// and the shutdown teardown. The daemon-LOSS path does NOT use this method: + /// it sweeps via [`Self::take_session_emitters`] inside its own `running` + /// critical section (see [`Self::lose_daemon`]) so a successor daemon's + /// fresh senders can never be wiped by a predecessor's late cleanup; this + /// whole-map form is correct only where no successor remains to protect + /// (final shutdown, liveness tests). pub fn emit_lost_for_all(&self) { - let emitters: Vec> = { - let mut map = self - .inner - .session_emitters - .lock() - .expect("session emitters mutex"); - let senders = map.values().cloned().collect(); - map.clear(); - senders - }; - for sender in emitters { + for sender in self.take_session_emitters() { let _ = sender.send(SessionSignal::Lost); } } - async fn discard_running(&self, _reason: &str) { - let taken = self.inner.running.lock().await.take(); - if let Some(running) = taken { - running.process.kill(); - } - self.emit_lost_for_all(); + /// The requested-loss arm: discard the running daemon the timed-out + /// request at `dispatched_base` was sent to (a Yes-lane request + /// timeout, …), WARN the Task-2 structured event, then run the shared + /// exactly-once loss path (Lost signal + backoff re-warm). A STALE + /// timeout — one whose daemon was already replaced — no-ops silently. + /// `reason` is `&'static` because it rides the [`DaemonSignal::Lost`] + /// broadcast to daemon-signal subscribers. + async fn discard_running(&self, reason: &'static str, dispatched_base: &str) { + self.lose_daemon(LossArm::Discard { + reason, + dispatched_base, + }) + .await; } // ── the IDLE edge (once_idle / await_idle, serve-manager.ts:440-520) ───────── @@ -1509,8 +2020,22 @@ impl OpencodeServeManager { /// the dropped handle), and signal all sessions lost (`shutdown`, `serve-manager.ts:573-591`). pub async fn shutdown(&self) { self.inner.shutdown.store(true, Ordering::SeqCst); - let taken = self.inner.running.lock().await.take(); + let taken = { + let mut running = self.inner.running.lock().await; + // The take retires the dispatch era under the lock — the same + // era-gate invariant as the loss path (a late event from the + // daemon being taken must never dispatch past the take). + self.retire_event_era(); + running.take() + }; if let Some(running) = taken { + // The requested-loss discipline: abort the watcher so the shutdown + // kill never raises the crash event. No `Lost` signal, no re-warm — + // the shutdown flag set above blocks `schedule_re_warm`, and the + // server is going down. + if let Some(watch) = &running._exit_watch { + watch.abort(); + } running.process.kill(); } self.emit_lost_for_all(); @@ -1518,19 +2043,54 @@ impl OpencodeServeManager { } fn dispatch_event_on(inner: &Arc, event: ParsedServeEvent) { + dispatch_event_on_era(inner, event, None); +} + +/// The era-verified dispatch (ep2-r3 fresheyes Major — the check-to-dispatch +/// TOCTOU): era verification is ATOMIC with dispatch. `Some(era)` (the sink +/// path, carrying the era its daemon connected under) is re-verified UNDER +/// the `session_emitters` mutex, and the mutex is held across the +/// check → sender-selection → send — the review's required invariant, +/// exactly: an event dispatches into a session's sender only if the event's +/// daemon era matches the CURRENT era, verified under the lock the sweep +/// itself must take. The take retires the era (an atomic increment) BEFORE +/// its sweep claims the emitters mutex, so a callback that re-verifies under +/// this lock either sees the retired era (the event is dropped BEFORE any +/// entry is selected or created — a retired-era event can never mint an +/// emitter entry) or holds the lock ahead of the sweep, in which case every +/// entry it selects was registered in its OWN era (the sweep clears the map +/// at every retirement). Either way no stale-era event can reach a +/// successor-era registration, whatever the OS scheduler does between a +/// sink's fast-path check and this dispatch. +/// +/// `None` (the pub [`OpencodeServeManager::dispatch_event`] seam) is the +/// generation-less legacy lane, deliberately ungated — its callers are tests +/// and test helpers; if a production caller ever appears it must go through +/// a sink (or carry an era here). +fn dispatch_event_on_era(inner: &Arc, event: ParsedServeEvent, era: Option) { let Some(session_id) = event.session_id.clone() else { return; }; - let sender = { - let mut emitters = inner - .session_emitters - .lock() - .expect("session emitters mutex"); - emitters - .entry(session_id) - .or_insert_with(|| broadcast::Sender::new(SESSION_CHANNEL_CAPACITY)) - .clone() - }; + let mut emitters = inner + .session_emitters + .lock() + .expect("session emitters mutex"); + if let Some(era) = era { + if inner.event_era.load(Ordering::Acquire) != era { + // The era this event originated from was retired while the + // callback was in flight (after its sink passed the check, + // before this lock): drop it under the lock — never dispatched, + // never an entry. + return; + } + } + let sender = emitters + .entry(session_id) + .or_insert_with(|| broadcast::Sender::new(SESSION_CHANNEL_CAPACITY)) + .clone(); + // The send rides inside the same critical section: the selected sender + // is the era-verified one, and `broadcast::Sender::send` is a + // non-blocking enqueue (no re-entrancy into this mutex). let _ = sender.send(SessionSignal::Event(event)); } @@ -1617,7 +2177,7 @@ fn encode_path_segment(segment: &str) -> String { #[cfg(test)] mod tests { use super::*; - use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; // ── display_error_chain (transport diagnostics preservation) ───────────── @@ -1774,18 +2334,24 @@ mod tests { // ── compact (POST /session/:id/summarize) + get_config (GET /config) ──────── /// A `ServeHttp` fake that records every request (`METHOD url body?`) and scripts - /// responses: healthy probes, summarize per `summarize_status`, fork per - /// `fork_status`/`fork_body`, `/config` per `config_body`, everything else a + /// responses: healthy probes, summarize per `summarize_status` (or a NEVER-resolving + /// response when `summarize_pending` — the wedged shape from + /// `tests/serve_health_bounded.rs`), fork per `fork_status`/`fork_body`, `/config` + /// per `config_body` (or never-resolving when `config_pending`), `/prompt_async` + /// never-resolving when `prompt_pending`, everything else a /// benign 200 `{}`. Per-request timeouts land in the index-aligned /// [`RecordingHttp::timeouts`] vec (`requests[i]`'s timeout is `timeouts[i]`). struct RecordingHttp { requests: Mutex)>>, timeouts: Mutex>>, summarize_status: u16, + summarize_pending: bool, fork_status: u16, fork_body: Vec, config_body: Vec, + config_pending: bool, revert_status: u16, + prompt_pending: bool, } impl RecordingHttp { @@ -1794,10 +2360,13 @@ mod tests { requests: Mutex::new(Vec::new()), timeouts: Mutex::new(Vec::new()), summarize_status: 200, + summarize_pending: false, fork_status: 200, fork_body: br#"{"id":"ses_child","directory":"/tmp/x"}"#.to_vec(), config_body: br#"{"model":null}"#.to_vec(), + config_pending: false, revert_status: 200, + prompt_pending: false, } } @@ -1839,6 +2408,14 @@ mod tests { return Box::pin(async { Ok(ServeHttpResponse::new(200, b"{}".to_vec())) }); } if req.url.contains("/summarize") { + if self.summarize_pending { + // A genuine wedge: the response NEVER resolves — only the + // caller's per-request bound can settle it. + return Box::pin(async { + std::future::pending::<()>().await; + unreachable!() + }); + } let status = self.summarize_status; let body = if status == 200 { // VALIDATED 1.18.18 contract: the summarize success body is a boolean. @@ -1863,9 +2440,25 @@ mod tests { return Box::pin(async move { Ok(ServeHttpResponse::new(status, body)) }); } if req.url.contains("/config") { + if self.config_pending { + // A genuine wedge: the response NEVER resolves — only the + // caller's per-request bound can settle it. + return Box::pin(async { + std::future::pending::<()>().await; + unreachable!() + }); + } let body = self.config_body.clone(); return Box::pin(async move { Ok(ServeHttpResponse::new(200, body)) }); } + if req.url.contains("/prompt_async") && self.prompt_pending { + // A genuine wedge: the response NEVER resolves — only the + // caller's per-request bound can settle it. + return Box::pin(async { + std::future::pending::<()>().await; + unreachable!() + }); + } Box::pin(async move { Ok(ServeHttpResponse::new(200, b"{}".to_vec())) }) } } @@ -1898,6 +2491,35 @@ mod tests { } } + /// A never-exiting serve process whose `kill()` calls are COUNTED — the + /// discard-on-timeout assertion seam (the `tests/serve_health_bounded.rs` + /// `NeverExitsProcess` pattern). + struct KillCountingProcess { + killed: Arc, + } + impl ServeProcess for KillCountingProcess { + fn exited(&self) -> Option { + None + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) { + self.killed.fetch_add(1, Ordering::SeqCst); + } + } + + struct KillCountingSpawner { + killed: Arc, + } + impl ProcessSpawner for KillCountingSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + Ok(Box::new(KillCountingProcess { + killed: self.killed.clone(), + })) + } + } + struct NoopHandle; impl EventStreamHandle for NoopHandle {} struct NoopEventSource; @@ -1907,6 +2529,22 @@ mod tests { } } + /// An [`EventSource`] that RECORDS every sink it is handed (one per cold + /// start, in connect order) — the REAL per-connection dispatch closures + /// the manager mints at each daemon's connect, so a test can call a + /// daemon generation's sink directly, carrying that generation's + /// identity (the `tests/serve_daemon_selfheal.rs` + /// `RecordingEventSource` shape, unit-side). + struct RecordingEventSource { + sinks: Mutex>, + } + impl EventSource for RecordingEventSource { + fn connect(&self, _url: String, sink: EventSink) -> Box { + self.sinks.lock().expect("recorded sinks mutex").push(sink); + Box::new(NoopHandle) + } + } + async fn started_recording_manager(http: Arc) -> OpencodeServeManager { started_recording_manager_with_config(http, ServeConfig::default()).await } @@ -1928,6 +2566,26 @@ mod tests { mgr } + /// [`started_recording_manager_with_config`] with a kill-counting spawner, + /// for the lanes that must NEVER kill the shared daemon. + async fn started_recording_manager_counting_kills( + http: Arc, + config: ServeConfig, + killed: Arc, + ) -> OpencodeServeManager { + let deps = ServeDeps { + spawner: Arc::new(KillCountingSpawner { killed }), + http, + ports: Arc::new(FakeAllocator), + events: Arc::new(NoopEventSource), + }; + let mgr = OpencodeServeManager::new(deps, config); + mgr.ensure_started() + .await + .expect("healthy fake serve starts"); + mgr + } + // ── b8ke focused round-2 review R2-4: the dispatch-boundary witness ───────── /// A `ServeHttp` fake whose `prompt_async` handler parks the response @@ -2135,6 +2793,53 @@ mod tests { } } + // 2026-09-20 incident: a compact timeout (600 s budget) ran the + // DiscardOnTimeout::Yes arm and KILLED the one shared `opencode serve` + // daemon for every freshopencode session. The compact lane must degrade + // like the FR2 snapshot lane: the POST times out, the daemon survives. + #[tokio::test] + async fn compact_timeout_does_not_kill_the_shared_daemon() { + let killed = Arc::new(AtomicUsize::new(0)); + let http = Arc::new(RecordingHttp { + summarize_pending: true, + ..RecordingHttp::new() + }); + let config = ServeConfig { + compact_timeout: Duration::from_millis(50), + ..ServeConfig::default() + }; + let mgr = + started_recording_manager_counting_kills(http.clone(), config, killed.clone()).await; + + let err = mgr + .compact("ses_timeout", "prov-a", "mdl-x", &None, None, None) + .await + .expect_err("the summarize POST must time out"); + assert!( + matches!(err, ServeError::RequestTimeout { .. }), + "got {err:?}" + ); + assert_eq!( + killed.load(Ordering::SeqCst), + 0, + "a compact timeout must NEVER kill the shared daemon" + ); + assert!( + mgr.base_url().await.is_some(), + "the running entry must survive a compact timeout" + ); + // The compact-timeout POST must still carry the dedicated budget. + let requests = http.recorded(); + let summarize_index = requests + .iter() + .position(|(method, url, _)| method == "POST" && url.contains("/summarize")) + .expect("a summarize POST was recorded"); + assert_eq!( + http.recorded_timeout(summarize_index), + Some(Duration::from_millis(50)) + ); + } + #[tokio::test] async fn get_config_returns_the_raw_config_body() { let http = Arc::new(RecordingHttp { @@ -2155,6 +2860,40 @@ mod tests { assert!(body.is_none(), "GET /config carries no body"); } + // The compact drive's pre-flight model-pair resolution reads /config; a slow + // config GET is the same defect class (a read must never kill the daemon). + #[tokio::test] + async fn get_config_timeout_does_not_kill_the_shared_daemon() { + let killed = Arc::new(AtomicUsize::new(0)); + let http = Arc::new(RecordingHttp { + config_pending: true, + ..RecordingHttp::new() + }); + let config = ServeConfig { + request_timeout: Duration::from_millis(50), + ..ServeConfig::default() + }; + let mgr = started_recording_manager_counting_kills(http, config, killed.clone()).await; + + let err = mgr + .get_config(&None) + .await + .expect_err("config GET must time out"); + assert!( + matches!(err, ServeError::RequestTimeout { .. }), + "got {err:?}" + ); + assert_eq!( + killed.load(Ordering::SeqCst), + 0, + "a config read timeout must NEVER kill the shared daemon" + ); + assert!( + mgr.base_url().await.is_some(), + "the running entry must survive a config read timeout" + ); + } + // ── fork (POST /session/:id/fork) ──────────────────────────────────────── /// The recorded `POST /session/:id/fork` request, if any. @@ -2754,6 +3493,341 @@ mod tests { ); } + /// 2026-09-20 incident: the daemon discard that killed the shared serve left + /// ZERO log trace (its reason parameter went unused), so the shared-daemon + /// death was undiagnosable from the structured JSONL log. The discard must + /// be observable: a WARN `freshagent.opencode.daemon_discarded` naming its + /// reason. Driven through `prompt_async` — a deliberate + /// `DiscardOnTimeout::Yes` lane — so a pending prompt POST times out and + /// takes the discard path. + #[tokio::test] + async fn discard_running_emits_a_structured_warn_with_its_reason() { + let killed = Arc::new(AtomicUsize::new(0)); + let http = Arc::new(RecordingHttp { + prompt_pending: true, + ..RecordingHttp::new() + }); + let config = ServeConfig { + request_timeout: Duration::from_millis(50), + ..ServeConfig::default() + }; + let (events, _guard) = config_capture::capture(); + let mgr = started_recording_manager_counting_kills(http, config, killed.clone()).await; + + let err = mgr + .prompt_async( + "ses_discard", + build_prompt_body("hi", None, None), + &None, + None, + ) + .await + .expect_err("the prompt POST must time out"); + assert!( + matches!(err, ServeError::RequestTimeout { .. }), + "got {err:?}" + ); + // The discard itself ran: the Yes-lane timeout took the daemon down. + assert_eq!( + killed.load(Ordering::SeqCst), + 1, + "the discard must actually kill the running daemon here" + ); + let events = events.lock().expect("capture lock"); + let discard = events + .iter() + .find(|fields| { + fields.get("message").map(String::as_str) + == Some("freshagent.opencode.daemon_discarded") + }) + .expect("a daemon discard must emit freshagent.opencode.daemon_discarded"); + assert_eq!( + discard.get("reason").map(String::as_str), + Some("request_timeout"), + "the discard warn carries its reason: {discard:?}" + ); + } + + // ── Task 3: the daemon exit watcher's loss path (unit side) ────────────── + + /// A serve whose "exit" is test-controlled: `exited()` reports `Some(0)` + /// once the shared flag is set (the `tests/serve_daemon_selfheal.rs` + /// `FlagExitProcess` shape, unit-side). + struct FlagExitProcess { + exited: Arc, + killed: Arc, + } + impl ServeProcess for FlagExitProcess { + fn exited(&self) -> Option { + self.exited.load(Ordering::SeqCst).then_some(0) + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) { + self.killed.fetch_add(1, Ordering::SeqCst); + } + } + + struct FlagExitSpawner { + exited: Arc, + killed: Arc, + } + impl ProcessSpawner for FlagExitSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + Ok(Box::new(FlagExitProcess { + exited: self.exited.clone(), + killed: self.killed.clone(), + })) + } + } + + /// The watcher's unrequested-exit arm must WARN + /// `freshagent.opencode.daemon_crash_detected` with the loss reason and + /// the dead daemon's base URL — the diagnosability complement of the + /// Task-2 discard log (a silent shared-daemon death was the incident's + /// undiagnosable half). The watcher task runs on this current-thread + /// runtime, so the thread-local capture sees its WARN; awaiting the + /// `DaemonSignal::Lost` edge first guarantees the loss path already ran. + #[tokio::test] + async fn unrequested_daemon_exit_warns_daemon_crash_detected_with_reason_and_base_url() { + let exited = Arc::new(AtomicBool::new(false)); + let killed = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + killed: killed.clone(), + }), + http: Arc::new(RecordingHttp::new()), + ports: Arc::new(FakeAllocator), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + daemon_watch_interval: Duration::from_millis(5), + re_warm_backoff_initial_ms: 5, + re_warm_backoff_max_ms: 50, + ..ServeConfig::default() + }; + let mgr = OpencodeServeManager::new(deps, config); + mgr.ensure_started() + .await + .expect("healthy fake serve starts"); + let mut signals = mgr.subscribe_daemon_signals(); + let (events, _guard) = config_capture::capture(); + + exited.store(true, Ordering::SeqCst); // the daemon "exits" + let signal = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + signal, + DaemonSignal::Lost { + reason: "process_exit" + } + ), + "got {signal:?}" + ); + assert!( + killed.load(Ordering::SeqCst) >= 1, + "the already-exited daemon is still kill()ed for /proc-reaper parity" + ); + + let events = events.lock().expect("capture lock"); + let warn = events + .iter() + .find(|fields| { + fields.get("message").map(String::as_str) + == Some("freshagent.opencode.daemon_crash_detected") + }) + .expect("an unrequested daemon exit must WARN daemon_crash_detected"); + assert_eq!( + warn.get("reason").map(String::as_str), + Some("process_exit"), + "the crash warn names the loss reason: {warn:?}" + ); + assert_eq!( + warn.get("base_url").map(String::as_str), + Some("http://127.0.0.1:1"), + "the crash warn names the dead daemon's base URL: {warn:?}" + ); + } + + // ── ep2-r3 fresheyes Major: the check-to-dispatch TOCTOU ────────────── + + /// The era check is not atomic with dispatch into `session_emitters` + /// (ep2-r3 fresheyes Major — the check-to-dispatch TOCTOU): a callback + /// can pass the check, be preempted before `dispatch_event_on` acquires + /// the emitter mutex, and resume only after its daemon's loss was + /// TAKEN (the era retired, the emitters swept) and the successor + /// registered a replacement sender — the stale event is then delivered + /// into the successor's registration; a stale `session.idle` would + /// falsely satisfy the successor's `await_idle` (the false + /// `freshAgent.turn.complete` precursor). + /// + /// The interleaving, forced deterministically through the + /// `cfg(test)` dispatch park: A's REAL sink passes the era check and + /// PARKS; while held, A is lost through the REAL watcher arm (take + + /// era retire + sweep) and the successor B re-warms (its own sink + /// connected); a B-era `await_idle` is subscribed and IN FLIGHT for + /// the durable session; only then is the parked callback released — + /// its era check ALREADY PASSED, so only a re-verification atomic + /// with dispatch can stop it. The stale A-era event must NOT satisfy + /// B's await. A GENUINE B-era idle through B's own sink still must + /// (the gate is an era fence, not a broken dispatch). Pre-fix, the + /// released callback dispatched straight into B's fresh registration + /// and the successor's await resolved Ok. + #[tokio::test] + async fn a_preempted_era_checked_event_never_reaches_the_successors_registration() { + let exited = Arc::new(AtomicBool::new(false)); + let killed = Arc::new(AtomicUsize::new(0)); + let events = Arc::new(RecordingEventSource { + sinks: Mutex::new(Vec::new()), + }); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + killed: killed.clone(), + }), + http: Arc::new(RecordingHttp::new()), + ports: Arc::new(FakeAllocator), + events: events.clone(), + }; + let config = ServeConfig { + daemon_watch_interval: Duration::from_millis(5), + re_warm_backoff_initial_ms: 5, + re_warm_backoff_max_ms: 50, + ..ServeConfig::default() + }; + let mgr = OpencodeServeManager::new(deps, config); + mgr.ensure_started() + .await + .expect("healthy fake serve starts"); + let mut signals = mgr.subscribe_daemon_signals(); + let sink_a = events + .sinks + .lock() + .expect("recorded sinks mutex") + .first() + .expect("A's cold start connected its event stream") + .clone(); + + // The park: `entered` fires once the callback has passed A's era + // check; dropping `release_tx` resumes it. The release receiver + // rides a Mutex because the park closure must be `Sync` (an + // `EventSink` requirement) and only the parked callback ever + // touches it. + let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let release_rx = Mutex::new(release_rx); + mgr.arm_dispatch_park_for_tests(Some(Arc::new(move || { + let _ = entered_tx.send(()); + let _ = release_rx.lock().expect("release latch mutex").recv(); + }))); + + // The preempted A-era callback, on its own OS thread: the sink is + // synchronous, and the runtime must stay free to run the loss and + // the re-warm while the callback is held. + let done = Arc::new(AtomicBool::new(false)); + let done_thread = done.clone(); + let idle_event = || { + crate::events::parse_serve_event(&serde_json::json!({ + "type": "session.idle", + "properties": { "sessionID": "ses_race" } + })) + .expect("parseable serve event") + }; + let sink_thread = std::thread::spawn(move || { + sink_a(idle_event()); + done_thread.store(true, Ordering::SeqCst); + }); + entered_rx + .recv_timeout(std::time::Duration::from_secs(2)) + .expect("the sink parks after passing the era check, before dispatch"); + + // While the callback is held: A dies through the REAL watcher arm — + // the take retires the dispatch era and sweeps the emitters inside + // the same running-lock critical section. + exited.store(true, Ordering::SeqCst); + let lost = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + lost, + DaemonSignal::Lost { + reason: "process_exit" + } + ), + "got {lost:?}" + ); + // The successor B re-warms — a NEW generation with its own sink. + exited.store(false, Ordering::SeqCst); + let started = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("re-warm within budget") + .expect("channel alive"); + assert!(matches!(started, DaemonSignal::Started), "got {started:?}"); + let sink_b = events + .sinks + .lock() + .expect("recorded sinks mutex") + .get(1) + .expect("B's cold start connected its event stream") + .clone(); + + // The B-era registration the stale event must not reach: an + // in-flight `await_idle` for the durable session. + let rx = mgr.subscribe("ses_race"); + let idle_manager = mgr.clone(); + let mut await_idle = tokio::spawn(async move { + idle_manager + .await_idle("ses_race", rx, Duration::from_secs(5), None) + .await + }); + tokio::time::sleep(Duration::from_millis(50)).await; + + // Release the preempted callback: the era check already passed — + // only an ATOMIC re-verification at dispatch can stop it now. + drop(release_tx); + let deadline = Instant::now() + Duration::from_secs(2); + while !done.load(Ordering::SeqCst) { + assert!( + Instant::now() < deadline, + "the parked callback must complete after the release" + ); + tokio::time::sleep(Duration::from_millis(2)).await; + } + + // THE assertion: the stale A-era event must NOT satisfy B's await — + // the successor stays pending through the grace window. + match tokio::time::timeout(Duration::from_millis(300), &mut await_idle).await { + Err(_still_pending) => {} + Ok(Ok(Ok(()))) => panic!( + "the preempted A-era event — era-checked BEFORE the take, \ + dispatched AFTER the successor registered — satisfied the \ + successor's await_idle: the false freshAgent.turn.complete \ + precursor (ep2-r3 check-to-dispatch TOCTOU)" + ), + other => panic!("await_idle settled unexpectedly: {other:?}"), + } + + // Positive control: a GENUINE B-era idle through B's OWN sink still + // satisfies it — the gate is an era fence, not a broken dispatch. + mgr.arm_dispatch_park_for_tests(None); + sink_b(idle_event()); + let outcome = tokio::time::timeout(Duration::from_secs(2), await_idle) + .await + .expect("the genuine B-era idle resolves within budget"); + assert!( + matches!(outcome, Ok(Ok(()))), + "the successor's own idle edge must satisfy await_idle, got {outcome:?}" + ); + sink_thread.join().expect("the sink thread ends cleanly"); + } + /// Spawn-level: a config-supplied inline document is MERGED into the launch /// (sibling keys survive, snapshot pinned), never replaced — and the spawn env /// carries EXACTLY ONE occurrence (the merged value). diff --git a/crates/freshell-opencode/tests/serve_daemon_selfheal.rs b/crates/freshell-opencode/tests/serve_daemon_selfheal.rs new file mode 100644 index 000000000..bedc97c86 --- /dev/null +++ b/crates/freshell-opencode/tests/serve_daemon_selfheal.rs @@ -0,0 +1,1030 @@ +//! Manager-level daemon-loss self-heal (the 2026-09-20 incident's silent half): +//! a shared `opencode serve` daemon that dies on its own must not leave a +//! poisoned running entry forever. +//! +//! Pins the Task 3 machinery end-to-end through fully-faked IO (NO real serve, +//! NO live API calls — the `serve_health_bounded.rs` / `serve_idle_edge.rs` +//! conventions): +//! * the daemon exit watcher: an unrequested process exit clears the running +//! entry, emits `SessionSignal::Lost` for in-flight turns, broadcasts +//! `DaemonSignal::Lost { reason: "process_exit" }`, and schedules a re-warm; +//! * the re-warm retry loop: a FAILED re-warm attempt retries (never strands +//! the daemon absent), and the automatic respawn BACKS OFF exponentially +//! (no spawn storm); +//! * the requested-discard arm: a Yes-lane timeout discard also signals +//! `DaemonSignal::Lost` (its reason) and schedules the same backoff-guarded +//! respawn — the runtime self-heal (Task 4) observes BOTH loss classes; +//! * the stale-timeout gate: a Yes-lane timeout may only discard the daemon +//! the wedged request was ACTUALLY dispatched against — never its +//! replacement; +//! * the successor-sweep fence (ep2-r1 fresheyes Major): a loss cleanup +//! still running past its running-entry take must never claim session +//! emitters registered by the replacement daemon that cold-started in +//! the overlap — A's late cleanup wiped B's fresh sender, and B's bridge +//! dead-ended with no recovery trigger left. +//! +//! The crash-detected WARN (`freshagent.opencode.daemon_crash_detected`) is +//! pinned unit-side in `serve.rs` (the `config_capture` idiom), where the +//! thread-local tracing capture hosts it. + +use std::sync::atomic::{AtomicBool, AtomicU16, AtomicUsize, Ordering}; +use std::sync::Arc; +use std::time::Duration; + +use freshell_opencode::events::parse_serve_event; +use freshell_opencode::serve::{ + build_prompt_body, DaemonSignal, Endpoint, EventSink, EventSource, EventStreamHandle, + OpencodeServeManager, PortAllocator, ProcessSpawner, ServeConfig, ServeDeps, ServeError, + ServeHttp, ServeHttpError, ServeHttpRequest, ServeHttpResponse, ServeProcess, SessionSignal, + SpawnRequest, +}; +use serde_json::json; +use tokio::sync::broadcast::error::TryRecvError; + +// ── injected fakes ─────────────────────────────────────────────────────────────── + +/// `/global/health` answers healthy; `/prompt_async` NEVER resolves when +/// `prompt_pending` (the Yes-lane timeout discard driver); everything else a +/// benign 200 `{}`. +struct HealthyHttp { + prompt_pending: bool, +} +impl ServeHttp for HealthyHttp { + fn request<'a>( + &'a self, + req: ServeHttpRequest, + ) -> std::pin::Pin< + Box< + dyn std::future::Future> + Send + 'a, + >, + > { + if req.url.contains("/prompt_async") && self.prompt_pending { + // A genuine wedge: the response NEVER resolves — only the caller's + // per-request bound can settle it (the discard driver). + return Box::pin(async { + std::future::pending::<()>().await; + unreachable!() + }); + } + Box::pin(async { Ok(ServeHttpResponse::new(200, b"{}".to_vec())) }) + } +} + +/// Per-generation health scripting: probes to a port in `fail_ports` NEVER +/// resolve (the wedged shape — that generation's bounded health wait fails as +/// `NotHealthy`); every other request is healthy/benign. Ports come from +/// [`CountingAllocator`], so port number == spawn generation. +struct GenerationHealthHttp { + fail_ports: Vec, +} +impl ServeHttp for GenerationHealthHttp { + fn request<'a>( + &'a self, + req: ServeHttpRequest, + ) -> std::pin::Pin< + Box< + dyn std::future::Future> + Send + 'a, + >, + > { + let wedged = req.url.contains("/global/health") + && self + .fail_ports + .iter() + .any(|port| req.url.contains(&format!(":{port}/"))); + if wedged { + return Box::pin(async { + std::future::pending::<()>().await; + unreachable!() + }); + } + Box::pin(async { Ok(ServeHttpResponse::new(200, b"{}".to_vec())) }) + } +} + +/// Hand out ports 1, 2, 3, … — one per cold start, so the spawn generation is +/// addressable in the URL (see [`GenerationHealthHttp`]). +struct CountingAllocator { + next: AtomicU16, +} +impl PortAllocator for CountingAllocator { + fn allocate(&self) -> Result { + let port = self.next.fetch_add(1, Ordering::SeqCst) + 1; + Ok(Endpoint { + hostname: "127.0.0.1".into(), + port, + }) + } +} + +/// A serve that never exits; `kill()` counts. +struct NeverExitsProcess { + killed: Arc, +} +impl ServeProcess for NeverExitsProcess { + fn exited(&self) -> Option { + None + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) { + self.killed.fetch_add(1, Ordering::SeqCst); + } +} + +/// A serve whose "exit" is test-controlled: `exited()` reports `Some(0)` once +/// the shared flag is set (and stays Some — the dead daemon stays dead). +struct FlagExitProcess { + exited: Arc, + killed: Arc, +} +impl ServeProcess for FlagExitProcess { + fn exited(&self) -> Option { + self.exited.load(Ordering::SeqCst).then_some(0) + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) { + self.killed.fetch_add(1, Ordering::SeqCst); + } +} + +/// A serve that "dies" immediately after every successful start: the FIRST +/// `exited()` consult is `None` (the readiness wait's liveness check — the +/// health probe answers healthy on that same iteration), every later consult +/// reports the exit. Each instance serves exactly one spawn generation, so +/// every generation is healthy-then-dead one watch interval later. +struct DieAfterHealthProcess { + exited_consults: AtomicUsize, +} +impl ServeProcess for DieAfterHealthProcess { + fn exited(&self) -> Option { + let n = self.exited_consults.fetch_add(1, Ordering::SeqCst) + 1; + (n >= 2).then_some(0) + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) {} +} + +/// Hands every generation a [`FlagExitProcess`] sharing the flag/counters. +struct FlagExitSpawner { + exited: Arc, + killed: Arc, + spawns: Arc, +} +impl ProcessSpawner for FlagExitSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + self.spawns.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(FlagExitProcess { + exited: self.exited.clone(), + killed: self.killed.clone(), + })) + } +} + +/// Hands every generation a fresh [`DieAfterHealthProcess`]. +struct DieAfterHealthSpawner { + spawns: Arc, +} +impl ProcessSpawner for DieAfterHealthSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + self.spawns.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(DieAfterHealthProcess { + exited_consults: AtomicUsize::new(0), + })) + } +} + +/// Scripted topology: generation 1 is a test-controlled [`FlagExitProcess`] +/// (the healthy daemon the watcher arms on); every later generation is a +/// [`NeverExitsProcess`] — the later generations fail/succeed via the scripted +/// HTTP health, never via their own exit. +struct GenerationSpawner { + first: Arc, + killed: Arc, + spawns: Arc, +} +impl ProcessSpawner for GenerationSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + let n = self.spawns.fetch_add(1, Ordering::SeqCst) + 1; + if n == 1 { + Ok(Box::new(FlagExitProcess { + exited: self.first.clone(), + killed: self.killed.clone(), + })) + } else { + Ok(Box::new(NeverExitsProcess { + killed: self.killed.clone(), + })) + } + } +} + +/// Hands every generation a [`NeverExitsProcess`]; kills/spawns counted. +struct NeverExitsSpawner { + killed: Arc, + spawns: Arc, +} +impl ProcessSpawner for NeverExitsSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + self.spawns.fetch_add(1, Ordering::SeqCst); + Ok(Box::new(NeverExitsProcess { + killed: self.killed.clone(), + })) + } +} + +/// A serve whose `kill()` PARKS until the test drops the release sender: the +/// loss cleanup stops between the running-entry take (the lock is already +/// released) and its session-emitter sweep — the exact production window in +/// which a successor daemon's cold start + bridge registration can overlap +/// the still-pending cleanup. `kill_entered` counts the park so the test can +/// wait for the cleanup to reach it. +struct LatchedKillProcess { + exited: Arc, + kill_entered: Arc, + release: std::sync::Mutex>, +} +impl ServeProcess for LatchedKillProcess { + fn exited(&self) -> Option { + self.exited.load(Ordering::SeqCst).then_some(0) + } + fn take_fatal_startup_error(&self) -> Option { + None + } + fn kill(&self) { + self.kill_entered.fetch_add(1, Ordering::SeqCst); + // Park the cleanup; the test releases the latch by dropping its + // sender, which ends this recv with an error. Only kill() ever + // touches the receiver, so parking under the mutex is safe. + // block_in_place: recv() is a BLOCKING sync wait, and a tokio + // worker must never block directly — the worker servicing the + // runtime's timer driver would starve every sleep/timeout on the + // runtime (observed: the whole test freezes). block_in_place + // hands the worker's runtime duties off first. Requires the + // multi_thread flavor this test runs under. + tokio::task::block_in_place(|| { + let _ = self.release.lock().expect("latch mutex").recv(); + }); + } +} + +/// Generation 1 is the latched daemon A (the test holds the release sender); +/// every later generation is a [`NeverExitsProcess`] — the replacement B, +/// which this spawner's test never loses. +struct LatchedKillSpawner { + exited: Arc, + kill_entered: Arc, + spawns: Arc, + /// The gen-1 latch receiver, handed to the first spawned process. + release: std::sync::Mutex>>, +} +impl ProcessSpawner for LatchedKillSpawner { + fn spawn(&self, _req: SpawnRequest) -> Result, String> { + let n = self.spawns.fetch_add(1, Ordering::SeqCst) + 1; + if n == 1 { + let release = self + .release + .lock() + .expect("latch mutex") + .take() + .expect("the gen-1 latch receiver is present"); + Ok(Box::new(LatchedKillProcess { + exited: self.exited.clone(), + kill_entered: self.kill_entered.clone(), + release: std::sync::Mutex::new(release), + })) + } else { + Ok(Box::new(NeverExitsProcess { + killed: Arc::new(AtomicUsize::new(0)), + })) + } + } +} + +struct NoopEventHandle; +impl EventStreamHandle for NoopEventHandle {} + +struct NoopEventSource; +impl EventSource for NoopEventSource { + fn connect(&self, _url: String, _sink: EventSink) -> Box { + Box::new(NoopEventHandle) + } +} + +/// The Task 3 self-heal knobs: a tiny watch interval and a tiny backoff. +fn selfheal_config(watch_ms: u64, backoff_initial_ms: u64, backoff_max_ms: u64) -> ServeConfig { + ServeConfig { + daemon_watch_interval: Duration::from_millis(watch_ms), + re_warm_backoff_initial_ms: backoff_initial_ms, + re_warm_backoff_max_ms: backoff_max_ms, + ..ServeConfig::default() + } +} + +async fn started_manager(deps: ServeDeps, config: ServeConfig) -> OpencodeServeManager { + let mgr = OpencodeServeManager::new(deps, config); + mgr.ensure_started() + .await + .expect("healthy fake serve starts"); + mgr +} + +// ── tests ────────────────────────────────────────────────────────────────────── + +/// A daemon that dies on its own must not leave a poisoned running entry +/// forever (the 2026-09-20 incident's silent half): the watcher clears the +/// entry, emits Lost for in-flight turns, signals daemon loss, and schedules +/// the backoff-guarded respawn. +#[tokio::test] +async fn unrequested_daemon_exit_clears_running_emits_lost_and_signals() { + let exited = Arc::new(AtomicBool::new(false)); + let killed = Arc::new(AtomicUsize::new(0)); + let spawns = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + killed: killed.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(HealthyHttp { + prompt_pending: false, + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: Arc::new(NoopEventSource), + }; + let manager = started_manager(deps, selfheal_config(10, 5, 50)).await; + // Subscribe BEFORE the daemon dies: tokio broadcast does NOT replay + // history to late subscribers, so a post-loss subscribe would miss the + // edge (the contract Task 4's level-triggered design depends on). + let mut signals = manager.subscribe_daemon_signals(); + let mut idle = manager.subscribe("ses_a"); + + exited.store(true, Ordering::SeqCst); // the daemon "exits" + + let signal = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + signal, + DaemonSignal::Lost { + reason: "process_exit" + } + ), + "an unrequested daemon exit must signal Lost{{process_exit}}, got {signal:?}" + ); + assert!( + manager.base_url().await.is_none(), + "the dead daemon's running entry must be cleared" + ); + assert!( + matches!(idle.try_recv(), Ok(SessionSignal::Lost)), + "in-flight session subscribers must see the Lost edge" + ); + assert!( + killed.load(Ordering::SeqCst) >= 1, + "the already-exited daemon is still kill()ed for /proc-reaper parity" + ); +} + +/// Crash-loop guard: the automatic re-warm must BACK OFF exponentially, not +/// spawn-storm. A daemon that dies immediately after every successful start +/// drives a continuous loss→re-warm cycle; within the 700 ms window the +/// observed spawn count must show the 50→100→200→400 ms escalation (a +/// non-backed-off loop would spawn dozens; no re-warm at all would strand the +/// count at 1). +#[tokio::test] +async fn daemon_loss_re_warm_backs_off_exponentially() { + let spawns = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(DieAfterHealthSpawner { + spawns: spawns.clone(), + }), + http: Arc::new(HealthyHttp { + prompt_pending: false, + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: Arc::new(NoopEventSource), + }; + // The manager binding stays alive for the window: its watcher/re-warm + // tasks hold their own Arc clones, but keeping the binding makes the + // driving ownership explicit. + let _manager = started_manager(deps, selfheal_config(5, 50, 400)).await; + + tokio::time::sleep(Duration::from_millis(700)).await; + let observed = spawns.load(Ordering::SeqCst); + assert!( + (2..=6).contains(&observed), + "re-warm must respawn (>= 2) AND back off exponentially (<= 6) — \ + observed {observed} spawns in 700 ms with 50 ms initial / 400 ms max backoff" + ); +} + +/// A FAILED re-warm attempt must RETRY (the loop), never strand the daemon +/// permanently absent: spawn #1 is healthy and exits on demand; the re-warm's +/// spawns #2 and #3 fail their bounded health waits; spawn #4 is healthy. The +/// daemon must eventually come back, having spawned at least 4 times. +#[tokio::test] +async fn a_failed_re_warm_retries_until_the_daemon_starts() { + let spawns = Arc::new(AtomicUsize::new(0)); + let first_exited = Arc::new(AtomicBool::new(false)); + let killed = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(GenerationSpawner { + first: first_exited.clone(), + killed: killed.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(GenerationHealthHttp { + fail_ports: vec![2, 3], + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + health_timeout: Duration::from_millis(40), + health_probe_timeout: Duration::from_millis(10), + health_retry_interval: Duration::from_millis(5), + daemon_watch_interval: Duration::from_millis(5), + re_warm_backoff_initial_ms: 10, + re_warm_backoff_max_ms: 50, + ..ServeConfig::default() + }; + let manager = started_manager(deps, config).await; + let mut signals = manager.subscribe_daemon_signals(); + + first_exited.store(true, Ordering::SeqCst); // the healthy daemon "exits" + let signal = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + signal, + DaemonSignal::Lost { + reason: "process_exit" + } + ), + "got {signal:?}" + ); + + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if manager.base_url().await.is_some() { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("the re-warm loop must eventually succeed (fail, fail, then healthy)"); + assert!( + spawns.load(Ordering::SeqCst) >= 4, + "the failed re-warm attempts must be retried — observed {} spawns \ + (initial + 2 failed re-warms + success)", + spawns.load(Ordering::SeqCst) + ); +} + +/// A discard (the intentional kill path) must ALSO signal daemon loss (with +/// its reason) and schedule the re-warm, so the runtime self-heal (Task 4) +/// observes the requested-loss class the same as the crash class. Driven +/// through `prompt_async` — a deliberate `DiscardOnTimeout::Yes` lane. +#[tokio::test] +async fn discard_running_signals_daemon_loss_and_schedules_re_warm() { + let spawns = Arc::new(AtomicUsize::new(0)); + let killed = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(NeverExitsSpawner { + killed: killed.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(HealthyHttp { + prompt_pending: true, + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + request_timeout: Duration::from_millis(50), + daemon_watch_interval: Duration::from_millis(5), + re_warm_backoff_initial_ms: 20, + re_warm_backoff_max_ms: 100, + ..ServeConfig::default() + }; + let manager = started_manager(deps, config).await; + let mut signals = manager.subscribe_daemon_signals(); + + let err = manager + .prompt_async( + "ses_discard", + build_prompt_body("hi", None, None), + &None, + None, + ) + .await + .expect_err("the prompt POST must time out"); + assert!( + matches!(err, ServeError::RequestTimeout { .. }), + "got {err:?}" + ); + + let lost = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + lost, + DaemonSignal::Lost { + reason: "request_timeout" + } + ), + "a discard must signal its reason, got {lost:?}" + ); + let respawned = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("re-warm within budget") + .expect("channel alive"); + assert!( + matches!(respawned, DaemonSignal::Started), + "the discard's scheduled re-warm must respawn the daemon, got {respawned:?}" + ); + assert!( + spawns.load(Ordering::SeqCst) >= 2, + "a respawn happened after the backoff (observed {} spawns)", + spawns.load(Ordering::SeqCst) + ); + assert!( + manager.base_url().await.is_some(), + "the re-warmed daemon serves again" + ); +} + +/// Delta-r1 review Finding 1 (the stale-timeout discard race): a Yes-lane +/// timeout may only discard the daemon the timed-out request was ACTUALLY +/// dispatched against — never its replacement. Two staggered wedged prompt +/// POSTs both capture daemon A's base (port 1); R1's timeout legitimately +/// discards A and the re-warm installs the replacement B (port 2); R2 — +/// still wedged against the dead A — then times out and must NO-OP: B +/// survives (no kill, no second `Lost`, no third spawn), the same silence +/// as a `None` take. Pre-fix, R2's timeout took+killed whichever entry was +/// CURRENT — the innocent replacement — defeating stable self-healing. +#[tokio::test] +async fn stale_request_timeout_never_discards_the_replacement_daemon() { + let spawns = Arc::new(AtomicUsize::new(0)); + let killed = Arc::new(AtomicUsize::new(0)); + let deps = ServeDeps { + spawner: Arc::new(NeverExitsSpawner { + killed: killed.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(HealthyHttp { + prompt_pending: true, + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: Arc::new(NoopEventSource), + }; + let config = ServeConfig { + request_timeout: Duration::from_millis(100), + daemon_watch_interval: Duration::from_millis(5), + re_warm_backoff_initial_ms: 10, + re_warm_backoff_max_ms: 50, + ..ServeConfig::default() + }; + let manager = started_manager(deps, config).await; + let mut signals = manager.subscribe_daemon_signals(); + + // R1 dispatches against daemon A (port 1) and wedges. + let r1_manager = manager.clone(); + let r1 = tokio::spawn(async move { + r1_manager + .prompt_async("ses_r1", build_prompt_body("r1", None, None), &None, None) + .await + }); + // Stagger: R2 still dispatches against A (well inside R1's timeout). + tokio::time::sleep(Duration::from_millis(40)).await; + let r2_manager = manager.clone(); + let r2 = tokio::spawn(async move { + r2_manager + .prompt_async("ses_r2", build_prompt_body("r2", None, None), &None, None) + .await + }); + + // R1's timeout: A is discarded — the legitimate same-identity discard. + let r1_err = r1 + .await + .expect("r1 settles") + .expect_err("the r1 prompt POST must time out"); + assert!( + matches!(r1_err, ServeError::RequestTimeout { .. }), + "got {r1_err:?}" + ); + let lost = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + lost, + DaemonSignal::Lost { + reason: "request_timeout" + } + ), + "the same-identity discard must signal its loss, got {lost:?}" + ); + assert_eq!( + killed.load(Ordering::SeqCst), + 1, + "the discard killed exactly the wedged daemon A" + ); + // The re-warm installs the replacement daemon B (port 2). + let started = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("re-warm within budget") + .expect("channel alive"); + assert!( + matches!(started, DaemonSignal::Started), + "the re-warm must install the replacement, got {started:?}" + ); + assert_eq!( + manager.base_url().await, + Some("http://127.0.0.1:2".to_string()), + "the replacement daemon B owns the running entry" + ); + assert_eq!( + spawns.load(Ordering::SeqCst), + 2, + "exactly A and B have spawned so far" + ); + + // R2 — still wedged against the dead A — now times out. + let r2_err = r2 + .await + .expect("r2 settles") + .expect_err("the r2 prompt POST must time out"); + match r2_err { + ServeError::RequestTimeout { url, .. } => assert!( + url.contains("127.0.0.1:1"), + "R2 was dispatched against daemon A: {url}" + ), + other => panic!("expected a RequestTimeout, got {other:?}"), + } + + // THE assertion: the stale timeout must not touch B — no kill, no second + // Lost, no third spawn — the silence of a `None` take. + assert_eq!( + killed.load(Ordering::SeqCst), + 1, + "the stale timeout must NOT kill the replacement daemon B" + ); + assert_eq!( + manager.base_url().await, + Some("http://127.0.0.1:2".to_string()), + "the replacement daemon B must survive the stale timeout" + ); + match signals.try_recv() { + Err(tokio::sync::broadcast::error::TryRecvError::Empty) => {} + other => panic!("the stale timeout must not signal a second loss, got {other:?}"), + } + // A sneaky deferred re-warm would have spawned by now (the ladder's + // first rung is 10-20 ms with these knobs). + tokio::time::sleep(Duration::from_millis(60)).await; + assert_eq!( + spawns.load(Ordering::SeqCst), + 2, + "the stale timeout must not schedule another re-warm" + ); + assert_eq!( + manager.base_url().await, + Some("http://127.0.0.1:2".to_string()), + "B is still the running daemon after the settle window" + ); +} + +// ── ep2-r2 fresheyes Major: cross-generation event contamination ──────────────── + +/// An [`EventSource`] that RECORDS every sink it is handed (one per cold +/// start, in connect order) — the REAL per-connection dispatch closures +/// [`OpencodeServeManager`] mints at each daemon's connect, so a test can +/// dispatch a late event through the exact path the transport would, +/// carrying the CONNECTING daemon's identity. The ep2-r2 requirement: the +/// prior successor-emitter test dispatched through the generation-less +/// `dispatch_event` seam and could not observe this defect class at all. +struct RecordingEventSource { + sinks: std::sync::Mutex>, +} +impl EventSource for RecordingEventSource { + fn connect(&self, _url: String, sink: EventSink) -> Box { + self.sinks.lock().expect("recorded sinks mutex").push(sink); + Box::new(NoopEventHandle) + } +} + +/// A late event from a LOST daemon generation must NEVER reach the +/// successor era (ep2-r2 fresheyes Major — cross-generation event +/// contamination, the false-chime precursor). The review's interleaving, +/// forced deterministically: daemon A is lost through the REAL watcher +/// arm (take + sweep + Lost), the re-warm installs the successor B, a +/// B-era `await_idle` is subscribed and IN FLIGHT for a durable session — +/// and only then does A's connection sink (the real dispatch closure +/// minted at A's cold start, still alive in the taken `RunningServe`'s +/// SSE-handle window) deliver a buffered `session.idle`. It must NOT +/// satisfy the successor's `await_idle` — the exact event that would +/// falsely produce `freshAgent.turn.complete` and clear busy for a +/// still-running B turn (the no-chime-on-daemon-loss contract). A +/// genuine B-era idle delivered through B's OWN sink must still satisfy +/// it (the dispatch is generation-fenced, not broken). Pre-fix, the +/// sink was generation-less: the late A event dispatched straight into +/// the shared emitter map and the successor's await resolved Ok. +#[tokio::test] +async fn a_lost_daemons_late_events_never_satisfy_the_successors_await_idle() { + let exited = Arc::new(AtomicBool::new(false)); + let killed = Arc::new(AtomicUsize::new(0)); + let spawns = Arc::new(AtomicUsize::new(0)); + let events = Arc::new(RecordingEventSource { + sinks: std::sync::Mutex::new(Vec::new()), + }); + let deps = ServeDeps { + spawner: Arc::new(FlagExitSpawner { + exited: exited.clone(), + killed: killed.clone(), + spawns: spawns.clone(), + }), + http: Arc::new(HealthyHttp { + prompt_pending: false, + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: events.clone(), + }; + let manager = started_manager(deps, selfheal_config(10, 5, 50)).await; + let mut signals = manager.subscribe_daemon_signals(); + + // Daemon A's connection sink — the REAL dispatch closure, carrying A's + // daemon-generation identity. + let sink_a = events + .sinks + .lock() + .expect("recorded sinks mutex") + .first() + .expect("A's cold start connected its event stream") + .clone(); + + // Daemon A dies — the watcher arm's REAL loss path (take + sweep + + // Lost all complete before the Lost broadcast is observable here). + exited.store(true, Ordering::SeqCst); + let lost = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("loss signal within budget") + .expect("channel alive"); + assert!( + matches!( + lost, + DaemonSignal::Lost { + reason: "process_exit" + } + ), + "got {lost:?}" + ); + // The re-warm installs the successor daemon B — a NEW generation with + // its own connection sink. + exited.store(false, Ordering::SeqCst); + let started = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("re-warm within budget") + .expect("channel alive"); + assert!(matches!(started, DaemonSignal::Started), "got {started:?}"); + assert_eq!( + events.sinks.lock().expect("recorded sinks mutex").len(), + 2, + "fixture: exactly two daemon generations have connected" + ); + let sink_b = events + .sinks + .lock() + .expect("recorded sinks mutex") + .get(1) + .expect("B's cold start connected its event stream") + .clone(); + let idle_event = || { + parse_serve_event(&json!({ + "type": "session.idle", + "properties": { "sessionID": "ses_late" } + })) + .expect("parseable serve event") + }; + + // B's `await_idle`, subscribed and IN FLIGHT for the durable session + // (the successor-era registration the late event must not reach). + let rx = manager.subscribe("ses_late"); + let idle_manager = manager.clone(); + let mut await_idle = tokio::spawn(async move { + idle_manager + .await_idle("ses_late", rx, Duration::from_secs(5), None) + .await + }); + // Let the await enter its select loop. (Broadcast buffers the event + // for an existing subscriber either way, but a live loop makes the + // in-flight premise unambiguous.) + tokio::time::sleep(Duration::from_millis(50)).await; + + // THE LATE A-ERA EVENT: dispatched through A's REAL sink — after A's + // take, with B installed and B's await_idle in flight. On the + // pre-fix generation-less sink this buffered `session.idle` + // satisfied the successor's await. + sink_a(idle_event()); + + // It must NOT satisfy: the await stays pending through the grace + // window. + match tokio::time::timeout(Duration::from_millis(300), &mut await_idle).await { + Err(_still_pending) => {} + Ok(Ok(Ok(()))) => panic!( + "a late event from the LOST daemon generation satisfied the \ + successor's await_idle — the false freshAgent.turn.complete \ + precursor (ep2-r2 cross-generation contamination)" + ), + other => panic!("await_idle settled unexpectedly: {other:?}"), + } + + // A GENUINE B-era idle through B's OWN sink still satisfies it — the + // gate is a generation fence, not a broken dispatch. + sink_b(idle_event()); + let outcome = tokio::time::timeout(Duration::from_secs(2), await_idle) + .await + .expect("the genuine B-era idle resolves within budget"); + assert!( + matches!(outcome, Ok(Ok(()))), + "the successor's own idle edge must satisfy await_idle, got {outcome:?}" + ); +} + +// ── ep2-r1 fresheyes Major: A's late emitter cleanup vs. B's fresh sender ──────── + +/// A daemon-loss cleanup must NEVER remove session emitters registered by a +/// SUCCESSOR daemon (ep2-r1 fresheyes Major — A's late emitter cleanup wipes +/// B's fresh sender). The review's interleaving, forced deterministically: +/// daemon A is lost and its cleanup is HELD at the kill/reap step — the take +/// already happened and the `running` lock is free; while held, the +/// fenced-attach-shaped recovery cold-starts the replacement daemon B +/// (`ensure_started`, which broadcasts B's `Started` — it comes and goes +/// BEFORE A's `Lost`, exactly as in the finding) and registers a B-era +/// session sender in the SHARED emitter map (the bridge subscribe, +/// `spawn_serve_bridge`'s `manager.subscribe(&real_id)`). Releasing A's +/// cleanup must send `Lost` ONLY to A's pre-existing subscribers; B's fresh +/// sender must survive untouched and still work — a dispatch through the +/// shared map must reach its subscriber (the bridge stays live and +/// functional; no revival pass is needed). Pre-fix, A's late +/// `emit_lost_for_all` swept the WHOLE shared map — claiming B's fresh +/// sender, so B's bridge drained its closed channel and exited with no +/// recovery trigger left (the dead-ended pane this self-heal exists to +/// prevent). +/// +/// Multi-thread runtime: the parked `kill()` blocks inside +/// `block_in_place`, which requires the multi_thread flavor, and the test's +/// own cold-start/registration work needs the remaining workers. +#[tokio::test(flavor = "multi_thread")] +async fn daemon_loss_cleanup_never_sweeps_a_successor_daemons_emitters() { + let exited = Arc::new(AtomicBool::new(false)); + let kill_entered = Arc::new(AtomicUsize::new(0)); + let spawns = Arc::new(AtomicUsize::new(0)); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let deps = ServeDeps { + spawner: Arc::new(LatchedKillSpawner { + exited: exited.clone(), + kill_entered: kill_entered.clone(), + spawns: spawns.clone(), + release: std::sync::Mutex::new(Some(release_rx)), + }), + http: Arc::new(HealthyHttp { + prompt_pending: false, + }), + ports: Arc::new(CountingAllocator { + next: AtomicU16::new(0), + }), + events: Arc::new(NoopEventSource), + }; + // Huge backoff: A's own scheduled re-warm must stay parked for the whole + // test window — the successor start under proof is the overlapped + // fenced-attach-shaped `ensure_started` below, and the manager's re-warm + // must not race it for the cold-start slot. + let manager = started_manager(deps, selfheal_config(10, 60_000, 120_000)).await; + let mut signals = manager.subscribe_daemon_signals(); + // A-era registration: the subscriber that was already live when A died. + let mut rx_a = manager.subscribe("ses_a"); + + // Daemon A dies; the watcher's loss path takes A out of the running slot + // and parks at the kill/reap step. + exited.store(true, Ordering::SeqCst); + tokio::time::timeout(Duration::from_secs(2), async { + while kill_entered.load(Ordering::SeqCst) < 1 { + tokio::time::sleep(Duration::from_millis(2)).await; + } + }) + .await + .expect("the loss cleanup must reach its kill/reap step within the budget"); + assert!( + manager.base_url().await.is_none(), + "fixture: A is already taken out of the running slot while its cleanup is parked" + ); + + // The overlapped fenced-attach shape: with A's cleanup still parked, the + // recovery cold-starts the replacement daemon B (a NEW generation — its + // `Started` broadcasts now, BEFORE A's `Lost`, the finding's ordering). + let b_base = manager + .ensure_started() + .await + .expect("the replacement daemon B cold-starts while A's cleanup is parked"); + assert_eq!( + b_base, "http://127.0.0.1:2", + "B is a cold start on the next allocator port, not the fast path" + ); + // ... and its bridge registers the successor's session sender in the + // SHARED emitter map. + let mut rx_b = manager.subscribe("ses_b"); + + // Release A's parked cleanup: kill/reap A, sweep the session emitters, + // and signal the loss. + drop(release_tx); + + let first = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("a daemon signal within budget") + .expect("channel alive"); + assert!( + matches!(first, DaemonSignal::Started), + "B's cold start mid-cleanup must broadcast Started first, got {first:?}" + ); + let second = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await + .expect("a daemon signal within budget") + .expect("channel alive"); + assert!( + matches!( + second, + DaemonSignal::Lost { + reason: "process_exit" + } + ), + "A's released cleanup signals its loss, got {second:?}" + ); + + // A's Lost went to A's pre-existing subscriber — and cleared A's own + // emitter (its channel closes once the swept senders drop). + assert!( + matches!(rx_a.try_recv(), Ok(SessionSignal::Lost)), + "A's pre-existing subscriber must see the Lost edge" + ); + assert!( + matches!(rx_a.try_recv(), Err(TryRecvError::Closed)), + "A's own emitter must be cleared from the shared map" + ); + + // THE invariant: B's sender was registered by a SUCCESSOR — A's cleanup + // must never claim it. No Lost edge may arrive on it ... + match rx_b.try_recv() { + Err(TryRecvError::Empty) => {} + other => panic!( + "A's loss cleanup must never touch the successor daemon's fresh \ + sender — a post-take registration was swept (ep2-r1): got {other:?}" + ), + } + // ... and it must still WORK: a dispatch through the shared map still + // reaches the B-era subscriber (the bridge stays live and functional). + manager.dispatch_event( + parse_serve_event(&json!({ + "type": "session.idle", + "properties": { "sessionID": "ses_b" } + })) + .expect("parseable serve event"), + ); + let got = tokio::time::timeout(Duration::from_secs(2), rx_b.recv()) + .await + .expect("the dispatch must arrive on the successor's emitter within the budget"); + assert!( + matches!(got, Ok(SessionSignal::Event(_))), + "the successor daemon's emitter must still deliver events, got {got:?}" + ); + // Stability: no further daemon edges in the window (the huge backoff + // keeps the scheduled re-warm parked). + match signals.try_recv() { + Err(TryRecvError::Empty) => {} + other => panic!("no further daemon edges expected, got {other:?}"), + } +} diff --git a/docs/plans/2026-09-21-opencode-daemon-death-recovery.md b/docs/plans/2026-09-21-opencode-daemon-death-recovery.md new file mode 100644 index 000000000..dae4d7940 --- /dev/null +++ b/docs/plans/2026-09-21-opencode-daemon-death-recovery.md @@ -0,0 +1,1120 @@ +# OpenCode Daemon Death Recovery Implementation Plan + +> **For agentic workers:** Execute this plan task by task with a fresh +> implementer and a specification-plus-quality review after every task. Track +> progress with the checkbox steps below. + +## User Request + +### Requested result +Fix the freshopencode shared-daemon death incident class in the Freshell repo (Rust server + React client): (1) an opencode compact request timeout must no longer kill the shared `opencode serve` sidecar; (2) a daemon discard must emit a structured log; (3) daemon loss must self-heal at the freshopencode runtime level with a client-visible status edge and a backoff-guarded respawn, mirroring the freshcodex onExit self-heal; (4) the client must not dead-end on the fresh-agent snapshot 409 RESTORE_UNAVAILABLE — it must drive the documented generation-fenced attach recovery and refetch. + +### Explicit constraints +- The user explicitly requested the-usual workflow (plan, load-bearing validation, independent fresh-eyes reviews, TDD execution, recap). +- Work in a dedicated worktree under `.worktrees/`; branch from `origin/main`; PR only after explicit user approval; never push behavior changes to `main` directly. +- Red/Green/Refactor TDD; ensure unit and e2e coverage; never reduce test coverage to get tests passing. +- Never restart the self-hosted production Freshell server on port 3001 without the user's explicit "APPROVED". +- TypeScript NodeNext relative imports require `.js` extensions. +- Follow repo test-coordination rules (coordinated broad runs, base-gate for green-base checks). + +### Accepted tradeoffs and residuals +- The user approved the proposed fix set "presumably step 1-4, unless analysis reveals otherwise": planning analysis may adjust the exact fix set if evidence shows a different cut is more idiomatic, but the four identified defects are the baseline scope. + +**Goal:** A freshopencode pane survives — and automatically recovers from — the loss of the shared `opencode serve` daemon, and no single slow request can kill that daemon for every session again. + +**Architecture:** Four layers, each mirroring an established in-repo precedent. (1) The compact request lane adopts the b8ke FR2 captured-base + `DiscardOnTimeout::No` pattern already used by `get_session_at`/`list_messages_at`/`abort_at`, so a timed-out summarize POST returns `RequestTimeout` without touching the shared daemon. (2) `discard_running` logs a structured WARN with its reason (the parameter already exists, unused). (3) The serve manager gains a daemon-level exit watcher + loss signal channel + backoff-guarded automatic re-warm (mirroring freshcodex `spawn_exit_watcher`), and the freshopencode runtime subscribes one listener task that fans a typed `freshAgent.error{OPENCODE_DAEMON_LOST}` edge to every materialized session and, after a successful respawn, restarts dead serve bridges and pushes idle snapshot edges. (4) The client's `handleSnapshotError` gains a 409 `RESTORE_UNAVAILABLE` arm that drives the documented recovery — one generation-fenced `freshAgent.attach` per pane identity plus a snapshot refetch — instead of dead-ending at a dismiss-only banner. + +**Tech Stack:** Rust (tokio, axum, tracing; crates `freshell-opencode`, `freshell-freshagent`), TypeScript/React (Redux Toolkit, Zod, Vitest + Testing Library, Playwright). + +## Global Constraints + +- All work happens in the worktree `.worktrees/opencode-daemon-death-recovery` on branch `the-usual/opencode-daemon-death-recovery` (base `855dae72a`). Never commit on `main`. +- The production self-hosted server on port 3001 must not be restarted without explicit user "APPROVED". All verification runs against locally spawned test servers or in-process harnesses only. +- The snapshot threads-route 409 envelope is a frozen contract: `status:"error"`, `code:"RESTORE_UNAVAILABLE"`, message `"Session is still running on the server."` are load-bearing (pinned by `snapshot.rs:1151` `opencode_cold_get_owned_or_transitioning_answers_the_typed_409`; client regexes depend on the text). Additive fields are allowed; changing/removing pinned fields is not. +- The snapshot GET stays side-effect-free: never spawn, kill, or discard from `get_opencode_snapshot` (pinned by `snapshot.rs:1070` and `lib.rs:7474`/`lib.rs:7529`). +- `ServeError::RequestTimeout` must stay OUTSIDE `never_dispatched()` (a timed-out POST may have reached the daemon — the compact redo-destroy stands; serve.rs:539-551 pins this forever). +- Structured logging: `tracing` macros, dotted event name as the message, structured fields (schema: `freshell-server/src/logging.rs`). New event names follow the `freshagent.opencode.*` family (existing: `freshagent.opencode.compact_failed`, `freshagent.opencode.handoff_stop_abort_undelivered`). +- The shared daemon is NEVER a per-session kill target (`freshAgent.kill` stays session-scoped; lib.rs:2318-2325 "the shared opencode serve daemon is NOT the per-session writer and must NEVER be killed" — that invariant refers to the ownership watchdog; the manager's own discard/re-warm lifecycle is the exception this plan carefully rebuilds). +- Client a11y: no new interactive elements without labels/roles; the recovery reuses existing banner/card components, so no new a11y surface should be introduced. +- Rust: `cargo fmt --all --check` and `cargo clippy --workspace --exclude freshell-tauri --all-targets -- -D warnings` must stay clean (pre-push gate). +- TypeScript: `npm run typecheck` clean; relative imports in NodeNext contexts need `.js` extensions (the client uses `@/` aliases). +- Focused test commands (delegated, non-coordinated — safe for TDD loops): + - `cargo test -p freshell-opencode` + - `cargo test -p freshell-freshagent opencode_ws::tests` + - `cargo test -p freshell-freshagent get_opencode_snapshot` (the lib.rs FR2 pins; crate-root tests are named `tests::...`, so filter by test-name substring — `lib::tests` matches nothing) + - `npm run test:vitest -- run test/unit/client/components/fresh-agent/FreshAgentView.test.tsx` + - `npm run test:e2e:local -- --project=chromium test/e2e-browser/specs/.ts` +- Broad/coordinated runs (`npm test`, `test:server` zero-arg, `test:integration` zero-arg) go through the shared coordinator; wait for a free gate, never kill a foreign holder. + +### Deliberate residuals (documented, out of scope) + +- `prompt_async` (the send-turn POST) and the thin `json_request` wrappers (`get_session`, `list_messages`, `get_session_status_map`, `abort`, `fork`, `revert`, `unrevert`) KEEP `DiscardOnTimeout::Yes`. Rationale: they are the deliberate wedged-daemon recycler for writes (FR2 kept Yes for writes on purpose), and the incident class was compact-specific (an LLM-scale budget routinely exceeded by a healthy-but-busy daemon). A wedged-alive daemon is also caught by the new exit-watcher only if it exits; a hung-but-alive daemon remains the send-lane's recycle responsibility. Revisit only with a dedicated wedged-detection design. +- The frozen 409 message text stays (even when the Live owner's daemon is dead, the text says "still running on the server" — clients' muscle memory depends on it). The new `OPENCODE_DAEMON_LOST` runtime edge is what tells the user the truth. +- **Terminal-owner 409s are a different scenario from the incident** (log-validated in the load-bearing stage: the incident-time key was `Live{FreshAgent, gen 1}` — the pane's own stale claim; the later-observed `ownerKind:"terminal", ownerGeneration:2` body was the post-salvage handoff state, minted ~2h17m after the daemon died). For a genuine terminal owner the fenced attach is refused by design, and the existing recovery doors are the session-directory handoff (the door the user actually used to salvage the session) or the owning terminal's exit. The client 409 recovery (Task 5) is scoped to `ownerKind:"fresh-agent"` — the stale-own-claim class the incident actually was. +- **The validated core gap for the incident state** (LB-05, falsified): a map-hit fenced attach re-subscribes the serve bridge but never respawns the daemon — `spawn_serve_bridge` never calls `ensure_started`, and `ensure_manager` returns the discarded manager. The attach was exercised 3× in the incident and recovered nothing. Task 4 therefore adds `ensure_started` to the attach tail's bridge-restart arm (mirroring what `resume_durable_session` already does for map-misses), turning the fenced attach into a real recovery verb for the map-hit daemon-dead state. +- After Task 1, a timed-out compact leaves the summarize turn running daemon-side until its own ~600 s budget expires or an interrupt arrives; the pane's busy state settles via the existing await-idle/turn-settle machinery. +- Runtime watcher arming (Task 4): armed from the WS materializing handlers (handle_send/handle_attach/handle_compact) plus an immediate level pass at arming (revive dead bridges if the daemon is already running). Residual: a REST-only, never-viewed pane misses the `OPENCODE_DAEMON_LOST` banner until its first WS interaction — accepted, because the pane renders nothing until viewed, and revival is level-triggered. +- E2E daemon-death/respawn coverage runs in TWO lanes (plan-review round 1): Task 6 adds the cloud-legal client-recovery spec (routed-fetch pattern — it satisfies the configured-cloud-backend PR gate), and Task 7 adds the real-daemon self-heal spec on the LOCAL lane, modeled on the existing `freshopencode-restart-recovery.spec.ts` harness (real RustServer + fake-opencode on PATH). The real-daemon spec lands in `CLOUD_SKIP_SPECS` (same provider-lifecycle-timing class as its model), so the cloud gate's e2e coverage is carried by Task 6 while the end-to-end server lanes (process-exit detection, backoff respawn, bridge revival, status edge) are proven by Task 7 locally plus the Rust unit tests (the same coverage strategy the freshcodex self-heal uses). + +--- + +### Task 1: Compact (and its pre-flight config read) no longer kill the shared daemon on timeout + +**Files:** +- Modify: `crates/freshell-opencode/src/serve.rs` (`compact()` at ~:1193-1232, `get_config()` at ~:1169, `json_request_maybe_witnessed` at ~:864-888 stays untouched) +- Test: `crates/freshell-opencode/src/serve.rs` `#[cfg(test)] mod tests` (beside `compact_uses_the_dedicated_compact_timeout_not_the_generic_request_bound`, ~:2083) + +**Interfaces:** +- Consumes: `json_request_over_base(method, path, body, not_found_value, base: String, discard_on_timeout: DiscardOnTimeout, dispatch_witnesses, timeout_override)` (serve.rs:890), `require_base()` (serve.rs:845), `DiscardOnTimeout` (serve.rs:345-355). +- Produces: unchanged public signatures for `compact()` / `get_config()` — the change is internal lane behavior only. Later tasks rely on: "a compact timeout does not clear the manager's running entry". + +**Behavior:** `compact` becomes the FR2 shape for writes: resolve the base once (`require_base()` — spawn-on-demand is preserved), then POST `/session/{id}/summarize` through `json_request_over_base` with `DiscardOnTimeout::No` and the dedicated `compact_timeout`. `get_config` (a read, and the compact drive's pre-flight model-pair resolution at opencode_ws.rs:3675-3696) gets the same treatment — a slow GET must never kill the shared daemon (the FR2 doc rule for reads; today it violates it). All other lanes keep their current discard policy (see residuals). + +- [x] **Step 1: Write the failing behavioral test** + +Add to the `#[cfg(test)] mod tests` module in serve.rs, reusing the existing fakes (`started_recording_manager_with_config`, `NeverExitsProcess` with its `killed: Arc` counter, and a recording HTTP fake scripted so health answers 200 and `/summarize` never resolves — the wedged shape from `tests/serve_health_bounded.rs:78` (`std::future::pending()`), exposed through a per-URL scripting seam like `RecordingHttp` at serve.rs:1781-1871): + +```rust +// 2026-09-20 incident: a compact timeout (600 s budget) ran the +// DiscardOnTimeout::Yes arm and KILLED the one shared `opencode serve` +// daemon for every freshopencode session. The compact lane must degrade +// like the FR2 snapshot lane: the POST times out, the daemon survives. +#[tokio::test] +async fn compact_timeout_does_not_kill_the_shared_daemon() { + let killed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let mut config = ServeConfig::default(); + config.compact_timeout = std::time::Duration::from_millis(50); + let (manager, http) = started_recording_manager_with_config( + /* summarize always times out: script `/summarize` responses to hang */ + RecordingHttpScript::SummarizePending, + config, + killed.clone(), + ); + let err = manager + .compact("ses_timeout", "anthropic", "claude-sonnet-4-5", &route_for("/w"), None, None) + .await + .expect_err("the summarize POST must time out"); + assert!(matches!(err, ServeError::RequestTimeout { .. }), "got {err:?}"); + assert_eq!( + killed.load(std::sync::atomic::Ordering::SeqCst), + 0, + "a compact timeout must NEVER kill the shared daemon" + ); + assert!( + manager.base_url().await.is_some(), + "the running entry must survive a compact timeout" + ); + // The compact-timeout POST must still carry the dedicated budget. + let summarize_index = http.index_of("POST", "/session/ses_timeout/summarize"); + assert_eq!(http.recorded_timeout(summarize_index), Some(std::time::Duration::from_millis(50))); +} + +// The compact drive's pre-flight model-pair resolution reads /config; a slow +// config GET is the same defect class (a read must never kill the daemon). +#[tokio::test] +async fn get_config_timeout_does_not_kill_the_shared_daemon() { + let killed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let mut config = ServeConfig::default(); + config.request_timeout = std::time::Duration::from_millis(50); + let (manager, _http) = started_recording_manager_with_config( + RecordingHttpScript::ConfigPending, + config, + killed.clone(), + ); + let err = manager.get_config().await.expect_err("config GET must time out"); + assert!(matches!(err, ServeError::RequestTimeout { .. })); + assert_eq!(killed.load(std::sync::atomic::Ordering::SeqCst), 0, + "a config read timeout must NEVER kill the shared daemon"); + assert!(manager.base_url().await.is_some()); +} +``` + +(Draft: adapt the fake construction to the actual `RecordingHttp`/`started_recording_manager*` signatures — the fakes already record per-request timeouts at serve.rs:1808-1817; extend their URL scripting to include a never-resolving `/summarize` and `/config` arm if not already scriptable. The two assertions that matter are `killed == 0` and `base_url().is_some()` after `Err(RequestTimeout)`.) + +- [x] **Step 2: Run the test and verify the intended failure** + +Run: `cargo test -p freshell-opencode does_not_kill` + +Expected: FAIL — `compact_timeout_does_not_kill_the_shared_daemon` fails with `killed: 1` (the discard-on-timeout arm killed the process) and/or `base_url()` is `None`; `get_config_timeout_does_not_kill_the_shared_daemon` fails the same way. (Single positional filter: cargo test accepts ONE TESTNAME — `does_not_kill` matches both new tests.) + +- [x] **Step 3: Add the minimal production implementation** + +In `crates/freshell-opencode/src/serve.rs`: + +```rust +/// `getConfig` for the compact drive's model-pair resolution (and any other +/// routed config read). Signature UNCHANGED (`route: &Route` stays — the +/// runtime calls `manager.get_config(&route)` for project-scoped config, and +/// the path must be `with_route("/config", route)`). A slow config read must +/// never kill the shared daemon — the FR2 read rule (b8ke): capture the base +/// once (spawn-on-demand is fine), then transport over the captured base +/// with `DiscardOnTimeout::No`. +pub async fn get_config(&self, route: &Route) -> Result { + let base = self.require_base().await?; + self.json_request_over_base( + HttpMethod::Get, &with_route("/config", route), None, None, + base, + DiscardOnTimeout::No, + &[], + None, + ).await +} +``` + +and inside `compact()` (serve.rs:1193-1232), replace the `json_request_maybe_witnessed(...)` call with the captured-base form (keep the exact path/body/witness/timeout logic): + +```rust +// 2026-09-20 incident: the summarize POST used the discard-on-timeout lane, +// so a 600 s budget exceeded on a healthy-but-busy daemon KILLED the one +// shared daemon for every freshopencode session. Mirror the FR2 captured-base +// transport (`get_session_at`, serve.rs:1003): a timed-out compact answers +// `RequestTimeout` and NEVER kills the shared daemon. The redo-destroy +// classification is unchanged — `RequestTimeout` stays outside +// `never_dispatched()` (a timed-out POST may have reached the daemon). +let base = self.require_base().await?; +self.json_request_over_base( + HttpMethod::Post, &path, Some(body), None, + base, + DiscardOnTimeout::No, + &witnesses, + Some(self.config().compact_timeout), +).await?; +Ok(()) +``` + +- [x] **Step 4: Run the focused test** + +Run: `cargo test -p freshell-opencode does_not_kill` + +Expected: PASS + +- [x] **Step 5: Refactor while green** + +None needed beyond the above (the change is already the FR2 mirror; `json_request_maybe_witnessed` remains for the lanes that intentionally keep `Yes`). + +- [x] **Step 6: Run impacted-test verification** + +Impacted set: all `freshell-opencode` tests (compact family, config, timeout plumbing) plus the `freshell-freshagent` compact-drive tests (the redo-destroy family pins `never_dispatched` classification and the `compact_failed` WARN — unchanged by this fix, but they exercise the compact lane end-to-end). + +Run: `cargo test -p freshell-opencode && cargo test -p freshell-freshagent compact` + +Expected: PASS (a pre-existing-failure comparison against the baseline ledger is not needed; baseline is green). + +- [x] **Step 7: Commit the task** + +```bash +git add crates/freshell-opencode/src/serve.rs +git commit -m "fix(opencode): compact and config timeouts never kill the shared serve daemon" +``` + +--- + +### Task 2: Daemon discards emit a structured log + +**Files:** +- Modify: `crates/freshell-opencode/src/serve.rs` (`discard_running` at ~:1348-1354) +- Test: `crates/freshell-opencode/src/serve.rs` `#[cfg(test)]` (beside the `config_capture` tracing-capture module, ~:2451-2502) + +**Interfaces:** +- Consumes: the `config_capture` thread-local tracing capture idiom (serve.rs:2451-2502, the DIAG-01 pattern), `prompt_async` (serve.rs:1091 — a lane that intentionally keeps `DiscardOnTimeout::Yes`). +- Produces: `discard_running(reason: &str)` logs `tracing::warn!(reason = ..., "freshagent.opencode.daemon_discarded")` before the kill. Task 3 builds on this exact site. + +- [x] **Step 1: Write the failing behavioral test** + +```rust +// 2026-09-20 incident: the daemon discard that killed the shared serve left +// ZERO log trace (discard_running's reason parameter is unused). The discard +// must be observable in the structured JSONL log. +#[tokio::test] +async fn discard_running_emits_a_structured_warn_with_its_reason() { + let capture = config_capture::capture(); + let mut config = ServeConfig::default(); + config.request_timeout = std::time::Duration::from_millis(50); + let (manager, _http) = started_recording_manager_with_config( + RecordingHttpScript::PromptPending, // a Yes-lane request that times out + config, + Default::default(), + ); + let err = manager.prompt_async(/* minimal args as in existing prompt tests */).await + .expect_err("prompt POST must time out"); + assert!(matches!(err, ServeError::RequestTimeout { .. })); + let events = capture.finish(); + let discard = events.iter().find(|e| e.get("message") + .map(|m| m.as_str() == Some("freshagent.opencode.daemon_discarded")).unwrap_or(false)) + .expect("a daemon discard must emit freshagent.opencode.daemon_discarded"); + assert_eq!(discard.get("reason").and_then(|r| r.as_str()), Some("request_timeout")); +} +``` + +(Draft: adapt to the actual `config_capture` helper API and `prompt_async` minimal-args shape used by `run_turn_arms_the_accepted_witness_at_the_dispatch_boundary` at serve.rs:1982. If `config_capture` needs the event on the test's own thread, note `#[tokio::test]` runs current-thread — `discard_running` executes inline on it, so the capture sees it.) + +- [x] **Step 2: Run the test and verify the intended failure** + +Run: `cargo test -p freshell-opencode discard_running_emits` + +Expected: FAIL — no `freshagent.opencode.daemon_discarded` event is captured (discard_running is tracing-silent today). + +- [x] **Step 3: Add the minimal production implementation** + +```rust +async fn discard_running(&self, reason: &str) { + let taken = self.inner.running.lock().await.take(); + if let Some(running) = taken { + tracing::warn!( + reason = reason, + "freshagent.opencode.daemon_discarded" + ); + running.process.kill(); + } + self.emit_lost_for_all(); +} +``` + +- [x] **Step 4: Run the focused test** + +Run: `cargo test -p freshell-opencode discard_running_emits` + +Expected: PASS + +- [x] **Step 5: Refactor while green** + +None (single-site change; keep the underscore removal as the whole diff). + +- [x] **Step 6: Run impacted-test verification** + +Impacted set: the whole `freshell-opencode` unit suite (tracing capture tests assert event sets; adding an event could affect any test asserting exact event streams — none do outside `config_capture`). + +Run: `cargo test -p freshell-opencode` + +Expected: PASS + +- [x] **Step 7: Commit the task** + +```bash +git add crates/freshell-opencode/src/serve.rs +git commit -m "feat(opencode): structured log for shared-daemon discards" +``` + +--- + +### Task 3: Manager-level daemon-loss machinery — exit watcher, loss signal, backoff re-warm + +**Files:** +- Modify: `crates/freshell-opencode/src/serve.rs` (Inner at ~:643-655, `RunningServe` at ~:634-641, `ensure_started` at ~:690-770, `discard_running` at ~:1348, `ServeConfig` at ~:557-603, `shutdown` at ~:1510-1517) +- Modify: `crates/freshell-opencode/src/lib.rs` (plan-review round 3: the crate uses an explicit `pub use serve::{...}` re-export list — `DaemonSignal` must be added there or Task 4's root-level import cannot compile) +- Test: `crates/freshell-opencode/src/serve.rs` `#[cfg(test)]` + a new integration file `crates/freshell-opencode/tests/serve_daemon_selfheal.rs` (follows `serve_idle_edge.rs` / `serve_health_bounded.rs` conventions) + +**Interfaces:** +- Consumes: `ServeProcess::exited()` (serve.rs:404-411), `emit_lost_for_all` (serve.rs:1332), Task 2's log site. +- Produces (used by Task 4 and later tests): + - `pub enum DaemonSignal { Lost { reason: &'static str }, Started }` (crate-root re-export) + - `OpencodeServeManager::subscribe_daemon_signals(&self) -> tokio::sync::broadcast::Receiver` (capacity 16; NOTE: tokio broadcast does NOT replay history to late subscribers — late `subscribe()` starts at the tail, so Task 4's watcher design is level-triggered, not event-history-dependent) + - `ServeConfig` gains: `daemon_watch_interval: Duration` (default 1000 ms), `re_warm_backoff_initial_ms: u64` (default 2000), `re_warm_backoff_max_ms: u64` (default 60_000). + - `RunningServe` gains an additive `ownership_id: String` field (currently only a local in `ensure_started`), and `process` becomes `Arc` (LB-06: the watcher needs a handle that outlives the entry; share the Arc, never move the Box). + - Semantics: `Started` is broadcast on every successful cold start (not on fast-path returns of an already-running daemon); `Lost{reason}` on discard (`"request_timeout"` today) and on unrequested process exit (`"process_exit"`). The shared loss path is exactly-once (LB-07): only the arm whose running-entry take yields `Some` logs/signals/schedules; a `None` take is a silent no-op (the watcher-vs-discard race resolves via the take). + +**Behavior:** +1. `ensure_started` spawns a daemon exit-watcher task after a successful health check (store its abort handle on `RunningServe` as `_exit_watch`). The watcher polls `process.exited()` every `daemon_watch_interval`; on `Some(exit)` it verifies the running entry is still ITS daemon (compare the captured `ownership_id`), then runs the manager's loss path. +2. The loss path (shared by watcher-exit and, minus the abort, by `discard_running`): WARN `freshagent.opencode.daemon_crash_detected` (watcher arm; fields `reason="process_exit"`, `base_url`) or the Task-2 discard WARN; take the running entry (killing it in the watcher arm is unnecessary — the process already exited; still call `process.kill()` for the /proc ownership reaper parity); `emit_lost_for_all()`; broadcast `DaemonSignal::Lost{reason}`; schedule a backoff-guarded re-warm. **Exactly-once (LB-07):** the take is the race arbiter — if it yields `None` (the other arm already ran), the whole path is a silent no-op: no log, no Lost, no re-warm. +3. Re-warm: a spawned retry loop sleeps `min(re_warm_backoff_initial_ms * 2^(attempts-1), re_warm_backoff_max_ms)`, then calls `ensure_started()`; a FAILED attempt schedules the next (escalating backoff), a SUCCESSFUL attempt exits the loop. `attempts` is an `AtomicUsize` on `Inner`, incremented per attempt. Reset semantics (corrected post-execution to the implemented D2 behavior — the executed authority is Task 3's commit `900742d1a` and its test `daemon_loss_re_warm_backs_off_exponentially`): the counter is NOT reset on re-warm success — reset-on-success voids the crash-loop escalation (a daemon dying immediately after every successful start would respawn at the initial delay forever, failing this task's own backoff test bound). The counter persists across re-warm successes and resets only at the NEXT loss, gated on the lost daemon having outlived the full ladder cap (`re_warm_backoff_max_ms`) — so each new incident starts at the initial delay while a crash-looping daemon still escalates to and retries at the max interval forever (self-heals when e.g. disk frees). Shutdown-flag checked per iteration. Log `tracing::info!(attempt = ..., "freshagent.opencode.daemon_re_warm")` on success and `tracing::warn!(..., error = ...)` on failure. +4. `discard_running` aborts the watcher FIRST (requested kill — no crash event), then the existing kill+lost, then `Lost` signal + re-warm schedule. +5. `shutdown`'s inline duplicate (serve.rs:1510-1517) also aborts the watcher; it must NOT schedule a re-warm (shutdown flag blocks it) and need not signal (server is going down) — keep it minimal: abort watcher + existing behavior. + +- [x] **Step 1: Write the failing behavioral tests** + +New integration file `crates/freshell-opencode/tests/serve_daemon_selfheal.rs` (drafts; adapt fakes from `serve_health_bounded.rs:41-127` — add an `ExitingProcess` fake whose `exited()` flips to `Some(0)` after the test sets a shared flag, plus a kill counter): + +```rust +// A daemon that dies on its own must not leave a poisoned running entry +// forever (the 2026-09-20 incident's silent half): the watcher clears the +// entry, emits Lost for in-flight turns, signals daemon loss, and schedules +// a backoff-guarded respawn. +#[tokio::test] +async fn unrequested_daemon_exit_clears_running_emits_lost_and_signals() { + let exiting = Arc::new(AtomicBool::new(false)); + let killed = Arc::new(AtomicUsize::new(0)); + let spawner = FakeSpawner::with_process(ExitingProcess::new(exiting.clone(), killed.clone())); + let mut config = ServeConfig::default(); + config.daemon_watch_interval = Duration::from_millis(10); + config.re_warm_backoff_initial_ms = 5; + let manager = started_manager_with(spawner, config); // health 200 fake + let mut signals = manager.subscribe_daemon_signals(); + let mut idle = manager.subscribe("ses_a").expect("subscribable"); + exiting.store(true, Ordering::SeqCst); // the daemon "exits" + let signal = tokio::time::timeout(Duration::from_secs(2), signals.recv()) + .await.expect("loss signal within budget").expect("channel alive"); + assert!(matches!(signal, DaemonSignal::Lost { reason: "process_exit" })); + assert!(manager.base_url().await.is_none(), "running entry must be cleared"); + assert!(matches!(idle.try_recv(), Ok(SessionSignal::Lost)), "in-flight subscribers must see Lost"); + // ...assert the WARN freshagent.opencode.daemon_crash_detected via the + // tracing capture if the integration file can host it (else assert in unit tests). +} + +// Crash-loop guard: the automatic re-warm must back off exponentially, not +// spawn-storm. +#[tokio::test] +async fn daemon_loss_re_warm_backs_off_exponentially() { + // Process that "dies" immediately after every successful start: + let spawns = Arc::new(AtomicUsize::new(0)); + let spawner = FakeSpawner::with_process(ExitAfterHealthProcess::new(spawns.clone())); + let mut config = ServeConfig::default(); + config.daemon_watch_interval = Duration::from_millis(5); + config.re_warm_backoff_initial_ms = 50; + config.re_warm_backoff_max_ms = 400; + let manager = started_manager_with(spawner, config); + manager.ensure_started().await.expect("first start"); + tokio::time::sleep(Duration::from_millis(700)).await; + let observed = spawns.load(Ordering::SeqCst); + // With 50ms initial doubling to 400ms cap: expected spawns within 700ms + // of the first loss are ~3-4. Un-backed-off would be dozens. Assert a + // conservative bound: + assert!(observed <= 6, "re-warm must back off (observed {observed} spawns)"); +} + +// Plan-review round 1: a failed re-warm attempt must RETRY (the loop), not +// give up — a transient spawn/health failure (e.g. disk pressure) must not +// leave the daemon permanently absent until unrelated user activity. +// Plan-review round 2: the exit watcher arms only after a SUCCESSFUL health +// check — so the test must first start a HEALTHY daemon, make THAT daemon +// exit (the watcher's loss path schedules the re-warm), and only then script +// the re-warm attempts to fail-fail-succeed. +#[tokio::test] +async fn a_failed_re_warm_retries_until_the_daemon_starts() { + let spawns = Arc::new(AtomicUsize::new(0)); + // Scripted topology: spawn #1 healthy (watcher armed) and exits on demand; + // re-warm spawns #2 and #3 fail health; spawn #4 is healthy. + let (manager, http) = started_manager_with( + HealthyThenDyingThenFailFailThenHealthy::new(spawns.clone()), + ServeConfig { + daemon_watch_interval: Duration::from_millis(5), + re_warm_backoff_initial_ms: 10, + re_warm_backoff_max_ms: 50, + ..ServeConfig::default() + }); + manager.ensure_started().await.expect("first start is healthy"); + make_fake_daemon_exit(&manager).await; // the unrequested exit the watcher detects + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if manager.base_url().await.is_some() { break; } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }).await.expect("the re-warm loop must eventually succeed (fail, fail, then healthy)"); + assert!(spawns.load(Ordering::SeqCst) >= 4, + "the failed re-warm attempts must be retried (observed {} spawns: initial + 2 failures + success)", + spawns.load(Ordering::SeqCst)); +} + +// A discard (the intentional kill path) must ALSO signal daemon loss so the +// runtime self-heal (Task 4) observes it. +#[tokio::test] +async fn discard_running_signals_daemon_loss_and_schedules_re_warm() { + // prompt-timeout discard (Yes-lane), then: + // - DaemonSignal::Lost { reason: "request_timeout" } arrives + // - after the backoff, a respawn happened (spawner count grew) +} +``` + +- [x] **Step 2: Run the tests and verify the intended failure** + +Run: `cargo test -p freshell-opencode --test serve_daemon_selfheal` + +Expected: FAIL — `subscribe_daemon_signals` does not exist (compile error is the intended missing behavior; write the enum + stub method returning a channel that never signals if needed to make it a runtime red instead — prefer the compile-first red, then a minimal stub for a runtime red on the watcher semantics). + +- [x] **Step 3: Add the minimal production implementation** + +In serve.rs (sketch — the implementer adapts to the actual Inner/ensure_started structure; LB-06/LB-07 corrections applied: the watcher holds an `Arc` clone of the process + the ownership id, never a moved Box): + +```rust +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum DaemonSignal { Lost { reason: &'static str }, Started } + +// Inner gains: +// daemon_signals: tokio::sync::broadcast::Sender, (capacity 16, on construction) +// re_warm_attempts: std::sync::atomic::AtomicUsize, +// RunningServe gains: +// ownership_id: String, +// process: Arc, // was Box — LB-06 +// _exit_watch: Option, + +// ensure_started, after storing RunningServe (the cold-start success path): +let watch = self.spawn_exit_watch(base_url.clone(), Arc::clone(&process), ownership_id.clone()); +running._exit_watch = watch; +let _ = self.inner.daemon_signals.send(DaemonSignal::Started); + +// The watcher polls the SHARED process Arc (the running entry keeps its own +// Arc; nothing is moved out of RunningServe): +fn spawn_exit_watch(&self, base_url: String, process: Arc, ownership_id: String) -> Option { + let manager = self.clone(); // OpencodeServeManager is Arc-backed-cheap + let interval = ...config.daemon_watch_interval...; + let handle = tokio::spawn(async move { + loop { + tokio::time::sleep(interval).await; + if process.exited().is_some() { + manager.handle_unrequested_exit(&base_url, &ownership_id).await; + return; + } + } + }); + Some(handle.abort_handle()) +} + +// Staleness-gated loss path. The take is the exactly-once arbiter (LB-07): +// a None take means the other arm already handled this loss — silent no-op. +async fn handle_unrequested_exit(&self, base_url: &str, ownership_id: &str) { + let taken = { + let mut running = self.inner.running.lock().await; + match running.as_ref() { + Some(r) if r.ownership_id == ownership_id => running.take(), // ours, still current + _ => return, // stale watcher (a newer daemon owns the entry) — no-op + } + }; + let Some(running) = taken else { return }; + tracing::warn!(reason = "process_exit", base_url = %base_url, "freshagent.opencode.daemon_crash_detected"); + running.process.kill(); // reaper parity only — the process already exited + self.emit_lost_for_all(); + let _ = self.inner.daemon_signals.send(DaemonSignal::Lost { reason: "process_exit" }); + self.schedule_re_warm(); +} + +// discard_running: abort the watcher (running._exit_watch) FIRST, Task-2 WARN, +// kill, emit_lost_for_all, send Lost{reason}, schedule_re_warm(). Same +// exactly-once discipline: the take yields None only if the watcher lost the +// race, in which case the abort already ran and the watcher returned. + +fn schedule_re_warm(&self) { + if self.inner.shutdown.load(Ordering::SeqCst) { return; } + // Fresh-incident gate (the executed D2 semantics, commit 900742d1a): the + // attempts counter resets HERE — at the NEXT loss — only when the lost + // daemon outlived the full ladder cap (`re_warm_backoff_max_ms`, tracked + // from `last_cold_start_at`); a faster death keeps the escalation. + let fresh_incident = /* last_cold_start_at.elapsed() >= re_warm_backoff_max_ms */; + if fresh_incident { self.inner.re_warm_attempts.store(0, Ordering::SeqCst); } + let manager = self.clone(); + tokio::spawn(async move { + // RETRY LOOP (plan-review round 1): a failed attempt must schedule + // the NEXT attempt — a single-shot spawn that only WARNs leaves the + // daemon permanently absent (the disk-pressure case). Retry forever at + // the capped interval; the shutdown flag is checked each iteration. + loop { + let attempts = manager.inner.re_warm_attempts.fetch_add(1, Ordering::SeqCst) + 1; + let delay_ms = (manager.config().re_warm_backoff_initial_ms + .saturating_mul(1u64 << (attempts - 1).min(16))) + .min(manager.config().re_warm_backoff_max_ms); + tokio::time::sleep(Duration::from_millis(delay_ms)).await; + if manager.inner.shutdown.load(Ordering::SeqCst) { return; } + match manager.ensure_started().await { + Ok(_) => { + // D2 (executed in commit 900742d1a): do NOT reset the + // attempts counter on success — a daemon dying + // immediately after every successful start must keep + // climbing the ladder, never respawn at the initial delay + // forever (pinned by daemon_loss_re_warm_backs_off_exponentially). + // The fresh-incident reset lives in the gate above, so a + // stable daemon's death still starts at the initial delay + // (the round-2 intent). + tracing::info!(attempt = attempts, "freshagent.opencode.daemon_re_warm"); + return; // started — the loop ends; the exit watcher arms again for this daemon + } + Err(err) => { + // Log and RETRY (backoff escalates via the attempts counter). + tracing::warn!(attempt = attempts, error = %err, "freshagent.opencode.daemon_re_warm"); + } + } + } + }); +} + +pub fn subscribe_daemon_signals(&self) -> tokio::sync::broadcast::Receiver { + self.inner.daemon_signals.subscribe() +} +``` + +- [x] **Step 4: Run the focused tests** + +Run: `cargo test -p freshell-opencode --test serve_daemon_selfheal && cargo test -p freshell-opencode` + +Expected: PASS (including all pre-existing tests — especially `rejects_on_sidecar_lost`, `settles_within_deadline_when_health_never_resolves` (its kill path now aborts the watcher too), and the FR2 trio). + +- [x] **Step 5: Refactor while green** + +Fold the shared loss path (take-entry + lost + signal + schedule) into one private helper used by both the watcher arm and `discard_running` (the only difference: the WARN text/reason and the pre-abort). + +- [x] **Step 6: Run impacted-test verification** + +Impacted set: all of `freshell-opencode` (manager core), plus `freshell-freshagent opencode_ws::tests` (the runtime drives the manager — its fakes seed via `set_manager_for_test`; new fields/behavior must not break the 119 existing tests) and `freshell-freshagent lib::tests` (FR2 pins). + +Run: `cargo test -p freshell-opencode && cargo test -p freshell-freshagent opencode_ws::tests && cargo test -p freshell-freshagent get_opencode_snapshot` + +Expected: PASS + +- [x] **Step 7: Commit the task** + +```bash +git add crates/freshell-opencode/src/serve.rs crates/freshell-opencode/src/lib.rs crates/freshell-opencode/tests/serve_daemon_selfheal.rs +git commit -m "feat(opencode): daemon exit watcher, loss signal, and backoff re-warm" +``` + +--- + +### Task 4: Runtime-level self-heal — daemon-loss fan-out, bridge restart, and pane revival + +**Files:** +- Modify: `crates/freshell-freshagent/src/opencode_ws.rs` (state struct ~:92-162, `handle_attach` ~:5166, `handle_send` ~:1428, `handle_compact` ~:3544, `spawn_serve_bridge` ~:5971) +- Modify: `crates/freshell-freshagent/src/lib.rs` (`ensure_manager` ~:2803 — expose the manager cell clone helper if needed) +- Modify: `AGENTS.md` (architecture prose, "Agent Status Indicators" freshopencode sentence) and `crates/freshell-server/src/logging.rs` canonical event-name list (~:48-54) if it enumerates event names +- Test: `crates/freshell-freshagent/src/opencode_ws.rs` `#[cfg(test)] mod tests` (mirror the codex self-heal test at codex.rs:15846) + +**Interfaces:** +- Consumes: Task 3's `subscribe_daemon_signals()` / `DaemonSignal`; `event_frame`/`emit_fresh_agent_error` (opencode_ws.rs:6068/686), `spawn_serve_bridge` (opencode_ws.rs:5971), `FreshAgentState.broadcast_tx` (lib.rs:1917), `set_manager_for_test` (lib.rs:2837), `ensure_manager` (lib.rs:2803 — the real seam; there is no `peek_or_ensure_manager`). +- Produces: + - A per-materialized-session typed edge on daemon loss: `freshAgent.event{provider:"opencode", sessionType:"freshopencode", event:{type:"freshAgent.error", code:"OPENCODE_DAEMON_LOST", message:"The opencode serve daemon was lost unexpectedly - it is restarting automatically."}}` — folds client-side through the EXISTING generic `sessionError` path (fresh-agent-ws.ts:514-520), showing the dismissible "Agent error:" banner and clearing busy. + - Level-triggered bridge revival: on arming, and again on every `DaemonSignal::Started`, restart bridges that are dead/absent for materialized sessions and push `freshAgent.session.snapshot{status:"idle"}` ONLY to sessions whose bridge was actually restarted (which the client treats as snapshot-invalidating → transcript refetch). No `saw_loss` heuristic — tokio broadcast does not replay history, so revival must not depend on having seen the `Lost` edge (LB-02). + - **The fenced attach becomes a real recovery verb (LB-05 redesign):** `handle_attach`'s dead-bridge restart arm (opencode_ws.rs:5460-5473) gains `manager.ensure_started().await` before `spawn_serve_bridge` — a map-hit fenced attach against a daemon-absent manager now respawns the shared daemon and re-bridges, exactly as `resume_durable_session` already does for map-misses. No chime: the recovery must never emit `freshAgent.turn.complete`. + +- [x] **Step 1: Write the failing behavioral tests** + +In `opencode_ws.rs` tests (drafts; mirror `onexit_self_heal_emits_exited_status_with_no_chime_and_keeps_session_mapped` at codex.rs:15846-15896 and the `state_with_bus` harness at codex.rs:9983-9991 — the opencode tests already have the bus pattern + `set_manager_for_test`): + +```rust +// 2026-09-20 incident: the shared daemon died and NOTHING told the panes — +// no status edge, no respawn, no bridge revival; panes dead-ended on the +// snapshot 409. The runtime self-heal must make daemon loss observable and +// recoverable per session. (LB-02: revival is level-triggered — it runs on +// arming and on Started, never dependent on having observed Lost.) +#[tokio::test] +async fn daemon_loss_fans_out_a_typed_edge_then_revives_bridges_after_respawn() { + let exiting = Arc::new(AtomicBool::new(false)); + let (state, rx) = opencode_state_with_bus(); // FreshAgentState::new(auth, tx) + FreshOpencodeState, per existing helpers + state.fresh_agent.set_manager_for_test(fake_manager_exiting_after( // health 200, prompt/summarize ok, ExitingProcess(exiting) + exiting.clone(), /* tiny watch + backoff config */)); + let session = materialized_opencode_session(&state, "ses_recover").await; // via handle_send against the fake http, as existing send tests do + exiting.store(true, Ordering::SeqCst); // the daemon dies + let frame = next_fresh_agent_frame(&rx).await; + assert_eq!(frame["event"]["type"], "freshAgent.error"); + assert_eq!(frame["event"]["code"], "OPENCODE_DAEMON_LOST"); + assert_eq!(frame["sessionId"], "ses_recover"); + // NO chime ever accompanies a daemon loss: + assert_no_turn_complete(&rx).await; + // Plan-review round 1 (Minor): the session is dual-keyed (placeholder + + // ses_*) in the map — exactly ONE OPENCODE_DAEMON_LOST edge per + // materialized session must be emitted (drain the bus and count). + assert_exactly_one_daemon_lost_edge(&rx, "ses_recover").await; + exiting.store(false, Ordering::SeqCst); // the re-warm's respawn now succeeds + let frame = next_fresh_agent_frame(&rx).await; // DaemonSignal::Started → level-triggered revival + assert_eq!(frame["event"]["type"], "freshAgent.session.snapshot"); + assert_eq!(frame["event"]["status"], "idle"); + assert!(session_serve_bridge_alive(&state, "ses_recover").await, "bridge restarted after respawn"); +} + +// LB-05 (falsified → redesign): in the incident, the pane's fenced attach was +// exercised 3× against the dead shared daemon and recovered NOTHING, because +// the attach tail only re-subscribed the bridge — nothing respawns the +// daemon for a map-hit. The attach tail must ensure the daemon exists. +#[tokio::test] +async fn map_hit_fenced_attach_respawns_the_daemon_and_rebridges() { + let spawns = Arc::new(AtomicUsize::new(0)); + let (state, rx) = opencode_state_with_bus(); + state.fresh_agent.set_manager_for_test(fake_manager_with( // health 200; daemon DISCARDED before the attach + FakeSpawner::counting(spawns.clone()))); + let session = materialized_opencode_session(&state, "ses_attach_recover").await; + discard_manager_running_entry(&state).await; // the shared daemon is dead; the session row persists + let fence = observed_fence_for(&state, "ses_attach_recover").await; // the runtime-owner pair + state.handle_attach(attach_msg("ses_attach_recover", fence)).await; + assert!(spawns.load(Ordering::SeqCst) >= 1, "a map-hit attach against a daemon-absent manager must respawn the daemon"); + assert!(session_serve_bridge_alive(&state, "ses_attach_recover").await, "the bridge must be restarted"); + let frame = next_fresh_agent_frame(&rx).await; // the attach tail's snapshot push + assert_eq!(frame["event"]["type"], "freshAgent.session.snapshot"); +} + +// Plan-review round 3: the revival pass must respect the ownership +// coordinator — a session killed/retired or handed to a terminal owner +// between the loss and the respawn must NOT be revived. +#[tokio::test] +async fn revival_skips_sessions_handed_off_or_removed_after_the_loss() { + let (state, rx) = opencode_state_with_bus(); + state.fresh_agent.set_manager_for_test(fake_manager_healthy()); + let _kept = materialized_opencode_session(&state, "ses_keeps").await; + let _gone = materialized_opencode_session(&state, "ses_gone").await; + // The concurrent-handoff shape: while the daemon is down, session B is + // retired from the map and its key transitions (Stopping/handoff → + // terminal owner): + retire_session_from_map(&state, "ses_gone").await; + mark_session_transition_or_terminal_owner(&state, "ses_gone").await; + drive_daemon_respawn(&state).await; // DaemonSignal::Started arrives + assert!(session_serve_bridge_alive(&state, "ses_keeps").await, "healthy fresh-agent sessions revive"); + assert!(!session_serve_bridge_alive(&state, "ses_gone").await, + "removed/transition/terminal-owned sessions must NOT be revived"); + assert_no_snapshot_push_for(&rx, "ses_gone").await; +} +``` + +(Drafts: adapt to the actual harness helpers — `opencode_state_with_bus`, session materialization via `handle_send` against the seeded fake http, and the manager's running-entry discard via the fake's own seams or `discard_running`. Lock discipline per LB-01: any test helper that walks the sessions map must clone the `Arc` session handles under a short map lock and drop the map guard before locking a session — the map guard is NEVER held across a per-session lock acquisition, per the documented contract at opencode_ws.rs:100-115.) + +- [x] **Step 2: Run the test and verify the intended failure** + +Run: `cargo test -p freshell-freshagent daemon_loss_fans_out` && `cargo test -p freshell-freshagent map_hit_fenced_attach` && `cargo test -p freshell-freshagent revival_skips` (three commands — cargo test accepts ONE positional TESTNAME) + +Expected: FAIL — `daemon_loss_fans_out...` fails with no `OPENCODE_DAEMON_LOST` frame (today `SessionSignal::Lost` is a no-op at opencode_ws.rs:6018; no listener exists); `map_hit_fenced_attach...` fails because the attach tail never spawns the daemon (spawns == 0, the LB-05-validated gap); `revival_skips...` fails because no revival machinery exists yet. + +- [x] **Step 3: Add the minimal production implementation** + +Two server-side changes (LB-01/LB-02/LB-08/LB-10/N-3 corrections applied): + +**(a) The attach tail's bridge-restart arm (opencode_ws.rs:5460-5473) ensures the daemon exists before re-bridging — the LB-05 redesign that turns the fenced attach into a real recovery verb:** + +```rust +// LB-05 (falsified → redesign): in the incident the fenced attach was +// exercised 3× against the dead daemon and recovered nothing — the tail only +// re-subscribed the bridge. A map-hit attach must respawn the shared daemon +// (mirroring resume_durable_session's map-miss behavior). ensure_started is +// single-flighted, so concurrent attach/send/compact callers cannot spawn a +// second daemon. +let manager = self.fresh_agent.ensure_manager().await; +if let Err(err) = manager.ensure_started().await { + // Bounded health failure: answer the attach with the typed error path + // (the existing emit_fresh_agent_error machinery) instead of a silent + // half-attached state. + ... +} +// ...existing dead-bridge restart + spawn_serve_bridge tail... +``` + +**(b) The daemon-loss watcher on `FreshOpencodeState` (idempotent; armed from handle_send/handle_attach/handle_compact; LB-10: the task holds a full state clone so `spawn_serve_bridge(&self, ...)` is callable directly):** + +```rust +// FreshOpencodeState gains: daemon_loss_watcher: Arc>. +fn ensure_daemon_loss_watcher(&self) { + if self.daemon_loss_watcher.set(()).is_err() { return; } // already armed + let manager = self.fresh_agent.ensure_manager().await; // lib.rs:2803 (N-3: the real seam) + let state = self.clone(); + let mut signals = manager.subscribe_daemon_signals(); + tokio::spawn(async move { + // Arming-time level pass (LB-02: broadcast does NOT replay history — + // if the daemon already re-warmed before we subscribed, revive now). + state.revive_dead_bridges_if_daemon_running().await; + loop { + match signals.recv().await { + Ok(DaemonSignal::Lost { reason }) => { + tracing::warn!(reason = reason, "freshagent.opencode.daemon_loss_observed"); + // LB-01: NEVER hold the sessions-map guard across a + // per-session lock (contract at opencode_ws.rs:100-115 — + // the reverse edge deadlocked production). Clone the + // (id, Arc) pairs under ONE short map lock, + // drop the guard, then read each session outside it. + // Plan-review round 1 (Minor): the map is keyed by BOTH + // the placeholder and the durable id pointing at the SAME + // session — dedupe by real_session_id (BTreeSet) so each + // materialized session gets exactly ONE edge. + let materialized: std::collections::BTreeSet = { + let map = state.sessions.lock().await; + let handles: Vec>> = map.values().cloned().collect(); + drop(map); + handles.into_iter().filter_map(|s| s.lock().await.real_session_id.clone()).collect() + }; + for id in materialized { + let _ = state.fresh_agent.broadcast_tx.send(event_frame_json(&id, json!({ + "type": "freshAgent.error", "sessionId": id, + "code": "OPENCODE_DAEMON_LOST", + "message": "The opencode serve daemon was lost unexpectedly - it is restarting automatically.", + }))); + } + } + Ok(DaemonSignal::Started) => { + // Level-triggered revival (LB-02): revive whatever is + // dead; push the idle snapshot ONLY to sessions whose + // bridge was actually restarted. No `saw_loss` heuristic. + state.revive_dead_bridges_if_daemon_running().await; + } + Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => continue, // LB-08: never disarm on Lagged + Err(tokio::sync::broadcast::error::RecvError::Closed) => return, + } + } + }); +} + +// The revival pass (also called at arming), respecting LB-01's lock order AND +// the ownership coordinator (plan-review round 3: a revival that checks only +// real_session_id + bridge state can race a concurrent handoff that kills the +// session, removes it from the map, and commits a terminal owner — the stale +// Arc would then spawn a fresh-agent bridge broadcasting beside the terminal +// owner. The attach path guards exactly this; revival must too.): +// 1. manager.base_url().await is None → return (daemon absent — nothing to +// revive into; the next Started signal or attach drives revival). +// 2. Snapshot the map: clone the (session_id, Arc>) +// pairs under ONE short map lock, dropping the guard immediately. +// 3. For each pair (OUTSIDE the map guard), per candidate: +// a. RE-LOOKUP the id in the sessions map at revival time — if the key is +// gone (killed/handoff removed it), skip. Never act on the retained Arc +// alone. +// b. Observe the canonical ownership state fresh (the runtime's +// canonical_ownership_snapshot): skip on ANY transition +// (Handoff/Starting/Stopping/Fenced — something else owns the session +// right now) and skip on Live{Terminal} (a terminal owner holds it). +// Revive only Live{FreshAgent} (this runtime's own sessions). +// c. Arm the SAME adopt-guard machinery handle_attach's restart tail uses +// (ownership_lane::arm_adopt_guard, held across the bridge restart) — +// factor handle_attach's guard-held restart tail into a shared helper +// (e.g. restart_session_bridge_guarded) that BOTH handle_attach and the +// revival pass call, so the coordinator's atomicity rides along instead +// of being re-implemented. +// d. Only then: spawn_serve_bridge(...) and broadcast +// snapshot_event(real_id, "idle") — the snapshot push goes ONLY to +// sessions whose bridge was actually restarted. +``` + +- [x] **Step 4: Run the focused test** + +Run: `cargo test -p freshell-freshagent daemon_loss_fans_out` && `cargo test -p freshell-freshagent map_hit_fenced_attach` && `cargo test -p freshell-freshagent revival_skips` + +Expected: PASS + +- [x] **Step 5: Refactor while green** + +Ensure the fan-out + bridge-revival helper is shared (not duplicated) with `handle_attach`'s restart arm; keep AGENTS.md's "Agent Status Indicators" paragraph truthful — update the freshopencode sentence to document the daemon-loss self-heal (edge shape, no chime, backoff respawn, bridge revival, and the fix-4 client recovery pointer). + +- [x] **Step 6: Run impacted-test verification** + +Impacted set: all `opencode_ws::tests` (119 tests) plus the lib.rs snapshot pins (`get_opencode_snapshot_*` — crate-root `mod tests` tests are named `tests::...`, so the filter is the test-name substring, not `lib::tests`). + +Run: `cargo test -p freshell-freshagent` + +Expected: PASS + +- [x] **Step 7: Commit the task** + +```bash +git add crates/freshell-freshagent/src/opencode_ws.rs crates/freshell-freshagent/src/lib.rs AGENTS.md crates/freshell-server/src/logging.rs +git commit -m "feat(freshopencode): daemon-loss self-heal edge, respawn revival, and bridge restarts" +``` + +--- + +### Task 5: Client drives fenced attach + refetch on the snapshot 409 RESTORE_UNAVAILABLE + +**Files:** +- Modify: `src/components/fresh-agent/FreshAgentView.tsx` (predicate beside `isLostFreshOpencodeThreadError` at ~:455-463; new arm in `handleSnapshotError` at ~:2770-2827; a one-shot guard ref; the pane-refresh attach-decision lane at ~:1718-1761 is the reuse pattern) +- Modify: the runtime-owner store fold (the slice/action behind `src/store/selectors/runtimeOwner.ts` + the WS refusal fold at `src/lib/fresh-agent-ws.ts:224` — reuse the existing "refresh the observed fence from the refusal" action if one exists; otherwise add a minimal reducer action mirroring that fold, e.g. `applyRefusalFence({ sessionId, ownerKind, ownerGeneration })` that advances the record's generation to the refusal's while preserving its epoch) +- Test: `test/unit/client/components/fresh-agent/FreshAgentView.test.tsx` (beside the 404 test at ~:1887-1930; owner-record seeding follows the patterns in `test/unit/client/store/selectors-runtime-owner.test.ts`) + +**Interfaces:** +- Consumes: `ApiError.details` (the full 409 body: `code`, `ownerKind`, `ownerGeneration`), `captureFreshAgentAttachmentAttempt` (the real wrapper at FreshAgentView.tsx — R-1: the pane-refresh reaction pattern at :1718-1761 is the exact reuse: bump `attachDecisionSerialRef`, capture, `sendFencedFreshAgentAttach(attempt)`), `sendFencedFreshAgentAttach` (:1262-1275), `requestSnapshotRefresh` / `requestRevealRefresh`, `selectPaneOwnerFence`. +- Produces: on a 409 `RESTORE_UNAVAILABLE` snapshot error for a freshopencode pane whose refusal names a `fresh-agent` owner (the incident class — the pane's own stale claim; LB-05 scoped terminal owners out to the session-directory handoff door): ONE generation-fenced `freshAgent.attach` whose `observedGeneration` is refreshed from the 409's own `ownerGeneration` (plan-review round 2: the fence binds to the refusal, not the possibly-stale owner record — an unfenced or stale attach is refused `FENCE_REQUIRED` and preserves the dead-end), followed by a snapshot refetch. Bounded (LB-03): the ENTIRE recovery — attach + refetch — runs once per pane identity (`createRequestId` + `snapshotThreadId`); a suppressed attach (`sendFencedFreshAgentAttach` returning false) does NOT consume the one-shot guard and does NOT refetch; a second 409 falls through to the existing error surfaces (loadError banner / reveal error), never re-triggering fetches. When the 409 arrived on the reveal lane with `snapshotDirty` set, the recovery drives the reveal-refresh path (`requestRevealRefresh(true)`) so the success-path reveal-dirty clear can run and the "Refreshing conversation" overlay lifts (LB-04). The pane identity is NOT reset (unlike the 404 lost-thread path). + +- [x] **Step 1: Write the failing behavioral test** + +In FreshAgentView.test.tsx (draft — template is the 404 test at :1887-1930; reuse its harness: `apiMock.getFreshAgentThreadSnapshot.mockRejectedValueOnce`, `StoreBackedFreshAgentView`, `sentFreshAgentMessages`): + +```ts +```ts +// 2026-09-20 incident (log-validated): the daemon died, the reveal GET +// answered the typed 409 RESTORE_UNAVAILABLE for the pane's OWN stale +// Live{FreshAgent, gen 1} claim, and the pane dead-ended on a dismiss-only +// banner forever. The documented recovery is the generation-fenced attach + +// refetch — drive it once. +// LB-09: the mount attach already sends ONE freshAgent.attach on mount, so a +// bare length assertion is vacuous — read the baseline AFTER the mount +// settles and assert the POST-409 delta. +// Plan-review round 1: DEFER the first rejection until after the baseline is +// read — an immediately-rejected mock races the mount fetch (the recovery +// attach may land before the test snapshots the count). +// Plan-review round 2: seed the runtime-owner record and make the 409 name a +// NEWER generation — the recovery attach MUST carry the 409's generation +// (fence bound to the refusal), or the wired server refuses it with +// FENCE_REQUIRED and the dead-end persists. Also: reject with a real ApiError +// instance so the banner text is the 409's own message. +it('recovers a freshopencode pane from a snapshot 409 with one fenced attach and a refetch', async () => { + seedRuntimeOwnerRecord(store, { sessionId: 'ses_live', ownerKind: 'fresh-agent', epoch: 1, generation: 1 }) // follow the selectors-runtime-owner test fold patterns + let rejectFirstSnapshot!: (error: unknown) => void + apiMock.getFreshAgentThreadSnapshot + .mockImplementationOnce(() => new Promise((_, reject) => { rejectFirstSnapshot = reject })) + .mockResolvedValue(freshopencodeSnapshot({ sessionId: 'ses_live', status: 'idle' })) // the refetch succeeds + renderFreshAgentPane({ provider: 'opencode', sessionId: 'ses_live', status: 'connected' }) + await waitFor(() => expect(apiMock.getFreshAgentThreadSnapshot).toHaveBeenCalledTimes(1)) + const attachCountBeforeRecovery = sentFreshAgentMessages('freshAgent.attach').length // the mount attach, settled + await act(async () => { + rejectFirstSnapshot(new ApiError(409, 'Session ses_live is still running on the server.', { + code: 'RESTORE_UNAVAILABLE', ownerKind: 'fresh-agent', ownerGeneration: 2, + })) + }) + await waitFor(() => { + expect(sentFreshAgentMessages('freshAgent.attach')).toHaveLength(attachCountBeforeRecovery + 1) // the RECOVERY attach (LB-09) + const recoveryAttach = sentFreshAgentMessages('freshAgent.attach').at(-1) + expect(recoveryAttach?.observedEpoch).toBe(1) // the record's epoch + expect(recoveryAttach?.observedGeneration).toBe(2) // the 409's CURRENT generation, not the stale record's 1 + }) + await waitFor(() => { + expect(apiMock.getFreshAgentThreadSnapshot).toHaveBeenCalledTimes(2) // exactly one recovery refetch + }) + // The pane kept its identity (the 409 is NOT the 404 lost-thread reset): + expect(getFreshAgentPaneContent(store).sessionId).toBe('ses_live') + // And no dead-end banner for the recovered pane: + await waitFor(() => expect(screen.queryByRole('alert')).not.toBeInTheDocument()) +}) + +it('does not loop recovery fetches on repeated 409s', async () => { + vi.useFakeTimers() // plan-review round 2: timers must be ENABLED before advancing + try { + apiMock.getFreshAgentThreadSnapshot.mockRejectedValue(new ApiError(409, 'Session ses_live is still running on the server.', { + code: 'RESTORE_UNAVAILABLE', ownerKind: 'fresh-agent', ownerGeneration: 2, + })) + renderFreshAgentPane({ provider: 'opencode', sessionId: 'ses_live', status: 'connected' }) + // Plan-review round 1: AWAIT the banner — findByText returns a promise. + // Plan-review round 2: reject with a real ApiError (an Error instance) so + // handleSnapshotError preserves the 409's message — plain objects render + // 'Failed to load session' instead. + await screen.findByText(/still running on the server/i) + const baseline = sentFreshAgentMessages('freshAgent.attach').length + await act(async () => { await vi.advanceTimersByTimeAsync(2_000) }) + expect(sentFreshAgentMessages('freshAgent.attach')).toHaveLength(baseline) // one recovery total, not per fetch (LB-03) + expect(apiMock.getFreshAgentThreadSnapshot.mock.calls.length).toBeLessThanOrEqual(3) // mount + recovery only — no loop + } finally { + vi.useRealTimers() + } +}) +``` + +- [x] **Step 2: Run the test and verify the intended failure** + +Run: `npm run test:vitest -- run test/unit/client/components/fresh-agent/FreshAgentView.test.tsx -t '409'` + +Expected: FAIL — no attach is sent; the pane shows the dismiss-only alert banner (the current dead-end). + +- [x] **Step 3: Add the minimal production implementation** + +In FreshAgentView.tsx (sketch): + +```ts +function isRestoreUnavailableSnapshotError(error: unknown): boolean { + if (!error || typeof error !== 'object') return false + const status = 'status' in error ? (error as { status?: unknown }).status : undefined + const details = 'details' in error ? (error as { details?: unknown }).details : undefined + const code = details && typeof details === 'object' && 'code' in details + ? (details as { code?: unknown }).code + : undefined + const ownerKind = details && typeof details === 'object' && 'ownerKind' in details + ? (details as { ownerKind?: unknown }).ownerKind + : undefined + // LB-05 scoping: fresh-agent owners are the pane's own stale-claim class + // (the incident state) — the fenced attach proceeds for them. Terminal + // owners are a different scenario (a genuinely terminal-owned session); + // their recovery door is the session-directory handoff, so the client does + // not attempt the (refused) attach for them. + return status === 409 && code === 'RESTORE_UNAVAILABLE' && ownerKind === 'fresh-agent' +} +``` + +In `handleSnapshotError`, after the opencode lost-404 arm and BEFORE the reveal arm — provider-gated to `opencode` (LB-03: the ENTIRE recovery — attach + refetch — sits inside the once-per-identity guard; a second 409 falls through to the honest error surfaces below, never looping): + +```ts +// 2026-09-20 incident: with the daemon dead, daemon-absent snapshot GETs +// answer the typed 409 RESTORE_UNAVAILABLE for as long as the session key +// stays Live. The documented recovery is the generation-fenced attach +// (b8ke Task-5: cold resume flows only through the explicit lifecycle +// commands) — drive it ONCE per pane identity, then refetch via the +// reveal-refresh path when reveal-dirty (LB-04) so the overlay can clear. +// Repeated 409s fall through to the honest error surfaces below; never +// reset the pane. +if (paneContent.provider === 'opencode' && isRestoreUnavailableSnapshotError(error)) { + const fresh = paneContentRef.current + const recoveryKey = `${fresh.createRequestId}:${sessionId}` + const refusal = (error as ApiError).details as { ownerGeneration: number } + if (restoreUnavailableRecoveryRef.current !== recoveryKey) { + const previousRecoveryKey = restoreUnavailableRecoveryRef.current + restoreUnavailableRecoveryRef.current = recoveryKey + // Plan-review round 2: bind the fence to the 409's CURRENT generation — + // refresh the observed owner fence from the refusal itself (the same + // fold the WS create.failed lane uses for its ownerKind/ownerGeneration/ + // ownerEpoch fields, fresh-agent-ws.ts:224). Without this, a stale or + // absent owner record sends an unfenced or stale-generation attach the + // wired server refuses with FENCE_REQUIRED — preserving the dead-end. + dispatch(refreshObservedFenceFromRefusal({ + sessionId, ownerKind: 'fresh-agent', ownerGeneration: refusal.ownerGeneration, + })) + attachDecisionSerialRef.current += 1 + const attempt = captureFreshAgentAttachmentAttempt(fresh) // R-1: the real wrapper's call shape + const sent = sendFencedFreshAgentAttach(attempt) + if (!sent) { + // Plan-review round 2: the attach was suppressed (lifecycle superseded + // or attempt-key mismatch). Do NOT consume the one-shot recovery and do + // NOT refetch — restore the guard and fall through to the honest error + // surfaces below. + restoreUnavailableRecoveryRef.current = previousRecoveryKey + } else { + // LB-04: a reveal-lane 409 with snapshotDirty set must refetch through + // the reveal path ('reveal' trigger), or the success-path reveal-dirty + // clear never runs and the pane hides behind the "Refreshing + // conversation" overlay forever. Otherwise refetch via 'manual'. + if (trigger === 'reveal' && snapshotDirtyRef.current) { + revealRefreshStartedAtRef.current = null + setSnapshotRevealError(null) + requestRevealRefresh(true) + } else { + setLoadError(null) + requestSnapshotRefresh('manual') + } + return // recovery fired for this error — the honest error surfaces below are for SUBSEQUENT 409s only + } + } + // Recovery already attempted for this identity: do NOT clear errors and do + // NOT refetch again — fall through to the reveal error arm / setLoadError + // below so the user sees the honest state. +} +``` + +(Adapt: the ref `restoreUnavailableRecoveryRef = useRef(null)` beside the other reveal refs ~:800-804; `captureFreshAgentAttachmentAttempt`'s real signature follows the pane-refresh reaction lane at :1735-1739; `refreshObservedFenceFromRefusal` is the reuse-or-add-mirror of the WS refusal fold at fresh-agent-ws.ts:224 — if the exact action differs in the slice, reuse it; the TEST pins the observable contract: the recovery attach carries `observedGeneration === `. Verify the reveal-refresh request helper's exact name/behavior (`requestRevealRefresh(true)` forces a reveal-tagged refresh) against the state machine at :2601-2624 and the arming sites ~:1233.) + +- [x] **Step 4: Run the focused test** + +Run: `npm run test:vitest -- run test/unit/client/components/fresh-agent/FreshAgentView.test.tsx -t '409'` + +Expected: PASS + +- [x] **Step 5: Refactor while green** + +If the 409 arm and the 404 arm now share reset-vs-recover structure, extract only what is genuinely shared (they intentionally differ: reset vs recover) — otherwise leave as-is. + +- [x] **Step 6: Run impacted-test verification** + +Impacted set: the full FreshAgentView suite (the snapshot error paths, reveal lanes, attach lanes, and scheduler tests all touch `handleSnapshotError`) plus the fresh-agent-ws fold tests. + +Run: `npm run test:vitest -- run test/unit/client/components/fresh-agent/ test/unit/client/lib/fresh-agent-ws.test.ts test/unit/client/lib/fresh-agent-turn-complete.test.ts` + +Expected: PASS + +- [x] **Step 7: Commit the task** + +```bash +git add src/components/fresh-agent/FreshAgentView.tsx src/store/freshAgentSlice.ts test/unit/client/components/fresh-agent/FreshAgentView.test.tsx +# Plan-review round 3: stage EVERY file the fold/refusal-fence change lands in +# (freshAgentSlice.ts, the runtimeOwner selector's store module, or wherever +# the reuse-or-mirror action lives — match the actual touched files). +git commit -m "fix(fresh-agent): recover freshopencode panes from snapshot 409 via fenced attach and refetch" +``` + +--- + +### Task 6: Cloud-legal e2e — the 409 recovery story end to end + +**Files:** +- Create: `test/e2e-browser/specs/freshopencode-snapshot-409-recovery.spec.ts` +- Test: itself (local chromium run; must NOT be added to `CLOUD_SKIP_SPECS` in `test/e2e-browser/playwright.cloud.config.ts`) + +**Interfaces:** +- Consumes: the model-picker "sidecar suppressed + routed fetch" pattern (the explicitly-cloud-legal pattern cited at playwright.cloud.config.ts:36-38; follow `test/e2e-browser/specs/freshopencode-model-picker.spec.ts`), the `TestHarness` (`test/e2e-browser/helpers/test-harness.js`), `RustServer` helper. +- Produces: an e2e proof of the Task-5 user story: a freshopencode pane whose snapshot fetch first 409s (`RESTORE_UNAVAILABLE` + `ownerKind`/`ownerGeneration` body) then 200s, recovers by sending `freshAgent.attach` and rendering the transcript — with no dismiss-only dead-end. + +- [x] **Step 1: Write the failing-passing spec (verification task; Task 5 already turned the behavior green)** + +Draft structure (follow the model-picker spec's routing/suppression mechanics exactly): + +```ts +test('freshopencode pane recovers from a snapshot 409 via fenced attach', async ({ page }) => { + // 1. Suppress the opencode sidecar (model-picker pattern: + // setSuppressAllFreshAgentNetworkEffects(true) routes freshAgent.* WS + // frames to the harness spy — getSentWsMessages() captures them). + // 2. Route /api/fresh-agent/threads/freshopencode/opencode/:id: + // first call -> fulfill(409, { status:'error', code:'RESTORE_UNAVAILABLE', + // message:'Session is still running on the server.', + // ownerKind:'fresh-agent', ownerGeneration:1 }) // the incident class (LB-05) + // subsequent -> fulfill(200, ) + // 3. Seed a freshopencode pane with a durable ses_* session (harness). + // 4. Assert: a POST-409 recovery freshAgent.attach frame is sent (R-2: the + // mount attach also appears in the spy log — count the delta after the + // 409 lands, not the total), the transcript renders from the 200 + // snapshot, and no dismiss-only dead-end alert remains. +}) +``` + +- [x] **Step 2: Run it locally** + +Run: `npm run test:e2e:local -- --project=chromium test/e2e-browser/specs/freshopencode-snapshot-409-recovery.spec.ts` + +Expected: PASS. (Sanity-check the red history: `git stash` the Task-5 commit is NOT needed — Task 5's unit red already proves the pre-fix dead-end; record that linkage in the commit message.) + +- [x] **Step 3: Verify cloud inclusion** + +Run: `FRESHELL_E2E_BACKEND=cloud npm run test:e2e` in the coordinated lane (or the narrow cloud invocation the repo sanctions for one spec) and confirm the spec is selected — it must not appear in `CLOUD_SKIP_SPECS`, `LOCAL_ONLY_SPECS`, or match any `CLOUD_SKIP_TITLES` pattern. + +Expected: the spec runs (and passes) on the cloud backend; per AGENTS.md, "a spec sitting in CLOUD_SKIP_SPECS is not coverage". + +- [x] **Step 4: Commit the task** + +```bash +git add test/e2e-browser/specs/freshopencode-snapshot-409-recovery.spec.ts +git commit -m "test(e2e): freshopencode snapshot-409 recovery runs cloud-legal end to end" +``` + +--- + +### Task 7: Local-lane e2e — the real daemon-death self-heal path end to end + +**Files:** +- Create: `test/e2e-browser/specs/freshopencode-daemon-death-selfheal.spec.ts` +- Modify: `test/e2e-browser/playwright.cloud.config.ts` (add the new spec to `CLOUD_SKIP_SPECS`, same provider-lifecycle-timing reason as `freshopencode-restart-recovery`) +- Test: itself (local chromium run) + +**Interfaces:** +- Consumes: the `freshopencode-restart-recovery.spec.ts` harness pattern — `installFakeOpencode` (`fixtures/fake-opencode.cjs` on the spawned server's PATH), the `RustServer` + `TestHarness` helpers, the fake's `FAKE_OPENCODE_AUDIT_LOG` JSONL for spawn/event assertions, and a fixture capability for an UNREQUESTED daemon death that is a scripted SELF-exit — the fake child exits on its own schedule/trigger (plan-review round 3: the spec must not kill any process — `AGENTS.md`'s destructive-test sandbox rule requires process-kill suites to run in `scripts/sandbox-test.sh`; a fixture child exiting itself is the test-sandbox doc's explicitly host-legal fake-child-lifecycle class, the same precedent as the codex onExit self-heal test spawning `true`. E.g. the fixture env-arms an exit-after-N-secs or polls a marker file and exits when it appears — no foreign PID is ever killed. If during implementation the spec cannot avoid killing something, run the spec via `npm run test:sandbox -- "..."` instead of host Playwright). +- Produces: an e2e proof of the SERVER-side self-heal chain with a real spawned server and a fake daemon process: (1) the pane is materialized and live; (2) the daemon dies an UNREQUESTED death (self-exit); (3) the pane shows the `OPENCODE_DAEMON_LOST` "Agent error:" banner; (4) the daemon respawns automatically within a bounded wait (audit log shows a second serve spawn); (5) the pane recovers (the idle snapshot push refetches the transcript; the banner is dismissible and no dead-end remains); (6) NO `freshAgent.turn.complete` chime during the window. + +- [x] **Step 1: Write the spec** (verification task; Tasks 3+4 turned the chain green — their Rust unit tests carry the TDD red history for this behavior) + +Follow the restart-recovery spec's structure (fake CLI on PATH, harness-seeded freshopencode pane with a durable `ses_*` id, deterministic waits on harness state — never wall-clock-sensitive provider-boot timing). + +- [x] **Step 2: Run it locally** + +Run: `npm run test:e2e:local -- --project=chromium test/e2e-browser/specs/freshopencode-daemon-death-selfheal.spec.ts` + +Expected: PASS + +- [x] **Step 3: Register the cloud-skip honestly** + +Add the filename to `CLOUD_SKIP_SPECS` (the spec is the same provider-lifecycle class as its model — 2-CPU/2-worker cloud contention cannot guarantee daemon-death timing). Cloud-backend PR coverage is carried by Task 6's cloud-legal spec; this spec is the local-lane end-to-end proof. + +- [x] **Step 4: Commit the task** + +```bash +git add test/e2e-browser/specs/freshopencode-daemon-death-selfheal.spec.ts test/e2e-browser/playwright.cloud.config.ts +git commit -m "test(e2e): real-daemon death self-heal recovery runs end to end (local lane)" +``` + +--- + +### Task 8: Whole-branch verification gates + +**Files:** none (verification only — the plan declares these gates, so the plan must run them; the-usual's Stage-5 exit additionally runs the coordinated full suite once at the final HEAD after the review loop closes) + +- [x] **Step 1: Rust formatting and lints** + +Run: `cargo fmt --all --check && cargo clippy --workspace --exclude freshell-tauri --all-targets -- -D warnings` + +Expected: PASS (clean) + +- [x] **Step 2: Client typecheck and lints** + +Run: `npm run typecheck && npm run lint` + +Expected: PASS (clean; eslint includes the jsx-a11y rules) + +- [x] **Step 3: Focused suite confirmation** + +Run: `cargo test -p freshell-opencode && cargo test -p freshell-freshagent && npm run test:vitest -- run test/unit/client/components/fresh-agent/ test/unit/client/lib/fresh-agent-ws.test.ts` + +Expected: PASS (all tasks' focused suites green together on the final HEAD) + +- [x] **Step 4: Record** + +No commit (verification only). Record the gate results in the run state; the coordinated full-suite gate at final HEAD runs per the-usual Stage 5 after the delta review loop ends. + +--- + +## Plan self-review (re-run after Stage-2 load-bearing corrections AND plan-review round 1) + +1. **Spec coverage:** defect 1 → Tasks 1 (compact/config no-kill, FR2 mirror); defect 2 → Task 2 (structured discard log); defect 3 → Tasks 3+4 (exit watcher + loss signal + retrying backoff re-warm + runtime fan-out edge + level-triggered bridge revival — the freshcodex-onExit mirror adapted to shared-daemon topology); defect 4 → Tasks 4a+5+6 (the fenced attach made a real recovery verb by respawning the daemon on map-hits, the client 409 arm driving it once, cloud-legal e2e). E2E coverage: Task 6 (cloud-legal client recovery) + Task 7 (local-lane real-daemon self-heal chain) + Rust unit tests (server lanes). Gates: Task 8 runs fmt/clippy/typecheck/lint + the focused suites on the final HEAD; the coordinated full suite runs at the-usual Stage-5 exit. The "backoff-guarded respawn" and "client-visible status edge" elements of defect 3 are both explicit (Task 3 re-warm config + retry loop; Task 4 `OPENCODE_DAEMON_LOST` edge). +2. **No silent deferrals:** the deliberate residuals (prompt_async and thin wrappers keep `DiscardOnTimeout::Yes`; frozen 409 text; terminal-owner 409s recover via the session-directory handoff door; REST-only never-viewed panes miss the banner until first WS interaction) are stated in Global Constraints, each with its reason and precedent. The real-daemon e2e is local-lane with an honest CLOUD_SKIP_SPECS entry (cloud PR coverage carried by Task 6). +3. **File and interface consistency:** all paths/signatures cross-checked against the six exploration reports at base 855dae72a, then corrected against the load-bearing ledger (LB-01 map lock order, LB-02 no-replay → level-triggered revival + arming pass, LB-03 bounded recovery, LB-04 reveal-trigger refetch, LB-05 attach-respawn redesign + fresh-agent-owner scoping, LB-06 Arc process + ownership_id, LB-07 exactly-once take, LB-08 Lagged tolerance, LB-09 attach-count delta, LB-10 state clone, R-1 `captureFreshAgentAttachmentAttempt`, N-3 `ensure_manager` seam), plan-review round 1 (routed `get_config(route)`, retrying re-warm with a fail-then-succeed test, single-filter cargo commands, deferred-rejection + awaited-banner client tests, dual-key dedupe, Tasks 7/8), AND plan-review round 2 (the re-warm retry test now starts healthy → kills the watched daemon → scripts failing re-warms; the recovery fence binds to the 409's `ownerGeneration` via the refusal-fold, suppressed attaches do not consume the one-shot guard; ApiError instances + fake timers in the client tests; the re-warm attempts counter resets only at a survival-gated fresh incident, never on re-warm success (the post-execution D2 correction — see Task 3 behavior item 3 and commit `900742d1a`); no trailing whitespace), AND plan-review round 3 (the revival pass re-looks-up each id, observes canonical ownership, skips transitions/terminal owners, and rides the same adopt-guard-protected restart helper as handle_attach; `DaemonSignal` re-exported from the crate root via lib.rs; Task 5's commit stages the store fold files; Task 7's daemon death is a scripted self-exit, never a foreign process kill). +4. **Executable tests:** each red test names its exact lane failure (killed counter, missing frame, missing spawn, missing attach) and reuses pinned fake/harness idioms (NeverExitsProcess kill counters, config_capture tracing capture, state_with_bus + set_manager_for_test, the 404 ApiError-mock template with delta-based attach counting and an explicit synchronization point). Every cargo invocation uses a single positional TESTNAME filter that matches the named tests. +5. **Placeholder scan:** drafts reference real helpers; where a fake needs a small extension (ExitingProcess, summarize/config hang scripting, fail-twice-then-healthy scripting, the fake-opencode death verb), the extension is named and its model (existing fakes) is cited — no TBDs. +6. **Operational completeness:** new structured event names are logged (Task 3/4) and registered in the logging docs if enumerated; AGENTS.md architecture prose updated (Task 4); no migrations; rollback = revert the commits (no persisted-state changes); verification gates explicit (Task 8 + the Stage-5 full suite). + +UNRESOLVED COVERAGE GAP: none. The original soft spot was resolved in Stage 2 (R-1), and the one falsified assumption (LB-05) reshaped Tasks 4+5 — the fenced attach now respawns the daemon (map-hit), and the client recovery is scoped to the fresh-agent-owner class the incident actually was. diff --git a/src/components/fresh-agent/FreshAgentView.tsx b/src/components/fresh-agent/FreshAgentView.tsx index 333db7440..7fafe1d72 100644 --- a/src/components/fresh-agent/FreshAgentView.tsx +++ b/src/components/fresh-agent/FreshAgentView.tsx @@ -21,7 +21,7 @@ import { createLogger } from '@/lib/client-logger' import { api, getFreshAgentModelCapabilities, getFreshAgentThreadSnapshot, setSessionMetadata } from '@/lib/api' import { clearReconcilePendingPane, consumePaneRefreshRequest, mergePaneContent, updatePaneContent } from '@/store/panesSlice' import { FRESH_AGENT_MODEL_CATALOG_UNAVAILABLE_NOTICE } from '@/lib/fresh-agent-model-capabilities' -import { clearPendingCreateFailure, clearRestoreFailure, clearSessionError, clearSessionLost, sessionError, setSessionStatus } from '@/store/freshAgentSlice' +import { applyRefusalFence, clearPendingCreateFailure, clearRestoreFailure, clearSessionError, clearSessionLost, sessionError, setSessionStatus } from '@/store/freshAgentSlice' import { openSessionTab } from '@/store/tabsSlice' import { buildReconcileRequestForPanes, foldVerdicts, isFreshAgentReconcileActive } from '@/lib/pane-reconcile' import { dismissTabGreen } from '@/store/turnCompletionAttention' @@ -462,6 +462,37 @@ function isLostFreshOpencodeThreadError(error: unknown): boolean { return status === 404 && code === 'FRESH_AGENT_LOST_SESSION' } +// LB-05 scoping: fresh-agent owners are the 2026-09-20 incident class (the +// pane's OWN stale-claim refusal — the daemon died while the session key +// stayed Live{FreshAgent}) and the fenced attach proceeds for them. Terminal +// owners are a different scenario (a genuinely terminal-owned session); their +// recovery door is the session-directory handoff, so the client does not +// attempt the (refused) attach for them. +function isRestoreUnavailableSnapshotError(error: unknown): boolean { + if (!error || typeof error !== 'object') return false + const status = 'status' in error ? (error as { status?: unknown }).status : undefined + const details = 'details' in error ? (error as { details?: unknown }).details : undefined + const code = details && typeof details === 'object' && 'code' in details + ? (details as { code?: unknown }).code + : undefined + const ownerKind = details && typeof details === 'object' && 'ownerKind' in details + ? (details as { ownerKind?: unknown }).ownerKind + : undefined + return status === 409 && code === 'RESTORE_UNAVAILABLE' && ownerKind === 'fresh-agent' +} + +// The 409 refusal always names its fence-relevant generation; a malformed +// envelope without one cannot fence the recovery attach, so the caller skips +// the recovery (the honest error surfaces below take it) instead of sending an +// attach bound to a stale or absent generation. +function readRestoreRefusalOwnerGeneration(error: unknown): number | undefined { + if (!error || typeof error !== 'object' || !('details' in error)) return undefined + const details = (error as { details?: unknown }).details + if (!details || typeof details !== 'object' || !('ownerGeneration' in details)) return undefined + const ownerGeneration = (details as { ownerGeneration?: unknown }).ownerGeneration + return typeof ownerGeneration === 'number' ? ownerGeneration : undefined +} + function getRestoreErrorMessage(reason: RestoreErrorReason): string { switch (reason) { case 'invalid_legacy_restore_target': @@ -802,6 +833,13 @@ export function FreshAgentView({ const revealRefreshRetryTimerRef = useRef(null) const snapshotRefreshSerialRef = useRef(0) const [snapshotRevealError, setSnapshotRevealError] = useState(null) + // 2026-09-20 incident (Task 5): the once-per-identity 409 RESTORE_UNAVAILABLE + // recovery latch. The ENTIRE recovery (fenced attach + refetch) runs at most + // once per pane identity (`${createRequestId}:${snapshotThreadId}`); a second + // 409 falls through to the honest error surfaces, never re-fetching. A + // suppressed attach restores the previous value so it does NOT consume the + // latch. + const restoreUnavailableRecoveryRef = useRef(null) // Non-null while the snapshot key is rate-limited (429/backoff): the last // good snapshot stays visible and a single retry is armed at expiry. // Task 17 also consumes this for the snapshot `trigger` query param. @@ -2818,6 +2856,69 @@ export function FreshAgentView({ })) return } + // 2026-09-20 incident: with the daemon dead, daemon-absent snapshot GETs + // answer the typed 409 RESTORE_UNAVAILABLE for as long as the session + // key stays Live{FreshAgent}. The documented recovery is the + // generation-fenced attach (a map-hit freshAgent.attach respawns the + // daemon and re-bridges server-side) — drive it ONCE per pane identity, + // then refetch. Repeated 409s fall through to the honest error surfaces + // below; never reset the pane (that is the 404 lost-thread arm above). + if (paneContent.provider === 'opencode' && isRestoreUnavailableSnapshotError(error)) { + const fresh = paneContentRef.current + const recoveryKey = `${fresh.createRequestId}:${sessionId}` + const refusalOwnerGeneration = readRestoreRefusalOwnerGeneration(error) + if ( + refusalOwnerGeneration !== undefined + && restoreUnavailableRecoveryRef.current !== recoveryKey + ) { + const previousRecoveryKey = restoreUnavailableRecoveryRef.current + restoreUnavailableRecoveryRef.current = recoveryKey + // Bind the fence to the 409's CURRENT generation — refresh the + // observed owner fence from the refusal itself (the refusal names + // the coordinator's live generation; the record's epoch is + // preserved). Without this, a stale owner record sends a + // stale-generation attach the wired server refuses with + // FENCE_REQUIRED — preserving the dead-end. The fold keys the + // CANONICAL session (Task 5 review M2): the recovery attach's + // fence read resolves the stored aliasOf chain, so a pane holding + // a superseded id must fold onto the same record the attach reads + // — the raw pane id would land on the inert alias mirror. + const canonicalSession = resolveCanonicalPaneSession(appStore.getState(), fresh) + dispatch(applyRefusalFence({ + provider: canonicalSession?.provider ?? fresh.provider, + sessionId: canonicalSession?.sessionId ?? sessionId, + ownerKind: 'fresh-agent', + ownerGeneration: refusalOwnerGeneration, + })) + attachDecisionSerialRef.current += 1 + const attempt = captureFreshAgentAttachmentAttempt(fresh) + if (sendFencedFreshAgentAttach(attempt)) { + // LB-04: a reveal-lane 409 with snapshotDirty set must refetch + // through the reveal path ('reveal' trigger), or the success-path + // reveal-dirty clear never runs and the pane hides behind the + // "Refreshing conversation" overlay forever. Otherwise refetch + // via 'manual'. + if (trigger === 'reveal' && snapshotDirtyRef.current) { + revealRefreshStartedAtRef.current = null + setSnapshotRevealError(null) + requestRevealRefresh(true) + } else { + setLoadError(null) + requestSnapshotRefresh('manual') + } + return + } + // The attach was suppressed (lifecycle superseded or attempt-key + // mismatch). Do NOT consume the one-shot recovery and do NOT + // refetch — restore the latch and fall through to the honest error + // surfaces below. + restoreUnavailableRecoveryRef.current = previousRecoveryKey + } + // Recovery already attempted for this identity (or the refusal + // carried no fenceable generation): do NOT clear errors and do NOT + // refetch again — fall through to the reveal error arm / + // setLoadError below so the user sees the honest state. + } if (trigger === 'reveal' && snapshotDirtyRef.current) { revealRefreshStartedAtRef.current = null setSnapshotRevealError(error instanceof Error ? error.message : 'Failed to refresh conversation') @@ -2879,6 +2980,7 @@ export function FreshAgentView({ // paneContentRef.current inside the effect. }, [ agentSession?.lost, + captureFreshAgentAttachmentAttempt, claudeSession, isRestoring, dispatch, @@ -2891,6 +2993,7 @@ export function FreshAgentView({ migratePendingAutoTitle, requestRevealRefresh, requestSnapshotRefresh, + sendFencedFreshAgentAttach, setLocalEcho, snapshotThreadId, snapshotRefreshNonce, diff --git a/src/store/freshAgentSlice.ts b/src/store/freshAgentSlice.ts index 9e2792e69..20c194b6f 100644 --- a/src/store/freshAgentSlice.ts +++ b/src/store/freshAgentSlice.ts @@ -694,6 +694,39 @@ const freshAgentSlice = createSlice({ } }, + /** + * 2026-09-20 incident (Task 5): refresh the observed owner fence from a + * typed refusal itself — the REST snapshot 409 RESTORE_UNAVAILABLE always + * names the coordinator's CURRENT generation, which may be newer than + * the client's record (the pane loaded while the server was restarting + * and the record fold lagged). The refusal carries no epoch, so the + * existing record's epoch is PRESERVED (the record's epoch comes from + * the server's own runtime-owner broadcasts); only the generation + * advances. The fold is ADVANCE-ONLY (Task 5 review M1), matching the + * applyRuntimeOwner invariant: a newer broadcast (e.g. gen 3) may fold + * between the server minting the refusal (gen 2) and the client + * processing it — regressing to the refusal's older generation would + * send a stale fence the wired server refuses with FENCE_REQUIRED. An + * absent record is left absent: minting one would fabricate an epoch + * the client never observed — the recovery attach then goes out + * unfenced, the wired server refuses it typed, and the pane surfaces + * that honestly instead of the store lying about the boot epoch. + */ + applyRefusalFence(state, action: PayloadAction<{ + provider: string + sessionId: string + ownerKind: 'terminal' | 'fresh-agent' + ownerGeneration: number + }>) { + const refusal = action.payload + const key = `${refusal.provider}:${refusal.sessionId}` + const existing = state.runtimeOwners[key] + if (!existing) return + if (refusal.ownerGeneration < existing.generation) return + existing.generation = refusal.ownerGeneration + existing.updatedAt = Date.now() + }, + /** * kata b8ke (round-2 review): the ready handler dispatches this BEFORE * folding the ready.runtimeOwners replay — the client resets its @@ -714,6 +747,7 @@ export const { addUserMessage, appendStreamDelta, applyRuntimeOwner, + applyRefusalFence, clearPendingCreate, clearPendingCreateFailure, clearPendingCreateFailureForSession, diff --git a/test/e2e-browser/fixtures/fake-opencode.cjs b/test/e2e-browser/fixtures/fake-opencode.cjs index 50975080e..391b044df 100644 --- a/test/e2e-browser/fixtures/fake-opencode.cjs +++ b/test/e2e-browser/fixtures/fake-opencode.cjs @@ -680,6 +680,34 @@ appendAudit({ dbPath, }) +// Daemon-death self-heal e2e (plan Task 7, the 2026-09-20 incident class): +// scripted UNREQUESTED daemon death by SELF-exit. Armed ONLY for the managed +// fresh-agent serve daemon (`opencode serve --hostname H --port P` — argv[0] +// === 'serve' and never '--pure', so the catalog probe's short-lived +// `serve --pure` sidecars are excluded). When FAKE_OPENCODE_SELF_EXIT_MARKER +// names a file that APPEARS, this daemon consumes the marker (one-shot, the +// same gate-file precedent as FAKE_OPENCODE_TUI_PARITY_CHILD_EVENT_GATE) and +// exits ON ITS OWN via process.exit — the spec never signals any PID +// (PROCESS-KILL SAFETY: creating/writing the marker file is the test's only +// death-triggering action). The respawned daemon re-arms the poll, but the +// marker is already consumed, so it stays up. With the env unset (every other +// spec) none of this code runs. +const selfExitMarkerPath = process.env.FAKE_OPENCODE_SELF_EXIT_MARKER +if (selfExitMarkerPath && command === 'serve' && !argv.includes('--pure')) { + const selfExitInterval = setInterval(() => { + if (!fs.existsSync(selfExitMarkerPath)) return + clearInterval(selfExitInterval) + try { + fs.rmSync(selfExitMarkerPath, { force: true }) + } catch { + // ignore + } + appendAudit({ event: 'self_exit', rootSessionId, childSessionId }) + process.exit(1) + }, 50) + selfExitInterval.unref?.() +} + process.stdout.write(`fake opencode ready root=${rootSessionId} child=${childSessionId}\n`) process.stdin.setEncoding('utf8') process.stdin.on('data', (data) => { diff --git a/test/e2e-browser/playwright.cloud.config.ts b/test/e2e-browser/playwright.cloud.config.ts index ebe0acce6..fbb82621e 100644 --- a/test/e2e-browser/playwright.cloud.config.ts +++ b/test/e2e-browser/playwright.cloud.config.ts @@ -39,6 +39,13 @@ export const CLOUD_SKIP_SPECS = [ 'freshopencode-db-history.spec.ts', 'freshopencode-restart-recovery.spec.ts', 'freshopencode-first-send-reload-repro.spec.ts', + // Same provider-lifecycle-timing class as its model above, plus a + // backoff-guarded daemon respawn window: 2-CPU/2-worker cloud contention + // cannot guarantee daemon-death + re-warm timing. Cloud PR coverage for + // the incident class is carried by the cloud-legal + // freshopencode-snapshot-409-recovery.spec.ts; this spec is the local-lane + // end-to-end proof. + 'freshopencode-daemon-death-selfheal.spec.ts', 'opencode-restart-recovery.spec.ts', 'opencode-terminal-restore-rust.spec.ts', // Requires codex binary diff --git a/test/e2e-browser/specs/freshopencode-daemon-death-selfheal.spec.ts b/test/e2e-browser/specs/freshopencode-daemon-death-selfheal.spec.ts new file mode 100644 index 000000000..c6c5fb526 --- /dev/null +++ b/test/e2e-browser/specs/freshopencode-daemon-death-selfheal.spec.ts @@ -0,0 +1,411 @@ +import { expect, test, type Page } from '@playwright/test' +import fsp from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import { fileURLToPath } from 'node:url' +import { openPanePicker } from '../helpers/pane-picker.js' +import { TestHarness } from '../helpers/test-harness.js' +import { RustServer } from '../helpers/rust-server.js' + +/** + * The 2026-09-20 daemon-death incident class, end to end on the LOCAL lane: + * a REAL spawned Rust server + a REAL fake `opencode serve` daemon process. + * The daemon dies an UNREQUESTED death — the fixture child consumes the + * FAKE_OPENCODE_SELF_EXIT_MARKER file and exits ON ITS OWN (the spec never + * signals any PID) — and the server-side self-heal chain must carry the pane + * through it: + * + * (1) the pane is materialized and live before the death; + * (2) the daemon self-exits (audit `self_exit` for the exact serving pid); + * (3) the pane shows the typed `OPENCODE_DAEMON_LOST` "Agent error:" banner + * (exactly ONE typed edge frame for the session); + * (4) the manager respawns the daemon automatically within a bounded wait + * (a second managed serve launch in the audit log, new pid); + * (5) the pane recovers: the revival pass pushes the idle snapshot (the + * client's transcript-refetch trigger), the refetch is SERVED by the + * respawned daemon (session_get/message_list audits at the new pid), + * the transcript still renders, and the banner is dismissible; + * (6) NO `freshAgent.turn.complete` chime during the death window — a + * crash is never a positive completion — with a positive control: the + * follow-up turn's real completion DOES chime, so the window absence + * is not vacuous. + * + * Mechanics follow freshopencode-restart-recovery.spec.ts (the harness + * pattern this spec consumes): installFakeOpencode on the spawned server's + * PATH, the RustServer + TestHarness helpers, the fake's + * FAKE_OPENCODE_AUDIT_LOG JSONL for spawn/event assertions, and deterministic + * waits on harness state. The cloud lane cannot guarantee daemon-death + + * backoff-respawn timing under 2-CPU/2-worker contention (the same + * provider-lifecycle class as its model), so the spec is registered in + * CLOUD_SKIP_SPECS; cloud-backend PR coverage is carried by the cloud-legal + * freshopencode-snapshot-409-recovery.spec.ts. + */ + +const __filename = fileURLToPath(import.meta.url) +const __dirname = path.dirname(__filename) +const fakeOpencodeSource = path.resolve(__dirname, '../fixtures/fake-opencode.cjs') + +type FakeAuditEvent = { + event?: string + pid?: number + hostname?: string + port?: number + sessionId?: string + routeDirectory?: string + prompt?: string + status?: string + argv?: string[] +} + +type FreshOpencodePaneState = { + sessionId?: string + resumeSessionId?: string + status?: string + initialCwd?: string + sessionRef?: { provider?: string; sessionId?: string } +} + +type ReceivedFrame = Record + +async function installFakeOpencode(binDir: string): Promise { + await fsp.mkdir(binDir, { recursive: true }) + const target = path.join(binDir, 'opencode') + await fsp.copyFile(fakeOpencodeSource, target) + await fsp.chmod(target, 0o755) +} + +function createSetupHome(sharedOpencodeDataDir: string) { + return async (homeDir: string): Promise => { + const xdgShare = path.join(homeDir, '.local', 'share') + const opencodeLink = path.join(xdgShare, 'opencode') + const freshellDir = path.join(homeDir, '.freshell') + await fsp.mkdir(xdgShare, { recursive: true }) + await fsp.mkdir(freshellDir, { recursive: true }) + await fsp.mkdir(sharedOpencodeDataDir, { recursive: true }) + await fsp.rm(opencodeLink, { recursive: true, force: true }).catch(() => {}) + await fsp.symlink(sharedOpencodeDataDir, opencodeLink, 'dir') + await fsp.writeFile(path.join(freshellDir, 'config.json'), JSON.stringify({ + version: 1, + settings: { + codingCli: { + enabledProviders: ['opencode'], + providers: { opencode: {} }, + }, + freshAgent: { enabled: true }, + }, + }, null, 2)) + } +} + +function createServerOptions(input: { + binDir: string + auditLogPath: string + logsDir: string + sharedOpencodeDataDir: string + selfExitMarkerPath: string + port?: number + token?: string +}) { + return { + ...(input.port ? { port: input.port } : {}), + ...(input.token ? { token: input.token } : {}), + setupHome: createSetupHome(input.sharedOpencodeDataDir), + env: { + PATH: `${input.binDir}${path.delimiter}${process.env.PATH ?? ''}`, + FAKE_OPENCODE_AUDIT_LOG: input.auditLogPath, + FAKE_OPENCODE_REQUIRE_DIRECTORY_ROUTE: '1', + FAKE_OPENCODE_SELF_EXIT_MARKER: input.selfExitMarkerPath, + FRESHELL_LOG_DIR: input.logsDir, + }, + } +} + +async function readAuditEvents(auditLogPath: string): Promise { + try { + const text = await fsp.readFile(auditLogPath, 'utf8') + return text.split('\n').filter(Boolean).map((line) => JSON.parse(line) as FakeAuditEvent) + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return [] + throw error + } +} + +/** + * The audit `launch` events of MANAGED serve daemons only — argv is + * `['serve', '--hostname', H, '--port', P]` — excluding the catalog probe's + * short-lived `serve --pure` sidecars (which share the audit log). + */ +function managedServeLaunches(events: FakeAuditEvent[]): FakeAuditEvent[] { + return events.filter((event) => + event.event === 'launch' + && Array.isArray(event.argv) + && event.argv[0] === 'serve' + && !event.argv.includes('--pure')) +} + +/** The `freshAgent.event` frames the client received for one inner event + * type and session (the server wraps every edge in the same envelope). */ +function freshAgentEventFrames( + frames: ReceivedFrame[], + sessionId: string, + innerType: string, +): ReceivedFrame[] { + return frames.filter((frame) => + frame?.type === 'freshAgent.event' + && frame?.event?.type === innerType + && frame?.event?.sessionId === sessionId) +} + +/** Capture the server→client frames of the page's Freshell /ws socket. + * Registered BEFORE page.goto so the whole session is covered. */ +function captureReceivedFrames(page: Page, wsOrigin: string, sink: ReceivedFrame[]): void { + page.on('websocket', (socket) => { + if (!socket.url().startsWith(wsOrigin)) return + socket.on('framereceived', ({ payload }) => { + try { + const frame = JSON.parse(String(payload)) + if (frame && typeof frame === 'object' && !Array.isArray(frame)) { + sink.push(frame as ReceivedFrame) + } + } catch { + // Ignore protocol frames that are not JSON. + } + }) + }) +} + +async function enableFreshOpencode(page: Page): Promise { + await page.evaluate(() => { + const harness = window.__FRESHELL_TEST_HARNESS__ + harness?.dispatch({ + type: 'connection/setAvailableClis', + payload: { opencode: true }, + }) + harness?.dispatch({ + type: 'settings/previewServerSettingsPatch', + payload: { + codingCli: { enabledProviders: ['opencode'] }, + freshAgent: { enabled: true }, + }, + }) + }) +} + +async function createFreshopencodePane(page: Page, cwd: string): Promise { + const picker = await openPanePicker(page) + await picker.getByRole('button', { name: /^Freshopencode$/i }).click({ force: true }) + const directoryInput = page.getByLabel(/^Starting directory for Freshopencode$/i) + await expect(directoryInput).toBeVisible({ timeout: 15_000 }) + await directoryInput.fill(cwd) + await directoryInput.press('Enter') + await expect(page.locator('[data-context="fresh-agent"]').last()).toBeVisible({ timeout: 15_000 }) +} + +async function getFreshOpencodePaneState(page: Page): Promise { + return page.evaluate(() => { + const state = window.__FRESHELL_TEST_HARNESS__?.getState() + const activeTabId = state?.tabs?.activeTabId + const findFreshOpencode = (node: any): any => { + if (!node) return undefined + if (node.type === 'leaf' && node.content?.kind === 'fresh-agent' && node.content.provider === 'opencode') { + return node.content + } + if (node.type === 'split') return findFreshOpencode(node.children?.[0]) ?? findFreshOpencode(node.children?.[1]) + return undefined + } + return findFreshOpencode(state?.panes?.layouts?.[activeTabId]) ?? {} + }) +} + +async function sendFreshAgentPrompt(page: Page, prompt: string): Promise { + const textbox = page.getByRole('textbox', { name: 'Chat message input' }) + await expect(textbox).toBeVisible({ timeout: 15_000 }) + await expect(textbox).not.toBeDisabled({ timeout: 15_000 }) + await textbox.fill(prompt) + await page.getByRole('button', { name: 'Send' }).click() +} + +async function waitForMaterializedSession(page: Page): Promise { + await expect.poll(async () => getFreshOpencodePaneState(page), { timeout: 30_000 }).toMatchObject({ + sessionId: expect.stringMatching(/^ses_/), + resumeSessionId: expect.stringMatching(/^ses_/), + initialCwd: expect.any(String), + sessionRef: { + provider: 'opencode', + sessionId: expect.stringMatching(/^ses_/), + }, + }) + const state = await getFreshOpencodePaneState(page) + expect(state.sessionId).toBe(state.sessionRef?.sessionId) + return state +} + +async function waitForSettledPane(page: Page, sessionId: string): Promise { + await expect.poll(async () => { + const state = await getFreshOpencodePaneState(page) + return { + sessionId: state.sessionId, + status: state.status, + } + }, { timeout: 30_000 }).toEqual({ + sessionId, + status: 'idle', + }) +} + +test.describe('Freshopencode daemon-death self-heal (local lane)', () => { + test.setTimeout(240_000) + + test('daemon self-exit self-heals end to end: loss banner, backoff respawn, revival refetch, no chime', async ({ page }) => { + const sharedRoot = await fsp.mkdtemp(path.join(os.tmpdir(), 'freshell-freshopencode-daemon-death-')) + const binDir = path.join(sharedRoot, 'bin') + const logsDir = path.join(sharedRoot, 'logs') + const auditLogPath = path.join(sharedRoot, 'fake-opencode-audit.jsonl') + const sharedOpencodeDataDir = path.join(sharedRoot, 'opencode-data') + const selfExitMarkerPath = path.join(sharedRoot, 'daemon-self-exit.marker') + const cwd = path.join(sharedRoot, 'project') + const firstPrompt = `freshopencode daemon-death first ${Date.now()}` + const followUpPrompt = `freshopencode daemon-death follow-up ${Date.now()}` + await fsp.mkdir(cwd, { recursive: true }) + await installFakeOpencode(binDir) + + const server = new RustServer(createServerOptions({ + binDir, + auditLogPath, + logsDir, + sharedOpencodeDataDir, + selfExitMarkerPath, + })) + + const receivedFrames: ReceivedFrame[] = [] + try { + const info = await server.start() + captureReceivedFrames(page, info.wsUrl, receivedFrames) + await page.goto(`${info.baseUrl}/?token=${info.token}&e2e=1`) + const harness = new TestHarness(page) + await harness.waitForHarness() + await harness.waitForConnection() + await enableFreshOpencode(page) + await createFreshopencodePane(page, cwd) + + // (1) The pane is materialized and live. + await sendFreshAgentPrompt(page, firstPrompt) + await expect(page.getByText(`Fake OpenCode response: ${firstPrompt}`)).toBeVisible({ timeout: 30_000 }) + const materialized = await waitForMaterializedSession(page) + expect(materialized.initialCwd).toBe(cwd) + const sessionId = materialized.sessionId! + await waitForSettledPane(page, sessionId) + + const eventsBeforeDeath = await readAuditEvents(auditLogPath) + const launchesBeforeDeath = managedServeLaunches(eventsBeforeDeath) + expect(launchesBeforeDeath.length, 'the pane is served by a managed daemon before the death').toBeGreaterThanOrEqual(1) + const firstDaemonPid = launchesBeforeDeath[launchesBeforeDeath.length - 1].pid! + const eventCountBeforeDeath = eventsBeforeDeath.length + const deathWindowStartIndex = receivedFrames.length + + // (2) The daemon dies an UNREQUESTED death: the fixture child consumes + // the marker and exits ON ITS OWN — this writeFile is the test's ONLY + // death-triggering action; no PID is ever signaled. + await fsp.writeFile(selfExitMarkerPath, 'self-exit now\n') + await expect.poll(async () => { + const events = await readAuditEvents(auditLogPath) + return events.slice(eventCountBeforeDeath).find((event) => + event.event === 'self_exit' && event.pid === firstDaemonPid) ?? null + }, { timeout: 15_000 }).toBeTruthy() + + // (3) The pane shows the OPENCODE_DAEMON_LOST "Agent error:" banner. + const daemonLostBanner = page.getByRole('alert').filter({ + hasText: 'The opencode serve daemon was lost unexpectedly', + }) + await expect(daemonLostBanner).toBeVisible({ timeout: 30_000 }) + await expect(daemonLostBanner).toContainText('Agent error:') + + // The typed edge: exactly ONE OPENCODE_DAEMON_LOST frame for the + // materialized session. + await expect.poll(async () => + freshAgentEventFrames(receivedFrames, sessionId, 'freshAgent.error') + .filter((frame) => frame.event.code === 'OPENCODE_DAEMON_LOST').length + , { timeout: 30_000 }).toBe(1) + + // (4) The daemon respawns automatically within a bounded wait — a + // second managed serve launch with a NEW pid in the audit log. + let respawnedPid: number | undefined + await expect.poll(async () => { + const events = await readAuditEvents(auditLogPath) + respawnedPid = managedServeLaunches(events) + .find((event) => event.pid !== firstDaemonPid)?.pid + return respawnedPid ?? null + }, { timeout: 30_000 }).toBeTruthy() + + // (5) The pane recovers: the revival pass pushed the idle snapshot + // (the client's transcript-refetch trigger)... + await expect.poll(async () => + freshAgentEventFrames(receivedFrames, sessionId, 'freshAgent.session.snapshot') + .filter((frame) => frame.event.status === 'idle').length + , { timeout: 30_000 }).toBeGreaterThanOrEqual(1) + // ...and the transcript refetch was SERVED by the respawned daemon + // (session_get + message_list audits at the new pid; the refetch is + // the only client-driven daemon read in this window). + await expect.poll(async () => { + const events = await readAuditEvents(auditLogPath) + const afterRespawn = events.filter((event) => event.pid === respawnedPid) + return { + sessionGet: afterRespawn.some((event) => + event.event === 'session_get' && event.sessionId === sessionId), + messageList: afterRespawn.some((event) => + event.event === 'message_list' && event.sessionId === sessionId), + } + }, { timeout: 30_000 }).toEqual({ sessionGet: true, messageList: true }) + + // The refetched transcript still renders the first turn. + await expect(page.getByText(`Fake OpenCode response: ${firstPrompt}`)).toBeVisible({ timeout: 15_000 }) + + // (6) NO freshAgent.turn.complete during the death window (a crash is + // never a positive completion). Asserted BEFORE the follow-up turn so + // the window contains only the death→revival frames. + const deathWindowFrames = receivedFrames.slice(deathWindowStartIndex) + const chimeFramesDuringDeath = deathWindowFrames.filter((frame) => + frame?.type === 'freshAgent.event' && frame?.event?.type === 'freshAgent.turn.complete') + expect(chimeFramesDuringDeath, 'no chime frames during the daemon-death window').toEqual([]) + + // The pane kept its identity through the death and recovery. + const paneStateAfterRecovery = await getFreshOpencodePaneState(page) + expect(paneStateAfterRecovery.sessionId).toBe(sessionId) + expect(paneStateAfterRecovery.status).not.toBe('create-failed') + + // The banner is dismissible — no dead-end remains. + await daemonLostBanner.getByRole('button', { name: 'Dismiss' }).click() + await expect(daemonLostBanner).toHaveCount(0) + + // The composer works against the RESPAWNED daemon: a follow-up turn + // is served end to end by the new pid. + const framesBeforeFollowUp = receivedFrames.length + await sendFreshAgentPrompt(page, followUpPrompt) + await expect(page.getByText(`Fake OpenCode response: ${followUpPrompt}`)).toBeVisible({ timeout: 30_000 }) + await waitForSettledPane(page, sessionId) + await expect.poll(async () => { + const events = await readAuditEvents(auditLogPath) + return events.some((event) => + event.event === 'prompt_async' + && event.sessionId === sessionId + && event.routeDirectory === cwd + && event.prompt === followUpPrompt + && event.pid === respawnedPid) + }, { timeout: 15_000 }).toBe(true) + + // Positive control for the chime absence: the follow-up turn's real + // positive completion DID emit freshAgent.turn.complete — the lane is + // alive, so the death-window silence was not vacuous. + await expect.poll(async () => + receivedFrames.slice(framesBeforeFollowUp) + .filter((frame) => + frame?.type === 'freshAgent.event' + && frame?.event?.type === 'freshAgent.turn.complete' + && frame?.event?.sessionId === sessionId).length + , { timeout: 15_000 }).toBeGreaterThanOrEqual(1) + } finally { + await server.stop().catch(() => {}) + await fsp.rm(sharedRoot, { recursive: true, force: true }).catch(() => {}) + } + }) +}) diff --git a/test/e2e-browser/specs/freshopencode-snapshot-409-recovery.spec.ts b/test/e2e-browser/specs/freshopencode-snapshot-409-recovery.spec.ts new file mode 100644 index 000000000..d27ecd829 --- /dev/null +++ b/test/e2e-browser/specs/freshopencode-snapshot-409-recovery.spec.ts @@ -0,0 +1,323 @@ +import type { Page } from '@playwright/test' +import { test, expect } from '../helpers/fixtures.js' +import { openPanePicker } from '../helpers/pane-picker.js' +import fsp from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' + +/** + * The 2026-09-20 daemon-death incident class, end to end (the cloud-legal + * leg): a freshopencode pane whose snapshot GET first answers the typed 409 + * RESTORE_UNAVAILABLE (ownerKind fresh-agent + the coordinator's CURRENT + * ownerGeneration — the exact envelope the Rust server mints for a session + * key that stayed Live{FreshAgent} across the daemon death) must NOT + * dead-end on the dismiss-only banner. It drives the documented recovery + * ONCE per pane identity: refresh the observed owner fence from the + * refusal's own generation, send ONE generation-fenced freshAgent.attach, + * and refetch through the 200. + * + * Mechanics follow freshopencode-model-picker.spec.ts exactly — every fetch + * is routed and the sidecar is suppressed through the test harness — so the + * spec needs no opencode binary and no provider-boot timing: the explicitly + * cloud-legal pattern (playwright.cloud.config.ts:36-38). It must never + * join CLOUD_SKIP_SPECS / LOCAL_ONLY_SPECS or match a CLOUD_SKIP_TITLES + * pattern. + */ + +const SESSION_ID = 'ses_e2e_409' +const RECOVERED_TEXT = 'Recovered transcript after the fenced attach and refetch.' +/** The pane's OWN stale Live{FreshAgent, gen 1} claim — the incident shape. */ +const SEEDED_RECORD_EPOCH = 1 +const SEEDED_RECORD_GENERATION = 1 +/** + * The refusal names the coordinator's CURRENT generation — newer than the + * client's stale record. The recovery attach MUST carry this generation (the + * fence binds to the refusal, not the possibly-stale record), which is what + * makes the post-409 recovery attach distinguishable from the mount attach + * in the WS spy (LB-09: a bare attach-count assertion is vacuous — the mount + * attach also appears in the spy log). + */ +const REFUSAL_OWNER_GENERATION = 2 + +/** A minimal valid FreshAgentSnapshot (schema-complete: tokenUsage is + * required by FreshAgentSnapshotSchema) whose assistant turn is the + * recovered transcript the 200 refetch must render. */ +function recoveredSnapshot() { + return { + sessionType: 'freshopencode', + provider: 'opencode', + threadId: SESSION_ID, + sessionId: SESSION_ID, + revision: 2, + latestTurnId: 'msg_recovered_1', + status: 'idle', + capabilities: { + send: true, + interrupt: true, + approvals: true, + questions: true, + fork: true, + }, + tokenUsage: { + inputTokens: 0, + outputTokens: 0, + totalTokens: 0, + costUsd: 0, + }, + pendingApprovals: [], + pendingQuestions: [], + turns: [ + { id: 'msg_user_1', turnId: 'msg_user_1', role: 'user', summary: 'go', items: [{ id: 'user-text', kind: 'text', text: 'go' }] }, + { id: 'msg_recovered_1', turnId: 'msg_recovered_1', role: 'assistant', summary: RECOVERED_TEXT, items: [{ id: 'assistant-text', kind: 'text', text: RECOVERED_TEXT }] }, + ], + } +} + +async function enableFreshClientsAndOpencode(page: Page): Promise { + await page.evaluate(() => { + const harness = window.__FRESHELL_TEST_HARNESS__ + harness?.dispatch({ + type: 'connection/setAvailableClis', + payload: { claude: true, codex: true, opencode: true }, + }) + harness?.dispatch({ + type: 'settings/previewServerSettingsPatch', + payload: { + codingCli: { enabledProviders: ['claude', 'codex', 'opencode'] }, + freshAgent: { enabled: true }, + }, + }) + }) +} + +async function routeFileApis(page: Page, cwd: string): Promise { + await page.route('**/api/files/candidate-dirs', async (route) => { + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ directories: ['/tmp'] }), + }) + }) + await page.route('**/api/files/validate-dir', async (route) => { + const body = route.request().postDataJSON() as { path?: string } + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ valid: true, resolvedPath: body?.path ?? cwd }), + }) + }) +} + +/** + * Seed the runtime-owner record the incident premise needs: the session key + * stayed Live{FreshAgent, gen 1} while the daemon was dead, so the client + * holds a stale owner record the 409's fence must refresh (applyRefusalFence + * is advance-only on the EXISTING record — without it there is no fence to + * refresh and the recovery attach would go out unfenced). + */ +async function seedIncidentOwnerRecord(page: Page): Promise { + await page.evaluate((record) => { + window.__FRESHELL_TEST_HARNESS__?.dispatch({ + type: 'freshAgent/applyRuntimeOwner', + payload: record, + }) + }, { + type: 'session.runtimeOwner', + provider: 'opencode', + sessionId: SESSION_ID, + epoch: SEEDED_RECORD_EPOCH, + generation: SEEDED_RECORD_GENERATION, + ownerKind: 'fresh-agent', + operationId: 'e2e-incident-live-claim', + transition: 'handoff-committed', + }) +} + +/** + * Create a freshopencode pane through the picker (the model-picker pattern) + * and hand it the durable ses_* session — the pane a user re-opens from a + * persisted layout while the server-side key is still Live{FreshAgent}. + * Sweeping caveat: the picker pane is REPLACED by the real fresh-agent pane + * with a NEW pane id on selection, so the suppression flag and the session + * handoff must target the post-creation fresh-agent leaf in + * state.panes.layouts, never an id captured from the picker DOM. + */ +async function createFreshopencodePaneWithDurableSession(page: Page, cwd: string): Promise { + // Suppress ALL fresh-agent network effects BEFORE the pane exists: creation + // fires its session-create effect immediately, and a per-pane flag set after + // the fact races it. Every freshAgent.* frame then lands in the harness's + // sent-message spy (getSentWsMessages) instead of the wire — including the + // mount attach AND the post-409 recovery attach this spec counts. + await page.evaluate(() => { + window.__FRESHELL_TEST_HARNESS__?.setSuppressAllFreshAgentNetworkEffects(true) + }) + const picker = await openPanePicker(page) + await expect(picker.getByRole('button', { name: /^Freshopencode$/i })).toBeVisible({ timeout: 10_000 }) + await picker.getByRole('button', { name: /^Freshopencode$/i }).click({ force: true }) + const directoryInput = page.getByLabel(/^Starting directory for Freshopencode$/i) + await expect(directoryInput).toBeVisible({ timeout: 15_000 }) + await directoryInput.fill(cwd) + await directoryInput.press('Enter') + await expect(page.locator('[data-context="fresh-agent"]').last()).toBeVisible({ timeout: 15_000 }) + + await page.evaluate((session) => { + const harness = window.__FRESHELL_TEST_HARNESS__ + if (!harness) return + const state = harness.getState() + const tabId = state.tabs.activeTabId as string | undefined + if (!tabId) return + // The tab may be a SPLIT (terminal + fresh pane), so walk the tree for + // the fresh-agent leaf rather than assuming the root is a leaf. + type LayoutNode = { id: string; type: string; content?: { kind?: string }; children?: LayoutNode[] } + const findFreshLeaf = (node: LayoutNode | undefined): LayoutNode | undefined => { + if (!node) return undefined + if (node.type === 'leaf' && node.content?.kind === 'fresh-agent') return node + for (const child of node.children ?? []) { + const found = findFreshLeaf(child) + if (found) return found + } + return undefined + } + const leaf = findFreshLeaf(state.panes.layouts[tabId] as LayoutNode | undefined) + if (!leaf) return + harness.dispatch({ + type: 'panes/updatePaneContent', + payload: { + tabId, + paneId: leaf.id, + content: { + ...leaf.content, + sessionId: session.sessionId, + sessionRef: { provider: 'opencode', sessionId: session.sessionId }, + resumeSessionId: session.sessionId, + status: 'idle', + }, + }, + }) + }, { sessionId: SESSION_ID }) +} + +/** The active tab's fresh-agent leaf content, via the harness store. */ +async function readFreshAgentPaneContent(page: Page): Promise | undefined> { + return page.evaluate(() => { + const harness = window.__FRESHELL_TEST_HARNESS__ + if (!harness) return undefined + const state = harness.getState() + const tabId = state?.tabs?.activeTabId as string | undefined + if (!tabId) return undefined + type LayoutNode = { id: string; type: string; content?: { kind?: string }; children?: LayoutNode[] } + const findFreshLeaf = (node: LayoutNode | undefined): LayoutNode | undefined => { + if (!node) return undefined + if (node.type === 'leaf' && node.content?.kind === 'fresh-agent') return node + for (const child of node.children ?? []) { + const found = findFreshLeaf(child) + if (found) return found + } + return undefined + } + return findFreshLeaf(state.panes.layouts[tabId] as LayoutNode | undefined)?.content as Record | undefined + }) +} + +test.describe('Freshopencode snapshot-409 recovery (cloud-legal)', () => { + test('freshopencode pane recovers from a snapshot 409 via fenced attach and refetch', async ({ + freshellPage, + page, + harness, + terminal, + }) => { + test.setTimeout(120_000) + await terminal.waitForTerminal() + const cwd = await fsp.mkdtemp(path.join(os.tmpdir(), 'freshell-snapshot-409-')) + + // Route the snapshot thread path BEFORE any pane exists: the FIRST GET + // answers the incident-class typed 409 (the snapshot_error_response + // envelope the Rust server mints for a Live{FreshAgent} key), every + // subsequent GET answers the recovered 200 snapshot. + const servedStatuses: number[] = [] + await page.route(`**/api/fresh-agent/threads/freshopencode/opencode/${SESSION_ID}**`, async (route) => { + if (servedStatuses.length === 0) { + servedStatuses.push(409) + await route.fulfill({ + status: 409, + contentType: 'application/json', + body: JSON.stringify({ + status: 'error', + code: 'RESTORE_UNAVAILABLE', + message: `Session ${SESSION_ID} is still running on the server.`, + ownerKind: 'fresh-agent', + ownerGeneration: REFUSAL_OWNER_GENERATION, + }), + }) + return + } + servedStatuses.push(200) + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify(recoveredSnapshot()), + }) + }) + await routeFileApis(page, cwd) + await enableFreshClientsAndOpencode(page) + await seedIncidentOwnerRecord(page) + await createFreshopencodePaneWithDurableSession(page, cwd) + + // The mount snapshot GET landed the typed 409 (the incident premise — + // a filter that never saw the 409 proves nothing). + await expect.poll(() => servedStatuses[0] ?? 0, { timeout: 20_000 }).toBe(409) + + // The post-409 delta (LB-09): a freshAgent.attach carrying the refusal's + // CURRENT generation appeared in the WS spy AFTER the 409 landed. Only + // applyRefusalFence can mint generation 2 on the seeded gen-1 record, so + // this frame is the recovery attach — the mount attach is the gen-1 one. + await expect.poll(async () => { + const sent = (await harness.getSentWsMessages()) as Array> + return sent.filter((message) => ( + message?.type === 'freshAgent.attach' + && message.observedGeneration === REFUSAL_OWNER_GENERATION + )).length + }, { timeout: 20_000 }).toBeGreaterThanOrEqual(1) + + // The recovery refetch landed and was answered 200. + await expect.poll(() => servedStatuses[1] ?? 0, { timeout: 20_000 }).toBe(200) + + // The transcript renders from the 200 snapshot — the pane is live again. + const transcript = page.locator('[data-context="fresh-agent-transcript"]') + await expect(transcript.getByText(RECOVERED_TEXT)).toBeVisible({ timeout: 15_000 }) + + // Settled state: exactly ONE recovery attach (the once-per-identity + // guard), carrying the record's PRESERVED epoch and the refusal's + // CURRENT generation, and exactly one recovery refetch (no 409 fetch + // loop). + const sent = (await harness.getSentWsMessages()) as Array> + const attaches = sent.filter((message) => message?.type === 'freshAgent.attach') + const mountAttaches = attaches.filter((message) => ( + message.observedEpoch === SEEDED_RECORD_EPOCH + && message.observedGeneration === SEEDED_RECORD_GENERATION + )) + expect(mountAttaches.length, 'the mount attach (fenced at the seeded record) must also be in the spy — the delta is not vacuous').toBeGreaterThanOrEqual(1) + const recoveryAttaches = attaches.filter((message) => message.observedGeneration === REFUSAL_OWNER_GENERATION) + expect(recoveryAttaches, 'exactly one post-409 recovery attach per pane identity').toHaveLength(1) + expect(recoveryAttaches[0]).toMatchObject({ + sessionType: 'freshopencode', + provider: 'opencode', + sessionId: SESSION_ID, + observedEpoch: SEEDED_RECORD_EPOCH, + observedGeneration: REFUSAL_OWNER_GENERATION, + }) + expect(servedStatuses, 'one mount 409 + one recovery 200 refetch, no loop').toEqual([409, 200]) + + // The pane kept its identity (the 409 arm is NOT the 404 lost-thread + // reset — that arm clears the session). + const content = await readFreshAgentPaneContent(page) + expect(content?.sessionId).toBe(SESSION_ID) + expect(content?.status).not.toBe('create-failed') + + // No dismiss-only dead-end banner remains, and the composer is usable + // again — the pane recovered instead of dead-ending. + await expect(page.getByRole('alert').filter({ hasText: 'still running on the server' })).toHaveCount(0) + const composer = page.getByRole('textbox', { name: 'Chat message input' }) + await expect(composer).toBeEnabled({ timeout: 15_000 }) + }) +}) diff --git a/test/unit/client/components/fresh-agent/FreshAgentView.test.tsx b/test/unit/client/components/fresh-agent/FreshAgentView.test.tsx index 541d36b77..205069b60 100644 --- a/test/unit/client/components/fresh-agent/FreshAgentView.test.tsx +++ b/test/unit/client/components/fresh-agent/FreshAgentView.test.tsx @@ -168,9 +168,11 @@ function createStore(tabTitleSetByUser = false, extraMiddleware: Middleware[] = function StoreBackedFreshAgentView({ tabId, paneId, + hidden = false, }: { tabId: string paneId: string + hidden?: boolean }) { const paneContent = useAppSelector((state) => { const layout = state.panes.layouts[tabId] @@ -179,7 +181,7 @@ function StoreBackedFreshAgentView({ } return layout.content }) - return + return