daemon: input delivery is activity — round-dispatch grace hardening - #537
Conversation
HandleSendInput never touched AgentActivityClock, so an agent that only ever received input (never producing its own output before the next reap sweep) looked idle to the reviewer reaper even while actively in use. Advance the same clock PTY output/ACP envelopes/turn transitions already use, gated on the delivery await returning without throwing so a failed/cancelled write advances nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up on the previous commit: add an ACP-mechanism test (SeedAcpAgent + the existing FakeAcpRuntime double) pinning that a non-throwing SendUserInputAsync (enqueue-accepted, not "the agent read it") also advances the clock — the deliberate, documented residual for that mechanism. SeedAcpAgent gains an optional activityClock override, mirroring SeedAgentForTest's existing param, so the test can control idle timing precisely. Also update AgentActivityClock's class/Advance doc comments to list delivered SendInput as a fifth activity source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A delivered SendInput now advances the activity clock (previous commit), but the server only learns about it on the next 60s report tick — long after the dispatch-relative grace it feeds has elapsed. Emit one DaemonStatusReport per successfully handled input invocation, right after the advance and gated on the same delivery success, fire-and-forget so a send failure can never fail the delivery. SendInputCommand carries no round/dispatch identity, so "one per round" is not implementable and duplicates are tolerated by design (a duplicate is content-honest). That second emitter makes report ordering load-bearing rather than hygiene: the server folds every flow-participant report and permanently latches "regressed" on ANY activity-seq regression, disabling that agent's liveness supervision for good. A report whose content was captured before the delivery but sent after the delivery-triggered one would present seq N after seq N+1 and trip exactly that latch — silently disabling supervision, the opposite failure direction. So every emission now captures its snapshot AND completes its hub send under one per-daemon SemaphoreSlim section, held through DaemonStatusReportAsync returning rather than merely through task creation. All existing emitters (periodic loop, on-request nudge, launch-stage transition) already funnel through SendDaemonStatusReportOnceAsync, so the gate lives there and covers them; the field doc states that every future emitter must come through it too. The settlement ack redelivery stays outside the section — it re-sends acks, not the report. Tests pin: the delivery emits exactly one report carrying the advanced seq and reset idle; a failed delivery emits none; content is monotone in send-COMPLETION order, driven by a probe connection that parks its first send (removing the gate makes the second emission overtake the parked one, which the test catches); and the report shape carries no correlation nonce, so an unsolicited report cannot masquerade as a correlated reply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the receive loop Review follow-up on the previous commit, three findings. The ordering test killed gate-removal and release-before-send but SURVIVED hoisting BuildStatusReport() above the acquire: both emissions build the same content either way, so entry count and wire order stayed as asserted. That mutation is unsound, not untidy — two emissions building outside the section can enqueue on the semaphore in the opposite order to their capture order (SemaphoreSlim is only approximately FIFO), putting older content on the wire second and permanently latching the server's fold into regressed. Pin it deterministically: once the EnteredCount == 1 assertion has proven the second emission is still parked on the gate, advance the clock again; a snapshot captured inside the section reads the new seq, a hoisted one is frozen at the delivery-time seq. Verified by hand-running that mutation — it now fails on the added assertion. The delivery-site emission was a bare discard, which is only asynchronous when it blocks: on an uncontended gate WaitAsync completes synchronously, so BuildStatusReport() ran inline on the SignalR receive-loop thread while this agent's BorrowedSnapshotGate was still held — and that build reads the PID record store off disk twice. Offload it with Task.Run; ordering is unaffected, since content is made monotone by the gate rather than by the order emissions are started in. Also name the delivered-input report as the fourth trigger of the settlement ack redelivery, which it silently moves from the 60s cadence to input rate (benign: normally-empty set, and duplicate acks are tolerated by design). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FindReviewersToReap selects from a snapshot and the teardown happens later, so a reviewer handed its next round in between was still torn down under it — a recheck immediately before the stop only narrows that window, it does not close it. The claim now runs inside the per-agent section that already wraps every vendor's whole delivery body, so a delivery's clock advance and the reaper's claim are mutually exclusive and exactly one side wins: the claim re-validates incarnation and the activity generation captured at selection and aborts if anything advanced; a claim that lands first marks the agent, and the next delivery refuses rather than writing into a runtime about to be terminated. Selection captures the generation BEFORE its threshold reads, which is what makes every interleaving of the clock's independent lock acquisitions safe. Scoped to the idle and wedge rules — both are 'nothing has happened' claims a delivery falsifies. The absolute lifetime cap reaps regardless of activity, unchanged. The end-reason stamp moved into the won claim, so an aborted reap no longer leaves a reap reason on a live agent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e lifetime cap The fence introduced a circular wait: a delivery holds the per-agent section across UnixPtyProcess.WriteAsync — a raw write(2) on the pty master, no timeout, not cancellable — which parks indefinitely if the child stops draining stdin, and the reap that would terminate that child (the only thing that releases the write) queued behind it. The unconditional reap used to break that cycle by construction. The claim now waits a bounded 20s (under the 30s heartbeat, so at most one waiter per agent is alive) and the two rules answer a timeout oppositely: a fenced idle/wedge claim gives up, since an in-progress delivery is activity anyway, while the absolute lifetime cap claims WITHOUT the section and proceeds — a rule a possibly-never- completing write can defer is not absolute. The claim latch is therefore an interlocked 0/1 CAS shared by both paths rather than a gate-guarded bool. Also: AgentActivityClock.Snapshot() gives selection all four observables under one lock acquisition, replacing the four separate reads and the ordering argument they needed; the reap sweep excludes IsPrivate explicitly, matching the stuck-Starting sweep beside it; both gates document the lock ordering they rely on (per-agent -> clock and ordering -> clock, never per-agent -> ordering, which is also why the delivery-triggered report is offloaded); and the single-flight claim collision logs instead of returning silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… write Two gaps the bounded claim left open. The circular wait was broken at the claim but not at the stop: StopAgentCoreAsync awaited RequestGracefulStopAsync() unbounded, and for a PTY that writes /exit to the same uncancellable master fd the delivery was parked on — so against a child that has stopped draining stdin the graceful send parked too and TerminateAsync, the one step that ends the wedge, was never reached. The hang is pre-existing (every stop, incl. a user's, could hit it) but the reaper now depends on reaching terminate, so the send is bounded by the existing GracefulExitWait and the abandoned task observed. No legitimate graceful path is truncated: ACP sends one notify, Antigravity returns immediately, Pi's own grace is 3s. And the delivery-side refusal was checked only on entry to the section, while the steps after it are slower than the claim's gate wait (the borrowed-snapshot refresh alone is budgeted 30s) — so the unfenced claim fires against healthy in-flight deliveries, which then completed into a condemned agent: wrote the round, advanced the clock, emitted a report. The latch is monotonic, so re-reading it immediately before the write is free and sound. Tests: the parked-delivery double gains a park-every-write mode (a non-draining child parks /exit too, which the earlier arms let through); a new arm proves the reap reaches terminate through it; another proves a healthy mid-flight delivery refuses at the pre-write re-read. That one needed a window seam — parking a borrowed refresh instead was vacuous, since the refresh's own auth failure returns before the write. Also pins ReapClaimGateWait < HeartbeatInterval, now a named constant rather than a comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR Summary by QodoDaemon: treat delivered input as activity and harden reap/report ordering
AI Description
Diagram
High-Level Assessment
Files changed (8)
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f049f0efe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| try { | ||
| if (!await TryClaimReapAsync(candidate)) return; | ||
|
|
||
| await HandleStopAgent(candidate.Id); |
There was a problem hiding this comment.
Stop the claimed agent instance, not a reused ID
If the selected reviewer exits and cleanup/relaunch replaces its _agents entry after TryClaimReapAsync validates the captured instance but before this call runs, HandleStopAgent(candidate.Id) resolves the reused ID again and can terminate the fresh incarnation. This defeats the incarnation check carried by ReapCandidate; pass the claimed candidate.Agent into the stop executor or revalidate ReferenceEquals at the stop boundary.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 3b364e1: the post-claim stop is split into StopClaimedReapAsync, which re-validates ReferenceEquals(_agents[candidate.Id], candidate.Agent) immediately before HandleStopAgent and aborts with the existing agent_gone log on mismatch — the claimed instance, not the reused id, is now what authorizes the stop. Agent ids are unique per launch today, so this makes the property structural rather than reachable; pinned by Relaunch_between_claim_and_stop_aborts_the_reap (map entry swapped between claim and stop, reap aborts).
Code Review by Qodo
1.
|
- Move the graceful-stop request inside the inner try so a synchronous throw falls through to terminate instead of skipping it via the outer catch (a reap-path skip would leave a permanent zombie). - Drop a stale duplicate GracefulExitWait <summary> block. - Reword the delivery/reap gate doc: the true rule is that a waiter whose own action unblocks the holder must bound its wait; the delivery's entry wait is exempt (holder class, not bounded class). - Give the late in-section reap-claim refusal its own log message, distinguishing it from the entry-check refusal. - Offload the on-request status-report nudge via Task.Run, matching the delivery site, so it doesn't park the SignalR receive loop behind a gated hub send. - Correct AgentActivityClock's source-list doc: six families, not five (was missing Antigravity/Pi agentActivity and LocalPermission- Bridge's second site). - Update PR #537's body to match: sixth clock source (not fifth), parked-delivery deferral semantics, and six claim tests + a relation pin (not seven claim tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, tighten comments Three follow-up review findings on round-dispatch grace: - ReapReviewerAsync's stop now re-validates the claimed AgentInstance is still the live _agents entry (StopClaimedReapAsync) immediately before handing off to HandleStopAgent, which re-resolves by id. Unreachable today (agent ids are unique per launch) but closes the TOCTOU structurally instead of relying on that invariant holding forever. - SendDaemonStatusReportOnceAsync no longer holds _statusReportOrderingGate for an unbounded hub send: the gate acquisition and the send invocation are each bounded by _shutdownCts, and the WAIT for send completion is capped by the new StatusReportSendTimeout, releasing the gate on timeout without waiting further (invocation order, not completion order, is what the gate's wire-FIFO invariant needs). The abandoned send is observed so a later fault isn't an unobserved task exception. - Compressed the essay-length comment blocks around the ordering gate and HandleSendInput's reap-claim checks into concise constraint statements, pointing at the spec for rationale instead of re-deriving it inline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Summary
This is the daemon (
kcap-cli) half of kurrent-io/kcap-server#1403 — round-dispatch grace for the participant-inactivity rule. Design:docs/superpowers/specs/2026-08-10-ai1842-round-dispatch-grace-design.md(kcap-server, PR #1403).The server side treats a stopped participant as reachable again once it can prove an in-flight round; this PR gives the daemon the two things that proof needs: input delivery is now activity (so a round that's actually being worked doesn't look idle), and a delivery-triggered status report so the server learns that fact promptly instead of waiting out the 60s cycle. It also closes the gap that made "input delivery is activity" unsafe on its own — a reviewer being idle-reaped while a round is mid-delivery.
What changed
1. Delivered input advances the per-agent activity clock — the sixth source.
HandleSendInputnow callsAgentActivityClock.Advance()on the delivery success path, joining PTY output, ACP envelope emission and turn start/end, Antigravity's and Pi's ownagentActivityevents, andLocalPermissionBridge's two reviewer tool-call-hit sites as clock sources. "Successfully handled" is decided per vendor mechanism: PTY and RPC throw on a genuine delivery failure, and Antigravity's exec-per-turn queue explicitly faults even its fire-and-forget branch — all three are clean. ACP's default (non-borrowed) path is not:EnqueueTurn(acknowledgeWrite: false)returns success on enqueue, not on the agent actually reading it, so a full_pendingTurnsqueue silently drops the input while still looking like a successful delivery. That residual is accepted and documented as kill-delaying-only — it can make a genuinely stuck reviewer look briefly alive to the clock, but it can never manufacture activity that isn't there for a healthy one, and it never suppresses a real timeout, only delays it by however long the delivery pipeline was already broken.2. A delivery-triggered, uncorrelated status report — and the ordering gate it needs. Every delivered input now fires an out-of-cycle
DaemonStatusReport(fire-and-forget, off the SignalR receive loop viaTask.Run, matching the existing launch-stage-triggered report), so the server sees the advancedActivitySeq/resetIdleForMswithout waiting on the 60s periodic loop. This is only safe because of a new per-daemon_statusReportOrderingGate(SemaphoreSlim(1,1)), held from snapshot construction (BuildStatusReport(), evaluated inside the section) through completion of the hub send — never just through task creation. Without it, two concurrent emissions can enqueue on the semaphore in the opposite order from their content's causal order (SemaphoreSlimis only approximately FIFO), putting older content on the wire after newer content and permanently tripping the server'sRegressedlatch on wire-order content regression. All existing emission sites (periodic loop, legacy on-request nudge, launch-stage transition) already funnel through one method (SendDaemonStatusReportOnceAsync), so the gate covers them for free; the new delivery-triggered report is the second site added by this PR.3. The atomic reap claim — fencing the reviewer-idle-reap race against the delivery window. Before this change, the heartbeat's
FindReviewersToReapselection and the actual stop were separated only by an "does the agent still exist" check — no revalidation that the agent was still idle by the time the stop executed. Now:HandleSendInputalready holds across its whole delivery body (BorrowedSnapshotGate, extended rather than renamed — its doc now names both purposes) — so a delivery and a reap claim are mutually exclusive.ReapClaimGateWait, 20s, deliberately under the 30s heartbeat period) and falls back to a lock-free CAS claim for the unfenced rule (the absoluteReviewerMaxLifetimecap) on timeout — this closes a real circular-wait hazard: an unbounded wait would let a parked, non-draining PTYwrite(2)(no timeout, no cancellation) defer the reap that's supposed to end it, forever.FindReviewersToReap's idle/wedge rules never select an agent whose delivery is still parked in the section, and reclaim shifts entirely to the unfenced absolute-lifetime cap (ReviewerMaxLifetime, ≤6h by default). WithReviewerMaxLifetime=0(that rule disabled) a permanently parked delivery defers reaping indefinitely — accepted, not fixed, by this PR.if (agent.IsReapClaimed) return;) sits immediately before the actual write, because the unfenced TTL claim's wait is deliberately shorter than a delivery's worst-case in-section time (borrowed-snapshot refresh can take up to 30s) — without the re-read, a perfectly healthy but slow delivery could complete into an agent already condemned by a claim that landed mid-flight, writing input and emitting an activity report for a stop that's already happening.GracefulExitWait, 15s) around the send half, not just the exit-wait half — this applies to every stop, reap or otherwise, not just this PR's new path. It closes a pre-existing (not new) hazard: a non-draining child parks the graceful/exitsend on the same uncancellablewrite(2)a parked delivery would, makingTerminateAsync— the step that actually ends the wedge — unreachable exactly when it's needed most. Per-runtime check before widening the bound: ACP sends onesession/cancelnotify, Antigravity's graceful stop returnsTask.CompletedTaskimmediately, and Pi's own stop grace is 3s — 15s truncates no legitimate graceful path on any of the four vendors; only a wedged PTY's rawwrite(2)was ever at risk.Selection now captures a
ReapCandidate(agent instance, reason, activity generation,FencedOnActivity) with the generation read before the age/idle/turn reads — the ordering is load-bearing (reasoned and documented in code; not independently testable without a clock-internal seam).FencedOnActivitydistinguishes the two rules explicitly rather than by string-matching the reason later: idle/wedge reaps abort on an activity advance, the 6h absolute-lifetime cap never does (by design — "absolute" means absolute) but still takes the section so incarnation/membership are revalidated.Scope: idle/wedge reaping only. User- and server-commanded stops are untouched —
StopAgentCoreAsync/HandleStopAgentare the same shared executor for every caller, unmodified except for the graceful-stop bound, which is deliberately global (see above).Test evidence
test/Capacitor.Cli.Tests.Unit, 7066 tests): 6981 passed, 50 failed, 35 skipped (gated live-vendor certification tests). Every one of the 50 failures was individually investigated and traced to one of two causes, neither caused by this branch:CodexConfigTomlTests,CodexConfigWriterTests,CodexLauncherTests,CodexProjectKeyTests,PluginCommandCodexInstallIntegrationTests,UninstallCommandTests) reproduce identically (same names, same counts) on a byte-for-byte pristineorigin/maincheckout in an isolated worktree — a pre-existing, machine-local environment issue (Codexconfig.tomltemp-path handling on this host), wholly unrelated to this diff, which touches no Codex/uninstall code.BorrowedReviewAuthBrokerTests×3,GitProviderRouterTests×1, and 2 in the touchedAgentOrchestratorVendorTestspartial class —PTY_output_chunk_advances_the_agents_activity_clock, part of the documented closed set of ~5 PTY/consent-dialog tests, andShutdown_with_live_children_and_queued_stops_discards_the_queue_and_kills_every_child, a real-child-process dispose test whose own file this branch never touches) are machine-load timing flakes: all pass cleanly and quickly (sub-second to ~1s) when re-run individually, including the shutdown test, which missed its 30s bound by ~1s under load (31.1s) and completed in 1.17s alone.ReviewerReapingTests.cs): six claim tests plus a bound-relation pin, written across three TDD rounds — the initial three (stale-selection-aborts-after-delivery, contention-has-exactly-one-winner, max-lifetime-reaps-regardless-of-delivery), the bounded-wait round's parked-delivery-does-not-defer-the-absolute-lifetime-reap, and the graceful-stop round's parked-exit-write-does-not-stall-the-reap's-terminate and healthy-in-section-delivery-refuses-after-a-claim-lands-mid-flight — plus the reap-claim-gate-wait-stays-under-the-heartbeat-interval relation pin (not itself a claim test).scripts/check-linear-ids.sh— clean (exit 0) against the full branch diff.Known residuals
IsReapClaimedread positioned after the write but beforeAdvance(), which was not built.RequestStatusReport2/EchoNonce, AI-1787) remains unbuilt. It did not exist in the daemon at this branch's base commit.The_report_shape_carries_no_correlation_nonceis a deliberate tripwire test — reflection overDaemonStatusReport/LiveAgentInfoasserting no*nonce*/*echo*member exists — so that whoever lands the correlated handler inherits a red test by design, forcing them to route it through the same ordering section rather than silently bypassing it.SendDaemonStatusReportOnceAsyncis the one production call site ofServerConnection.DaemonStatusReportAsync, and the gate's field doc states that every future emission site must go through it — but nothing stops a future call site from invokingDaemonStatusReportAsyncdirectly, compiling cleanly and silently reintroducing the server'sRegressedlatch. No architectural enforcement (visibility narrowing, a call-site-count test) was added; flagged as a candidate follow-up.🤖 Generated with Claude Code
Do NOT merge.