Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 61 additions & 19 deletions docs/HERDR.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,26 +75,53 @@ to their own direct dispatch so visibility never becomes a hard dependency.

### Sweep safety (FR-VIS-07, FR-VIS-10)

`devagent herdr-sweep` closes only panes that satisfy both per-pane guards:

1. **Automation ownership**: pane cwd sits inside `.devagent-worktrees/` —
operator scratch panes in the same session are never listed, let alone
closed.
2. **No live dispatch**: `pane process-info` shows the foreground process; a
pane running `omp`/`pi`/`claude`/`opencode` is mid-run and skipped. This
guard exists because a busy pane can still report `agent_status: idle`
(the pane wrapper polls the done-marker, not the agent state machine) —
the 2026-09-05 in-flight-close regression.
`devagent herdr-sweep` orders its checks so each pane is judged by scope (may
it be swept at all) → operator at the wheel → orphan evidence → roster spare →
status classes. The two per-pane guards:

1. **Automation ownership**: a pane whose cwd sits inside `.devagent-worktrees/`
was spawned by the dispatcher, so the status classes reach it. Anywhere else —
the main checkout, where the loop's research/PO dispatches run beside the
operator's own windows — the cwd proves nothing: the pane is swept only by the
orphan class (`--orphans`) and only on positive **dispatch evidence**, the
run's capture contract (`<tmp>/devagent-herdr-<n>/{out,err,done}`) open on
fd 1 or fd 2 of one of its foreground processes. A worker the operator ran by
hand points at the pane tty and never matches, so scratch panes are
structurally spared.
2. **No live dispatch**: `pane process-info` — asked **in the session under
sweep**; an unscoped probe answers for herdr's own default session, which made
every devagent pane read idle until 2026-09-13 — shows the foreground process,
and a pane running `omp`/`pi`/`claude`/`opencode` is mid-run and skipped. This
guard exists because a busy pane can still report `agent_status: idle` (the
pane wrapper polls the done-marker, not the agent state machine) — the
2026-09-05 in-flight-close regression.

The one exception to guard 2 is the orphan class itself: a live worker whose
collector is dead has nobody polling its done marker or closing its workspace, so
`--orphans` reaps it. Ownership is the process that survives the whole run — the
dispatching `devagent task` / `devagent pane-run` that polls the marker — **not**
the `herdr pane run` client, which types the script into the pane's shell and
exits; if that dispatcher is gone, or its `ps` ppid ancestry holds no live loop
driver, the run is an orphan. The probes must ANSWER before anything is reaped:
pgrep's own "no match" (exit 1, no output) means there is no collector and the
pane goes, while a `pgrep`/`ps` that could not run — absent binary, the 5s cap, a
rejected pattern, an ancestry walk that never completed — is no evidence and the
class goes inert, because reaping on a probe that never inspected anything would
close every live worker on a host that cannot read its own process table.
Attribution is process-wide: any dispatcher still hanging off a live driver
spares every live pane in the session, so the error direction is "leave a
leftover running", never "close a run somebody is still collecting".

The session name alone is not a safety property (PRD §18 Q23), so two
operator-side bounds sit on top of the per-pane checks:

3. **Operator-attach exemption**: a pane the FR-VIS-02 roster reports as live
(`state: running`) is never closed, and neither is anything in the session
while `DEVAGENT_OPERATOR_ATTACHED` is set in the sweep's environment — an
operator is at the wheel. Spared panes are still reported, with
`reason=operator-attached` (CLI line prefix `[spared]`), so a dry-run
explains why a pane survived. The exemption outranks the `--orphans` class.
3. **Operator-attach exemption**: nothing in the session is sweepable while
`DEVAGENT_OPERATOR_ATTACHED` is set in the sweep's environment — an operator
is at the wheel, and that outranks every other class including `--orphans`.
Below it, the FR-VIS-02 roster spares a worktree pane it reports as live
(`state: running`) when no foreground worker was found. Spared panes are still
reported, with `reason=operator-attached` (CLI line prefix `[spared]`), so a
dry-run explains why a pane survived.

4. **Managed deny toggle** (`devagent.json`):

Expand Down Expand Up @@ -123,6 +150,14 @@ anything, and states the bound when the sweep is disabled or denied.

### Orphaned worker-helper class (`--orphan-brokers`)

**Status (2026-09-13): documented, not yet ported to Go.** `devagent
herdr-sweep --orphan-brokers` is rejected as an unknown flag by the current
binary (its surface is `--session`, `--dry-run`, `--orphans`); the class shipped
in the Node implementation retired by #205 and has no Go counterpart, so nothing
below runs. Read it as the port contract, not as live behavior — including the
answered-probe rule above: a reaper must never kill on a process table it could
not inspect.

The per-pane guards above cannot see a different leak class: omp's
`__omp_worker_daemon_broker` worker outliving its session. When an omp process
dies without taking its broker down, the broker reparents to launchd (ppid 1)
Expand Down Expand Up @@ -151,6 +186,13 @@ against a functional stub CLI (`DEVAGENT_HERDR_BIN` injects the binary): stdout
capture, exit-code propagation, env injection without leakage, timeout teardown,
keep-panes mode, fallback behavior, and config validation — plus the sweep guards
above (FR-VIS-07 per-pane checks, the FR-VIS-10 deny toggle, the operator-attach
exemption, `herdr.sweep` parsing/validation/env precedence). The orphaned-broker
class is tested against a real child process that traps SIGTERM, so the
SIGTERM→SIGKILL escalation is proven rather than stubbed.
exemption, `herdr.sweep` parsing/validation/env precedence). The orphan probes
are pinned from two directions: their test seams are keyed by what the sweep
actually asks (the `pgrep` seam by the pattern, the capture seam by pane id, a
miss counting as an unanswered probe), and the shapes behind those keys are
checked against reality — the real `lsof` probe against a child holding a capture
file on fd 1/2, and the owner pattern against the command lines
`internal/loopdriver` builds — because on 2026-09-13 the stubs stayed green while
live reaping matched nothing. The orphaned-broker class above has no Go
implementation and therefore no test; porting it inherits the same rule — an
unreachable process table yields no candidates, never a kill list.
94 changes: 82 additions & 12 deletions docs/PRD.html
Original file line number Diff line number Diff line change
Expand Up @@ -2771,14 +2771,15 @@ <h2 id="18-open-questions">18. Open Questions</h2>
toggle, or keep <code>--all</code> out of scope permanently?</del>
Resolved 2026-09-07: both halves of the safety requirement, and
<code>--all</code> stays out of scope permanently — the sweep lists
exactly one session and nothing widens it. Per-pane verification shipped
exactly one session and nothing widens it. Per-pane verification ships
as FR-VIS-07; the managed-settings-style toggle ships as FR-VIS-10:
<code>herdr.sweep</code> (<code>src/config.ts:39</code>, resolved by
<code>herdrSweepConfig</code> at <code>src/config.ts:427</code>) carries
<code>enabled</code> (env `DEVAGENT_HERDR_SWEEP=0</td>
<td>1<code>) and </code>denySessions<code>, and </code>findStalePanes<code>/</code>sweepStalePanes<code> (</code>src/integrations/herdr.ts:462<code>, </code>:699<code>) return nothing when the sweep is disabled or the resolved session is denied. A pane the FR-VIS-02 roster reports as live, or any sweep launched while </code>DEVAGENT_OPERATOR_ATTACHED<code>is set, is spared ahead of every other class — including</code>--orphans<code>— and reported with</code>reason:
operator-attached<code>instead of closed. Unset defaults are today's behavior; an invalid</code>herdr.sweep`
block fails closed. Removed.</td>
operator-attached<code>instead of closed. Unset defaults are today's behavior; an invalid</code>herdr.sweep<code>block fails closed. **Corrected 2026-09-13:** only the env half of that ordering survives — the roster's</code>running<code>state is derived from the same foreground-worker probe the orphan class reads, so sparing it ahead of</code>--orphans<code>left the class unreachable; a live-worker pane is now orphan-reaped on collector evidence (FR-VIS-07),</code>DEVAGENT_OPERATOR_ATTACHED`
still outranks everything, and the roster spare still covers worktree
panes that are busy with no live worker. Removed.</td>
<td>product</td>
</tr>
<tr>
Expand Down Expand Up @@ -4170,7 +4171,13 @@ <h3 id="208-visible-worker-sessions-and-terminal-tui-fr-vis-fr-tui">20.8
an idle/unknown row whose <code>pane process-info</code> foreground
process is a worker binary maps to <code>running</code>, agent_status
stays the signal for interactive panes
(<code>internal/herdr/roster.go</code> <code>paneState</code>)</td>
(<code>internal/herdr/roster.go</code> <code>paneState</code>).
<strong>Fixed 2026-09-13:</strong> that discriminator was inert in
production — the <code>pane process-info</code> probe went out
<strong>without <code>--session</code></strong>, so herdr answered for
its own default session and every devagent pane reported no foreground
process; <code>PaneForegroundWorker</code> now takes the session and
<code>paneState</code> passes it, so the upgrade actually fires</td>
<td>M</td>
</tr>
<tr>
Expand Down Expand Up @@ -4233,12 +4240,25 @@ <h3 id="208-visible-worker-sessions-and-terminal-tui-fr-vis-fr-tui">20.8
</tr>
<tr>
<td>FR-VIS-07</td>
<td>Sweep safety: <code>herdr-sweep</code> closes only panes whose cwd
sits inside <code>.devagent-worktrees/</code> (automation-owned;
operator scratch panes in the same session are untouchable) AND whose
foreground process is not a worker CLI (<code>pane process-info</code>
distinguishes a live omp/pi/claude/opencode from an idle shell) — an
in-flight pane is never sweepable</td>
<td>Sweep safety: <code>herdr-sweep</code> closes a pane only where it
can prove automation owns it. Inside <code>.devagent-worktrees/</code>
the cwd <strong>is</strong> that proof (the dispatcher cd's the run into
the task's worktree), and the pane must additionally not be running a
worker CLI — <code>pane process-info</code>, asked in the session under
sweep, distinguishes a live omp/pi/claude/opencode from an idle shell
(the 2026-09-05 in-flight-close regression). Outside the worktrees the
cwd proves nothing, so a pane joins the sweep only when the orphan class
is armed (<code>--orphans</code> / <code>herdr.sweep.orphans</code>)
<strong>and</strong> carries positive dispatch evidence — the run's
capture contract
(<code>&lt;tmp&gt;/devagent-herdr-&lt;n&gt;/{out,err,done}</code>) open
on fd 1/2 of one of its foreground processes; an operator's hand-run
worker never carries it. An in-flight pane is closed only by that orphan
class, and only when its collector is dead: the dispatching
<code>devagent task</code> / <code>devagent pane-run</code> (the process
that polls the done marker) is missing or detached from any live loop
driver. <strong>Repaired 2026-09-13</strong> — both probes were dead and
the class unreachable; see the 2026-09-13 footer entry</td>
<td>M</td>
</tr>
<tr>
Expand Down Expand Up @@ -4925,8 +4945,58 @@ <h3 id="231-requirements">23.1 Requirements</h3>
chaos schedule) so "the driver works perfectly" is a checkable claim,
not a hope.</p>
<hr />
<p>*Last updated: 2026-09-12 (the rescue reaches CLOSED-but-unmerged
pull requests: a goal-named closed PR is reopened and merged, or lands
<p>*Last updated: 2026-09-13 (herdr orphan-pane reaping: both probes
were dead, repaired as one change) —
<code>devagent herdr-sweep --orphans</code> had never reaped a live
orphan: <code>pane process-info</code> went out without
<code>--session</code>, so herdr answered for its own default session
and "no such pane" was every devagent pane's liveness, which left the
FR-VIS-07 in-flight guard, the FR-VIS-02 roster upgrade (#317) and the
orphan class — gated on a live foreground worker — inert at once; the
owner probe was <code>herdr.*pane run .*&lt;pane_id&gt;</code>, a
process that does not exist while a pane runs (<code>pane run</code>
types the script into the pane's shell and exits; verified live parent
chain
<code>omp -&gt; -zsh -&gt; herdr --session &lt;s&gt; server -&gt; launchd</code>),
so <code>PaneRunOwnerOrphaned</code> always took its "no owner CLI -&gt;
orphaned" exit and the driver spare behind it was unreachable. Each half
alone is a regression — a working owner probe over an inert liveness
probe closes live workers — so the fix lands atomically:
<code>PaneForegroundWorker(cli, session, paneID)</code> scopes the probe
(roster and sweep share it), the owner is now the surviving dispatcher
(argv0-anchored <code>devagent</code>/<code>devagent-go</code>
<code>task</code>/<code>pane-run</code>, optionally behind the driver's
<code>timeout</code> wrapper, so a prompt quoting devagent cannot
impersonate its own collector), non-worktree panes need positive
per-pane dispatch evidence (the capture contract on the worker's fd
1/2), and the orphan class moved ahead of the roster spare. Reaping also
fails closed: <code>psPidsMatching</code> and
<code>pidAncestryCommands</code> now report whether the probe answered
at all — pgrep's exit-1 "no match" is an answer (no collector, reap), an
absent binary / the 5s cap / a rejected pattern / an unfinished ancestry
walk is not (spare) — because a reaper that read "could not inspect" as
"no collector" would close every live pane on such a host. Test seams:
<code>DEVAGENT_SWEEP_OWNER_PIDS_JSON</code> is keyed by the pattern
asked for and a miss now counts as an unanswered probe (never a clean
no-match), <code>DEVAGENT_SWEEP_PANE_CAPTURE_JSON</code> supplies
pane-keyed evidence — and because the 2026-09-13 bug survived a green
stub suite, both shapes are pinned from outside:
<code>TestProcFdsCaptureSeesRealCaptureContract</code> runs the real
<code>lsof</code> probe against a live child holding
<code>&lt;tmp&gt;/devagent-herdr-&lt;n&gt;/out</code> on fd 1/2 (skipped
where lsof is absent),
<code>TestPaneRunOwnerPatternMatchesRealDispatchShapes</code> checks
<code>paneRunOwnerPattern</code> against the command lines
<code>internal/loopdriver</code> really builds and against the transient
<code>herdr pane run</code> shape it used to search for, and
<code>TestOrphanClassGoesInertWhenProbesCannotAnswer</code> pins the
spare-on-no-answer direction at the matrix level.
<code>TestPsPidsMatchingAnswersOnlyWhenPgrepAnswers</code> pins the same
split against real <code>pgrep</code> exit codes: 1 means no collector
(reap), 2 means the probe could not run (spare). Docs corrected:
FR-VIS-02/07/10, §18 Q23, <code>docs/HERDR.md</code> sweep safety. *Last
updated: 2026-09-12 (the rescue reaches CLOSED-but-unmerged pull
requests: a goal-named closed PR is reopened and merged, or lands
nothing) — the driver-side verify-and-merge rescue, and the goal-text
derivation feeding it, only accepted subjects that read
<code>OPEN</code> before the dispatch, so the false closes were
Expand Down
Loading
Loading