fix(cli): honest exit codes — stalled runs fail, --cwd validated, lookup misses exit 1 - #145
Conversation
|
The direction is right: a stalled run and a bad 1. No tests for the new contract. A PR titled "honest exit codes" changes the return code in three places ( 2. This is a breaking change and is not flagged as one. A one-shot 3. The unification is half done. The PR justifies The changes themselves are small and correct once tests and the breaking-change note land. |
|
All three addressed, and point 3 turned out to run deeper than the 3 first, since it drove the rest. You were right that A bug the audit surfaced, worth its own mention. The While there I also fixed 1. Tests. Every changed code is pinned, and each was verified to fail without its fix by
The stalled test is not a mock: 2. Breaking change. Flagged in the description with a full before/after table (17 rows). Verification. Left alone and listed under "Not done here": the six legacy commands, |
…kup misses exit 1
63566bd to
ee63c7a
Compare
What
Exit codes that scripts and CI can trust. The CLI now separates usage errors from
operational failures:
run,skilland the top-level dispatcher implement this. The split is documented onCommandDescriptor.runinsrc/cli/index.ts, including the commands that do notfollow it, so a caller is not misled:
config,serve,trace,task,modelsandimportpredate the split and return1for usage errors too;trace replayreturns2for "stable-prefix drift detected" (a diff-style result code);
tuipasses a relaunchedchild's status straight through, so any code is possible;
replis a scaffold and alwaysreturns
0. Widening the split to those is a separate change.Reference point is pre-PR. Every previously-failing case was already non-zero, so scripts
testing
if ! atomic-agent …are unaffected by theskillrows; scripts comparing againsta literal
1/2need updating. The tworunrows turn previously-silent successes intofailures — that is the point of the PR.
run— turn exhaustsagent.maxSteps(sessionstalled)atomic-agent(TUI) — same stalled turnrun --cwd <nonexistent>run --cwd <file>/--working-dir <file>skill show <missing>skill uninstall <missing>skill installalready installedskill install <owner/repo>hub failureskill install @owner/slugClawHub failureskill browse/searchwhen every source failed and nothing was foundskill <unknown-subcommand>skill installwith no sourceskill show|uninstall|enable|disablewith no<name>skill browse --sourcewith no valueskill searchwith an empty queryskill tap add|removewith no repo, or a malformedowner/reposkill tap <unknown-verb>Known in-repo consumers. The eval harnesses gate on
atomic-agent runexiting 0 —eval/harness/run-case.ts:104(the actual pass/fail gate),eval/cases.eval.ts:74,eval/harness/append-jsonl-row.ts:109(failure-reason telemetry), and the threeeval-memory/experiments/*/runner.ts. A scenario that exhausts the step budget now flagsas a subprocess failure where it previously scored as a normal run. Arguably correct — a
stalled turn is degraded — but it will move eval bookkeeping.
Why
skill.tsmoved in both directionsThe original version of this PR claimed "2 means you invoked this wrong across the CLI"
while
skill.tsdid the opposite: missing-argument usage errors returned1andoperational failures returned
2. Fixing onlyskill showsplit it further —show <missing>exiting 1 whileuninstall <missing>exited 2 for the same class of error.The file is now internally consistent, and the false blanket claim is replaced by the
documented contract above.
The
stalledrule also lived in two places:run-agent.tswas updated andchat-orchestrator.ts(the TUI, i.e. bareatomic-agent) was not, so the default commandstill exited 0 on the exact condition this PR is named for. Both now read
isFailedSessionStatusfromsrc/session/session-state.ts, so they cannot drift again.cancelleddeliberately stays a success — the operator asked to stop.Testing
npm run lint(tsc) clean, verified against the branch as pushed to origin.npx vitest run src/cli src/session— 100 passed.these commits. The extra failure is
llm-health-poller, a pre-existing flake reproducedat baseline (~40%, passes 12/12 solo); no new failure.
src/cli/run-agent.test.ts(5 tests) drives the real agent loop, tool registryand status transitions to step-budget exhaustion with only the LLM stubbed — the stalled
assertion checks
"status": "stalled"in the run summary, not a mock.src/session/session-exit-status.test.tspins the shared predicate across every memberof
SessionStatus, so a future status cannot silently default to "success".re-running): stalled →
expected +0 to be 1;--cwdguard → both path tests fail;uninstall/show→expected 2 to be 1; usage codes →expected 1 to be 2;browse/search→expected 0 to be 1.Not done here
1. Documented rather than changed, to keep thebreaking surface deliberate.
skill enable <not-installed>returns 0 (disableat least warns).cancelledsession exits 0 in bothrunand the TUI. Probably intended, but aSIGINT-killed CI run exiting 0 is arguably the same class of dishonesty.