Skip to content

feat(enhance): the degenerate exits get their own diagnostic channel — exit_kind + the incomplete_summary histogram (#615) - #636

Open
gadievron wants to merge 1 commit into
masterfrom
fix/issue615-exit-kind-diagnostics
Open

gadievron wants to merge 1 commit into
masterfrom
fix/issue615-exit-kind-diagnostics

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

Fixes #615.

What was wrong

The agentic enhance loop's four degenerate exits are conflated under one classification: "incomplete", but they differ in kind and remedy: three cheap model-behavior exits (end-turn-without-finish, finish truncated at max_tokens, no-tool-calls — confidence 0.3, remedied by retry/prompt work) vs one budget-exhaustion exit (MAX_ITERATIONS — confidence 0.2, remedied by raising the budget, not retrying). In the referenced run the split was 41 "Analysis incomplete" + 4 "Could not complete analysis within iteration limit" — a 4-unit class a consumer reading only classification cannot distinguish from the 41.

The fix

No fourth classification (that would break every == "incomplete" consumer — checkpoint.py, enhancer.py, the degenerate-exit tests). Instead:

  • exit_kind on AgentResult — the four exit sites stamp their kinds (end_turn_without_finish / finish_truncated / no_tool_calls / max_iterations); a completed analysis carries the empty string and serializes without the key (present-only — the completed-record byte-identity).
  • The incomplete_summary histogram beside the existing error_summary: the enhancer's counting loop aggregates exit_kind (legacy unstamped rows fall back to "unstamped"); EnhanceResult carries the histogram (present-only in to_dict); and both step-report writers — the scanner's pipeline summary and the standalone CLI's enhance summary — thread it, so the 4-vs-41 split is visible in the artifacts without stderr mining.
  • The classification and every existing consumer are unchanged (pinned by the classification-unchanged test).

Evidence

  • RED (pristine): AgentResult had no field; the four exits stamped nothing; no histogram anywhere.
  • GREEN: 8/8 on the new file — the four kinds + the completed negative (present-only), an incomplete serializing its kind, the classification-unchanged pin, the histogram census (max_iterations 2 / end_turn 1 / unstamped 1), the EnhanceResult carry + the present-only negative, both summary writers' pins, and two real-exit receipts driving the actual agent loop through the existing degenerate-exit harness (the end-turn-without-finish and the R2-B finish-truncated stamps on real AgentResults, not hand-constructed ones); the full suite 4171 passed + 33 skipped; ruff clean; Semgrep 0.
  • The hunt (one round, all findings fixed): the CLI's enhance summary was a second writer the first pass missed (threaded); the stamp sites were untested against the real loop (the two real-exit receipts added); the typing convention + a dead splat (cleaned). The #635 merge shape named (four adjacent-insert hunks; the post-merge invariant sum(incomplete_summary.values()) == incomplete_count recorded for the reconciler).

Notes for the reviewer

Related: #611 (the sentinel-handling round that surfaced this), #609 (the same stage's metadata-loss fix).

…— exit_kind + the incomplete_summary histogram (#615)

The four degenerate exits are conflated under one classification
'incomplete', but they differ in kind and remedy: three cheap
model-behavior exits (end-turn-without-finish, finish truncated at
max_tokens, no-tool-calls — confidence 0.3, remedied by retry/prompt
work) vs ONE budget-exhaustion exit (MAX_ITERATIONS — confidence 0.2,
remedied by raising the budget). A consumer reading only
classification cannot split the 4-vs-41.

NO fourth classification (that would break every =='incomplete'
consumer): an exit_kind field on AgentResult (present-only in
to_dict; empty = a completed analysis; the four sites stamp their
kinds), and the incomplete_summary kind histogram beside the existing
error_summary — EnhanceResult carries it, and BOTH step-report
writers (the scanner and the standalone CLI) thread it, so the split
is visible without stderr mining. Legacy unstamped rows fall back to
'unstamped' in the aggregate; the per-record to_dict stays present-
only. The classification and its consumers are UNCHANGED (pinned).

serves: I-enh, I-out. de-balances-checked: every ==incomplete
consumer keeps its semantics (the classification-unchanged test);
the completed analysis serializes byte-identically (present-only).

Refs #615
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant