Skip to content

feat(analyze): the enhancement non-verdicts are visible, out of the prompt, counted, and migrated on re-enhancement (#611) - #635

Open
gadievron wants to merge 2 commits into
masterfrom
fix/issue611-sentinel-handling
Open

gadievron wants to merge 2 commits into
masterfrom
fix/issue611-sentinel-handling

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

Fixes #611.

What was wrong

Enhancement non-verdicts (45 agent-loop incompletes + 12 raise-class errors of 1,730 units in the referenced run) leaked into analysis, disappeared under filtering, and survived re-enhancement in analyze checkpoints — four gaps sharing one missing contract: the enhance sentinel values are not verdicts, and nothing treated them accordingly.

The fix (four sub-fixes, one sentinel contract)

The shared constant: SENTINEL_CLASSIFICATIONS = {incomplete, error, unknown} in core/verdict_taxonomy (the vocabulary's home — a circular import found in-flight moved it out of analysis_core; both consumers import directly).

  1. The shape: EnhanceResult carries incomplete_count + total_units — the three-bucket identity (total = units_enhanced + incomplete + errors) is recoverable without summing classifications (which omits errors by construction); the step-report summary threads them.
  2. The hint: the sentinel set omits the Stage-1 "Pre-analysis hint" line entirely — a non-verdict presented to the model as a classification is the wrong channel; the durable signal is the per-row stamp.
  3. The census + counter: a sentinel-stamped unit is not classified for the --exploitable warning's census (an all-sentinel dataset fires the loud warning — the same un-enhanced shape); the metrics publish enhance_unclassified_analyzed from the per-row stamp — not a third analyze state (Progress/summary counters use a two-branch error vs else split over three states, so incomplete work is counted as completed (5 sites) #293 stands: an input-provenance counter, placed after the consistency check's counts rebuild so it survives correction-runs). The filter's own retention policy is deliberately unchanged (the written contract; the drop is now visible).
  4. The recovery blocker: the two-sided _cp_is_stale predicate — a sentinel-stamped analyze checkpoint re-analyzes when re-enhancement changes its classification to completed; a still-sentinel checkpoint stays adopted (matching the run's own dataset). _seed_summary excludes the stale row from completed (the migration-resume double-count, completed > total) while its usage still accumulates (the spend happened — the same shape as the errored-row gate).

Evidence

  • RED (pristine): the EnhanceResult lacked both slots; the sentinel hint rendered Pre-analysis hint: classified as "incomplete"; the census counted sentinels as classified; a sentinel-stamped checkpoint was adopted unconditionally.
  • GREEN: 11/11 on the new file (the identity; the hint-omission with the clean-classification control; the census; the counter pin; the four-cell behavioral predicate test; the behavioral seed test — the stale row excluded from completed, its usage retained; the no-units legacy shape); the full suite 4160 passed + 33 skipped; ruff clean; Semgrep 1 pre-existing.
  • The hunt loop (two rounds, all findings fixed): round 1 caught the seed double-count (the HIGH — a stale row seeded as completed would make completed > total on the migration resume), the counter vanishing on consistency-correction runs (moved after the rebuild), the single-shot stamp never flowing (the counter's scope documented honestly — agentic-only today), and the tautology tests (the predicate extracted + the four-cell behavioral test; the seed's behavioral receipt). Round 2 (the exhaustion) caught the stale row's usage vanishing (the gate split: completed excluded, usage accumulated) — re-verified clean after the fix.

Notes for the reviewer

  • The counter's scope is agentic-only today — the single-shot row carries no security_classification stamp (the mode-agnostic census uses _unit_security_classification, but the results row is stamped from agent_context only). The helper documents this; extending the stamp to llm_context is a separate change.
  • The --exploitable filter still drops sentinel units — the issue's "retain" ask is a policy reversal against the filter's written contract; this PR makes the drop visible (the warning + the counter) and leaves the retention question to a deliberate maintainer decision.
  • The adoption/seed id-key divergence (unit.get("id", f"unit_{i}") vs {u.get("id")}) is pre-existing (analyze resume narration counts error checkpoint rows as 'already done': prints '0 units to process' while one runs, progress counter reaches 3/2 #435's lane), not introduced here.

Related: #609 (the enhance metadata-loss family), #293 (the three-bucket ruling this preserves), #311 (the summary-derivation precedent).

…rompt, counted, and migrated on re-enhancement (#611)

Enhancement non-verdicts (the sentinel set incomplete/error/unknown)
leaked into analysis, disappeared under filtering, and survived
re-enhancement in analyze checkpoints. Four fixes, one sentinel
contract (SENTINEL_CLASSIFICATIONS in core/verdict_taxonomy — the
vocabulary's home; a circular-import found in-flight moved it out of
analysis_core):

1. THE SHAPE: EnhanceResult carries incomplete_count + total_units
   (the three-bucket identity total = units_enhanced + incomplete +
   errors, recoverable without summing classifications — which omits
   errors by construction); the step-report summary threads them.
2. THE HINT: the sentinel set omits the Stage-1 'Pre-analysis hint'
   line ENTIRELY — a non-verdict presented to the model as a
   classification is the wrong channel; the durable signal is the
   per-row stamp.
3. THE CENSUS + COUNTER: a sentinel-stamped unit is NOT classified
   for the --exploitable warning's census (an all-sentinel dataset
   fires the loud warning); the metrics publish
   enhance_unclassified_analyzed from the per-row stamp — NOT a
   third analyze state (#293 stands: an input-provenance counter,
   placed after the consistency check's counts rebuild). The
   --exploitable filter's own retention policy is deliberately
   unchanged (the written contract; the drop is now VISIBLE).
4. THE RECOVERY BLOCKER: the two-sided _cp_is_stale predicate — a
   sentinel-stamped analyze checkpoint re-analyzes when
   re-enhancement CHANGES its classification to completed; a
   still-sentinel checkpoint stays adopted (matching the dataset).
   _seed_summary excludes the stale row from completed (the
   migration-resume double-count) while its usage still accumulates
   (the spend happened — the same shape as the errored-row gate).

serves: I-ana, I-enh, I-out. de-balances-checked: the #293
three-bucket ruling stands; the counter's agentic-only scope
documented (the single-shot row carries no stamp today).

Refs #611
Comment thread libs/openant-core/tests/test_issue611_sentinel_handling.py Fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhancement non-verdicts leak into analysis hints, disappear under --exploitable filtering, and survive re-enhancement in analyze checkpoints

1 participant