Conversation
…rompt, counted, and migrated on re-enhancement (#611) Enhancement non-verdicts (the sentinel set incomplete/error/unknown) leaked into analysis, disappeared under filtering, and survived re-enhancement in analyze checkpoints. Four fixes, one sentinel contract (SENTINEL_CLASSIFICATIONS in core/verdict_taxonomy — the vocabulary's home; a circular-import found in-flight moved it out of analysis_core): 1. THE SHAPE: EnhanceResult carries incomplete_count + total_units (the three-bucket identity total = units_enhanced + incomplete + errors, recoverable without summing classifications — which omits errors by construction); the step-report summary threads them. 2. THE HINT: the sentinel set omits the Stage-1 'Pre-analysis hint' line ENTIRELY — a non-verdict presented to the model as a classification is the wrong channel; the durable signal is the per-row stamp. 3. THE CENSUS + COUNTER: a sentinel-stamped unit is NOT classified for the --exploitable warning's census (an all-sentinel dataset fires the loud warning); the metrics publish enhance_unclassified_analyzed from the per-row stamp — NOT a third analyze state (#293 stands: an input-provenance counter, placed after the consistency check's counts rebuild). The --exploitable filter's own retention policy is deliberately unchanged (the written contract; the drop is now VISIBLE). 4. THE RECOVERY BLOCKER: the two-sided _cp_is_stale predicate — a sentinel-stamped analyze checkpoint re-analyzes when re-enhancement CHANGES its classification to completed; a still-sentinel checkpoint stays adopted (matching the dataset). _seed_summary excludes the stale row from completed (the migration-resume double-count) while its usage still accumulates (the spend happened — the same shape as the errored-row gate). serves: I-ana, I-enh, I-out. de-balances-checked: the #293 three-bucket ruling stands; the counter's agentic-only scope documented (the single-shot row carries no stamp today). Refs #611
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
September 14, 2026 21:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #611.
What was wrong
Enhancement non-verdicts (45 agent-loop incompletes + 12 raise-class errors of 1,730 units in the referenced run) leaked into analysis, disappeared under filtering, and survived re-enhancement in analyze checkpoints — four gaps sharing one missing contract: the enhance sentinel values are not verdicts, and nothing treated them accordingly.
The fix (four sub-fixes, one sentinel contract)
The shared constant:
SENTINEL_CLASSIFICATIONS = {incomplete, error, unknown}incore/verdict_taxonomy(the vocabulary's home — a circular import found in-flight moved it out ofanalysis_core; both consumers import directly).EnhanceResultcarriesincomplete_count+total_units— the three-bucket identity (total = units_enhanced + incomplete + errors) is recoverable without summingclassifications(which omits errors by construction); the step-report summary threads them.--exploitablewarning's census (an all-sentinel dataset fires the loud warning — the same un-enhanced shape); the metrics publishenhance_unclassified_analyzedfrom the per-row stamp — not a third analyze state (Progress/summary counters use a two-brancherrorvselsesplit over three states, so incomplete work is counted as completed (5 sites) #293 stands: an input-provenance counter, placed after the consistency check's counts rebuild so it survives correction-runs). The filter's own retention policy is deliberately unchanged (the written contract; the drop is now visible)._cp_is_stalepredicate — a sentinel-stamped analyze checkpoint re-analyzes when re-enhancement changes its classification to completed; a still-sentinel checkpoint stays adopted (matching the run's own dataset)._seed_summaryexcludes the stale row fromcompleted(the migration-resume double-count,completed > total) while its usage still accumulates (the spend happened — the same shape as the errored-row gate).Evidence
EnhanceResultlacked both slots; the sentinel hint renderedPre-analysis hint: classified as "incomplete"; the census counted sentinels as classified; a sentinel-stamped checkpoint was adopted unconditionally.completed > totalon the migration resume), the counter vanishing on consistency-correction runs (moved after the rebuild), the single-shot stamp never flowing (the counter's scope documented honestly — agentic-only today), and the tautology tests (the predicate extracted + the four-cell behavioral test; the seed's behavioral receipt). Round 2 (the exhaustion) caught the stale row's usage vanishing (the gate split:completedexcluded, usage accumulated) — re-verified clean after the fix.Notes for the reviewer
security_classificationstamp (the mode-agnostic census uses_unit_security_classification, but the results row is stamped fromagent_contextonly). The helper documents this; extending the stamp tollm_contextis a separate change.--exploitablefilter still drops sentinel units — the issue's "retain" ask is a policy reversal against the filter's written contract; this PR makes the drop visible (the warning + the counter) and leaves the retention question to a deliberate maintainer decision.unit.get("id", f"unit_{i}")vs{u.get("id")}) is pre-existing (analyze resume narration counts error checkpoint rows as 'already done': prints '0 units to process' while one runs, progress counter reaches 3/2 #435's lane), not introduced here.Related: #609 (the enhance metadata-loss family), #293 (the three-bucket ruling this preserves), #311 (the summary-derivation precedent).