Rerank provenance: preserve every candidate and record explicit rejection reasons

[PhysShell]

Classification

• Change type: correctness fix + provenance improvement
• Priority: P0
• Expected value: high — prevents silent candidate loss and protects S8/S9 lineage semantics
• Estimated effort: low-medium
• Implementation risk: low
• Recommendation: YES, immediate improvement; small scope with disproportionate correctness value

Problem

rerank_candidates() currently scores candidates through filter_map. When candidate_axes() fails, the candidate is silently removed from the returned ranking.

That creates several bad states:

• input and output cardinality no longer match;
• the caller cannot distinguish measurement failure from policy rejection;
• S8 cannot explain why a candidate disappeared;
• S9 may treat absence as if the user rejected the candidate;
• future Evolution Lab lineage loses children without a recorded cause;
• aggregate metrics can be biased because unmeasurable candidates vanish from the denominator.

A candidate should not disappear like a witness after an unusually persuasive legal consultation.

Goal

Preserve one explicit evaluation outcome for every input candidate.

Replace silent filtering with a typed result model such as:

pub struct CandidateEvaluation {
    pub candidate: SetCandidate,
    pub outcome: CandidateOutcome,
}

pub enum CandidateOutcome {
    Ranked {
        axes: Axes,
        aggregate: f64,
        rank: usize,
    },
    Rejected {
        reason: CandidateRejection,
    },
}

pub enum CandidateRejection {
    ClosureMeasurement(ClosureIssue),
    NoveltyMeasurement(NoveltyIssue),
    InvalidScore(ScoreInvariant),
}

Exact type names are reviewable. The invariant is not.

Required semantics

1. Every input candidate yields exactly one CandidateEvaluation.
2. Ranked candidates receive a deterministic rank.
3. Rejected candidates retain strategy, seed, gesture and score provenance.
4. Rejection reason is typed and inspectable; no string-only error bucket.
5. Ranking order is computed only among ranked candidates, while the full evaluated population remains available.
6. Rejection is not confused with human dislike/reject feedback.
7. Metrics report both ranked and rejected counts.
8. Existing deterministic tie-breaking remains unchanged for successfully ranked candidates.

API compatibility

A staged approach is acceptable:

new canonical API: evaluate_and_rerank_candidates(...)
compat helper: rerank_candidates(...) -> ranked-only projection

The compatibility helper must be documented as a projection that discards explicit rejection records, not remain the only product surface.

S8/S9-facing code should consume the canonical full evaluation result.

Required controls

1. N valid candidates → N evaluations, all ranked.
2. One candidate fails closure measurement → N evaluations, one typed rejection.
3. One candidate fails novelty measurement → distinct typed rejection.
4. Ranked subset order matches the current implementation for all-success fixtures.
5. Rejected candidate retains exact strategy/seed/gesture provenance.
6. Re-running fixed inputs is byte-stable after normalization.
7. UI/provenance projection shows rejection reason without recomputing scoring.
8. Population metrics include rejected candidates and rejection-rate breakdown.

Acceptance

• Canonical rerank API returns one explicit outcome per input candidate.
• Silent filter_map disappearance is removed from the canonical path.
• Typed rejection reasons cover current closure/novelty failures.
• Successful ranking behavior and deterministic tie-breaking remain unchanged.
• Headless tests verify cardinality preservation and provenance retention.
• S8/S9 integration notes identify absence, scoring rejection and human feedback as separate states.

Non-goals

• No new scoring axes or tuned weights.
• No automatic retry/regeneration of rejected candidates.
• No S9 preference-learning implementation.
• No requirement to display rejected candidates in every UI immediately.
• No broad refactor of Score validation.