S9 Counterfactual Preference Lab: pairwise A/B feedback over single-axis candidate deltas

[PhysShell]

Classification

• Change type: enhancement / product preference-learning experiment
• Priority: P1
• Expected value: high — produces cleaner preference signals than whole-candidate likes alone
• Estimated effort: medium-high
• Implementation risk: medium — pair construction must prove that only the declared axis changed
• Recommendation: YES, strong S9 differentiator after stable candidate IDs/provenance; keep behind an experiment flag initially
• Stage ownership: S9 semantics; S8 owns the comparison surface
• Related prerequisite: Rerank provenance: preserve every candidate and record explicit rejection reasons #104

Problem

The planned S9 baseline records like/dislike/favorite feedback and updates feature weights. Whole-candidate feedback is useful, but it is heavily confounded:

Did the user prefer the rhythm?
The register?
The cadence?
The tonal hypothesis?
The gesture density?
One lucky note?

One heart icon applied to a six-axis candidate is not a clean learning signal. It is a small philosophical crisis stored as a boolean.

Griff already has the right deterministic machinery for a stronger experiment:

• fixed generation requests and seeds reproduce candidates;
• strategy × variant identity is explicit;
• seed derivation is deterministic;
• gestured/plain variants can pair seed-for-seed;
• scoring axes and provenance are inspectable.

Goal

Add a Counterfactual Preference Lab that presents pairwise A/B candidates differing on exactly one declared musical/control axis.

The user chooses:

left | right | tie | skip

The resulting feedback updates preference evidence for the changed axis without pretending that every other candidate property caused the choice.

Initial comparison axes

Start with axes that can be isolated deterministically:

register window / center
ending-stability or cadence treatment
gesture density
variation amount
complexity axis value
rhythm-template choice
tonal candidate/hypothesis (only after S15 scope/confidence policy is ready)

Each axis requires a pair constructor and an equivalence contract for everything that must remain unchanged.

Pair contract

Suggested model:

pub struct CounterfactualComparison {
    pub comparison_id: ComparisonId,
    pub left: CandidateId,
    pub right: CandidateId,
    pub changed_axis: PreferenceAxis,
    pub left_value: AxisValue,
    pub right_value: AxisValue,
    pub controlled_context_digest: Digest,
    pub provenance: ComparisonProvenance,
}

pub enum PairwiseVerdict {
    Left,
    Right,
    Tie,
    Skip,
}

Skip must remain distinct from Tie and from absent feedback.

Controlled-context requirements

For a declared single-axis pair, pin and verify all applicable controls:

base seed
strategy
variant index
source rhythm/template palette
pitch material / tonal scope
bar and meter structure
candidate-set version
generator/scoring version
all non-target control values

The pair builder must emit a machine-readable diff of expected and observed feature deltas.

If an uncontrolled axis changes beyond an explicit tolerance, the comparison is invalid and must not update the preference profile.

Learning baseline

Do not require a neural/RL system for the first slice.

Acceptable deterministic baselines include:

• per-axis signed bounded updates using the known left/right delta;
• Bradley–Terry-style pairwise scores;
• per-axis Beta/log-odds estimates for discrete choices.

Requirements:

1. preference state remains inspectable and resettable;
2. updates are deterministic for a fixed feedback sequence;
3. confidence/evidence count is separate from preference magnitude;
4. no-feedback/skip does not become negative evidence;
5. learned values never overwrite raw feedback events.

S8 surface

S8 should display:

A/B candidates
changed axis and values
what was held constant
play/listen controls
left/right/tie/skip
optional explanation after voting

The UI must consume the canonical comparison contract and must not reconstruct pair semantics from labels.

Evaluation

Measure whether pairwise evidence improves ranking with held-out synthetic and human sessions.

Initial deterministic evals:

1. Synthetic user consistently prefers higher ending stability → later ranking shifts accordingly.
2. Synthetic user prefers lower register but is indifferent to rhythm → register changes while untouched rhythm preference remains stable.
3. Tie feedback increases evidence count without directional movement.
4. Skip produces no preference update.
5. Invalid/confounded pair produces no update and a typed rejection.

Human experiment metrics:

pair completion/skip rate
per-axis evidence count
preference consistency on repeated hidden pairs
ranking lift on held-out comparisons
time per decision
axis coverage and imbalance

Required controls

1. Same request/seed/comparison spec reproduces the same semantic pair.
2. Pair differs only on the declared axis within documented tolerances.
3. Deliberate second-axis drift rejects the pair.
4. Left/right order randomization does not change the underlying comparison identity.
5. Repeated identical feedback sequence yields byte-stable profile state.
6. Tie, Skip, missing feedback and rejection remain distinct.
7. Candidate scoring rejection from Rerank provenance: preserve every candidate and record explicit rejection reasons #104 is not recorded as human feedback.
8. Raw comparison and feedback events retain candidate/session/generator provenance.
9. Reset/rebuild from raw events reproduces the profile.
10. Existing ordinary like/dislike/favorite flow remains available and semantically separate.

Acceptance

• Versioned comparison, feedback and provenance contracts are documented.
• At least two single-axis pair constructors are implemented and headless-testable.
• Pair validation proves controlled context and rejects confounded pairs.
• S8 has a minimal A/B capture surface or a headless CLI fixture surface first.
• Deterministic baseline preference updater consumes pairwise events.
• Synthetic eval demonstrates directional learning without unrelated-axis drift.
• Fixed inputs, events and seed reproduce candidates, comparison identity and profile.
• Metrics expose skip/tie/confounded-pair rates before product adoption.

Non-goals

• No gradient descent, RL, contextual bandit or model training requirement in the first slice.
• No automatic mutation/evolution loop.
• No claim that pairwise feedback replaces ordinary whole-candidate feedback.
• No S15 tonal comparison until confidence/scope semantics are accepted.
• No hidden update from rejected/unmeasurable candidates.
• No product-default rollout before experiment metrics justify it.