diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 9c9917c..0feaf55 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -36,6 +36,11 @@ live contract. - [`hard-constraint-contract.md`](hard-constraint-contract.md) — one typed rule layer separating legality from preference, plus MiniZinc as a strictly offline oracle (Constraint Lab). Status: for discussion. +- [`deterministic-search-and-llm-boundary.md`](deterministic-search-and-llm-boundary.md) + — one-best DP as the baseline; deterministic k-best and measurable diversity + before A*; `ArrangementIntent` as the only optional LLM-facing generation + contract; explicit provenance, oracle, benchmark, and kill criteria. Status: + for discussion. - [`preference-and-similarity-learning.md`](preference-and-similarity-learning.md) — the staged, benchmark-gated order in which ML enters Griff: preference evidence, retrieval-only embeddings, a human similarity benchmark, then a diff --git a/docs/proposals/deterministic-search-and-llm-boundary.md b/docs/proposals/deterministic-search-and-llm-boundary.md new file mode 100644 index 0000000..fc329e1 --- /dev/null +++ b/docs/proposals/deterministic-search-and-llm-boundary.md @@ -0,0 +1,366 @@ +# Proposal: Deterministic search, diversity, and the LLM boundary + +Status: proposal for discussion +Scope: docs-only; binds nothing until accepted + +## 1. Goal + +Make Griff less dependent on opaque model behaviour without reducing the +musical search space to one sterile optimum. + +The proposed boundary is: + +```text +optional LLM or human authoring + | + v +ArrangementIntent (typed, versioned, validated) + | + v +candidate construction + hard-rule admission + | + v +exact or explicitly approximate search + | + v +ranked, explainable, reproducible alternatives +``` + +The LLM may describe an arrangement role and preferences. It does not emit the +canonical score, assign strings and frets, bypass hard rules, or decide that its +own result is valid. + +This proposal extends the direction already fixed by ADR-0013 and ADR-0030: +DP/Viterbi remains the primary route over an enumerated layered graph. It does +not replace the existing Constraint Lab, hard-constraint proposal, scoring +contract, or deterministic generators. + +## 2. Existing ground truth + +This proposal starts from what Griff already has, rather than presenting old +work in a new hat. + +- `core/src/layered_path.rs` provides domain-free layered DP with deterministic + tie-breaking and a normative floating-point association (ADR-0030). +- ADR-0017 defines axes, versioned weights, rationale, derived aggregate, and + provenance; hard gates remain separate from soft ranking. +- `docs/proposals/hard-constraint-contract.md` defines typed admissibility and + keeps external solvers outside the production path. +- `lab/` already implements the first MiniZinc oracle spike with a solver-neutral + IR, an exact reference solver, archived solver identities, and five agreeing + fixtures. The next search work must consume that evidence instead of proposing + the same spike again. + +## 3. Decision under discussion + +### 3.1 Keep one-best DP as the normative baseline + +For an already enumerated layered graph, exact DP/Viterbi remains the default. +It visits the supplied transitions directly, requires no heuristic, and returns +one globally optimal path under the selected `WeightPolicy`. + +The one-best result is the baseline against which every later search mode is +characterized. A new mode may produce more alternatives or trade optimality for +latency, but it may not silently change what `best` means. + +### 3.2 Add deterministic k-best traversal before A* + +The next search capability should return the first `k` distinct complete paths +in total order: + +```text +primary: derived path cost +secondary: canonical path identity +tertiary: per-layer candidate ordinal sequence +``` + +The exact algorithm is deferred to an ADR and benchmark. Candidate families +include k-best Viterbi, lazy k-shortest-path enumeration, and deviation-based +methods. The implementation must reuse the same local and transition-cost +contract as one-best DP; a second cost implementation is forbidden. + +Required properties: + +- `k = 1` is byte-for-byte equivalent to the normative one-best result; +- output order is total and stable; +- duplicate canonical paths are impossible; +- each path carries the same `Scored` axes, rationale, weights version, and + generation provenance as an ordinary result; +- requesting more results never changes the prefix already returned; +- finite budgets yield a typed incomplete result, never a falsely complete set. + +This creates reproducible alternatives without treating repeated RNG runs as a +search algorithm. + +### 3.3 Diversity is a policy over valid alternatives + +`DiversityPolicy` is separate from score weights and hard rules. It selects a +useful subset from an ordered valid candidate stream; it does not redefine +legality and does not mutate the underlying score. + +Candidate measurable terms: + +```text +minimum canonical path distance +maximum shared-candidate ratio +maximum identical-technique run +motif reuse budget +rhythm-cell distance +register-region distance +fretboard-route distance +``` + +Every term must name a deterministic measurement over canonical material. +Words such as `fresh`, `creative`, and `swancore enough` are presets or human +judgements, not diversity metrics. + +Two modes are worth distinguishing: + +1. **Filter mode** — walk k-best order and admit the next path satisfying the + policy relative to those already selected. +2. **Penalty mode** — apply a versioned diversity objective during subsequent + enumeration. + +Filter mode should be attempted first because it preserves the original score +ordering and keeps diversity inspectable. Penalty mode requires a separate ADR +because it changes the optimization problem. + +A diversity result records: + +```text +source search contract + version +DiversityPolicy identity + version +selected path identities +pairwise measurements used for admission/refusal +search budget and completeness status +``` + +### 3.4 Add lexicographic and Pareto views without weakening gates + +A weighted aggregate is useful for total ordering, but the axes remain the +truth. Griff should support analysis views that expose: + +- lexicographic ordering over named axes; +- Pareto-nondominated candidate sets; +- weighted ordering within a chosen admissible set. + +These are ranking views only. The pipeline remains: + +```text +hard-rule admission + -> optional lexicographic/Pareto selection + -> weighted total order where required +``` + +No Pareto or weighted comparison may revive a candidate rejected by a hard +rule. A beautiful unplayable part remains unplayable, however moving its +rationale may be. + +### 3.5 Define `ArrangementIntent` as the only LLM-facing generation contract + +An LLM integration, if added, produces a typed intent rather than notes or a +patch to the canonical score. + +Illustrative shape: + +```text +ArrangementIntentV1 + reference material identity + requested section/range + relationship mode + section roles + rhythm strategy + harmonic strategy + register policy + technique budgets + energy curve + enabled opt-in rules + soft-preference preset / weight-policy reference +``` + +Contract rules: + +- all referenced score objects and ranges must resolve before execution; +- enums and numeric ranges are closed and versioned; +- intent canonicalization occurs before hashing or execution; +- unsupported or contradictory fields produce typed refusals; +- the LLM cannot provide executable code, arbitrary rule expressions, direct + file paths, raw MIDI events, or unrestricted scoring formulas; +- generated score material is produced only by deterministic Griff components; +- every result binds the exact intent digest, model/provider receipt where an + LLM was involved, generator version, rule set, weights version, and seed where + a stochastic component is explicitly part of the contract. + +A human-authored intent and an LLM-authored intent are the same object after +validation. No downstream component branches on whether prose or silicon +produced it. + +### 3.6 Treat an LLM as an optional proposer, never an authority + +The minimum integration protocol is: + +```text +untrusted model response + -> syntax validation + -> structural validation + -> repository/score-reference validation + -> policy validation + -> canonical ArrangementIntent + -> deterministic execution + -> hard-rule admission +``` + +Provider-side structured output is an ergonomic aid, not the trust boundary. +The local validator remains authoritative because closed providers expose +different schema subsets and can still return semantically invalid but +well-formed values. + +The production core must remain usable with no LLM configured. Closed and open +models are adapters above the same intent contract; no safety or validity +property may depend on access to logits, weights, or a provider seed. + +## 4. When A* becomes justified + +A* is not the next replacement for layered DP. It becomes a candidate only for +a client whose state graph is implicit and generated lazily, for example a +partial phrase with unresolved rhythmic obligations, motif callbacks, +fretboard state, and a target closure condition. + +An A* proposal must name all of the following: + +```text +state identity and canonicalization +actions and deterministic successor order +goal predicate +path-cost contract +heuristic definition +proof or test of admissibility/consistency where exact optimality is claimed +budget and incomplete-search result +provenance for approximate modes +``` + +Acceptance requires a representative benchmark against the simplest applicable +baseline: exact layered DP when the graph can be enumerated, otherwise Dijkstra +or uniform-cost search over the same implicit client. + +Kill criteria: + +- no material reduction in expanded states or latency; +- heuristic computation consumes the saved work; +- state canonicalization cannot prevent duplicate exploration; +- result quality depends on undocumented successor order; +- the client can be represented naturally as a manageable layered graph; +- an approximate mode cannot clearly report its suboptimality contract. + +Weighted A* may later serve an interactive preview, but it must use a distinct +search-mode identity. `best` may not quietly become `first result before the UI +became impatient`. + +## 5. Constraint Lab role + +The Constraint Lab remains an offline oracle, never a production dependency. +Its next useful targets are: + +- chord-voicing feasibility once the deferred model exists; +- bounded small instances used to verify k-best uniqueness and ordering; +- counterexample generation for new hard-rule interactions; +- exact feasibility witnesses for implicit-search clients; +- differential checks between production search and solver-neutral models. + +The Lab is not a shadow runtime. Production results may be compared with pinned +oracle runs in tests and audits, but preview and generation cannot shell out to +MiniZinc or a CP solver. + +Every oracle extension preserves the current discipline: exact input, frontend +and backend identities, archived outcome, independently checked SAT witnesses, +and solver failure distinct from UNSAT. + +## 6. Determinism and provenance contract + +Every search result must be reproducible relative to a complete contract, not a +seed alone. + +Minimum provenance: + +```text +canonical input identities and digests +search mode and version +state/client version +hard-rule set and version +weight policy and version +diversity policy and version, if any +normative arithmetic contract +canonical tie-break contract +budget and completeness status +seed only for explicitly stochastic upstream construction +``` + +Changing any item creates a different result contract. Stored paths without +this provenance are display artifacts, not reproducible search evidence. + +## 7. Proposed delivery order + +This proposal does not assign roadmap stages. If accepted, implementation work +should be split into independently reviewable ADRs/slices: + +1. **Status reconciliation** — confirm whether ADR-0017 is still truthfully + `Proposed` now that `Scored` has multiple code consumers; update canonical + status only through the repository's normal review process. +2. **k-best contract and characterization** — specify total ordering, + canonical path identity, completeness, and `k = 1` equivalence before code. +3. **k-best implementation** — tests first, same cost evaluator, bounded API. +4. **DiversityPolicy v1** — deterministic filter mode over k-best results with + measurable path-distance evidence. +5. **ArrangementIntent v1** — schema, canonicalization, typed refusals, and a + human-authored adapter before any LLM provider adapter. +6. **LLM adapter spike** — one optional provider behind the local validator; + no production authority and no core dependency. +7. **A* client spike, only if earned** — a named implicit client plus benchmark + and kill criteria. + +## 8. Validation expectations + +Documentation acceptance should require consistency with ADR-0013, ADR-0030, +ADR-0017, the hard-constraint proposal, the Constraint Inventory, and the +existing Lab report. + +Later implementation acceptance should include: + +- exhaustive tiny layered graphs compared with brute-force enumeration; +- metamorphic tests for stable prefixes as `k` grows; +- duplicate-state and duplicate-path adversarial fixtures; +- deterministic tie fixtures including equal aggregate costs; +- arithmetic edge cases and non-finite accumulation refusals; +- diversity fixtures proving every selected/refused pair measurement; +- intent canonicalization and hash stability tests; +- invalid, contradictory, stale-reference, and unsupported intent fixtures; +- differential oracle cases where the Lab has a matching bounded model; +- performance thresholds stated before selecting a more complex algorithm. + +## 9. Non-goals + +- Replacing DP/Viterbi with A* globally. +- Making an LLM mandatory for generation. +- Letting an LLM write canonical notes, tabs, rules, or score weights directly. +- Treating random seeds as a substitute for explicit diversity. +- Adding MiniZinc, CP-SAT, or another solver to the production path. +- Collapsing hard constraints, diversity, and musical preferences into one + scalar. +- Claiming that k-best paths are musically diverse without a measured policy. +- Training or fine-tuning a model in this proposal. + +## 10. Prior art to evaluate in the accepting ADRs + +The implementation ADRs should survey, benchmark, and cite the original or +primary descriptions of: + +- Viterbi and k-best dynamic-programming variants; +- Yen-style deviation enumeration for loopless k-shortest paths; +- Eppstein-style k-shortest-path enumeration where its graph assumptions fit; +- diverse M-best / diversity-aware structured prediction; +- A* and weighted A* for implicit state spaces; +- MiniZinc solution enumeration and diversity mechanisms; +- CP/SAT no-good constraints for enumerating distinct feasible solutions. + +Idea reuse is expected; code reuse remains subject to Griff's licence and +dependency policy.