Add v0.4.0 sign convention to the manuscript; real-archive validation section

.claude/skills/codameter-advisor/SKILL.md

@@ -10,12 +10,12 @@ description: >
   glacier / geothermal site," or wants their parameter choices adapted to a
   specific monitoring use case. The skill is an orchestrator: it elicits the
   user's use case, maps it to a recommended processing-choice set from the
-  literature survey (codameter.use_cases), then proves the recommendation by
-  running the real synthetic engine live, quantifying the bias and error-bar cost
+  literature survey (codameter.use_cases), then assesses the recommendation conditionally by
+  running the real synthetic engine live, quantifying the recovery error and availability
   of the recommended versus the user's current choices on a matched synthetic
   with known ground truth (codameter.golden + codameter.deviations). It never
   invents parameters from memory; every recommendation is grounded in
-  literature/best_practices.md and validated numerically. Also use it to add or
+  literature/best_practices.md and checked on the stated synthetic scenario. Also use it to add or
   regenerate golden synthetic datasets. Do not use it to review a manuscript
   (that is pre-submission-reviewer).
 ---
@@ -52,7 +52,7 @@ So you split the work into four steps and wire them together:
 1. **Elicit** the use case and the user's current choices.
 2. **Map** to a recommended config from `codameter.use_cases`.
 3. **Validate** live: run the recommended and the user's config on a matched
-   synthetic, report the bias and error-bar difference.
+   synthetic, report the recovery error and availability.
 4. **Report** the config, the rationale with citations, and a reproducible snippet.
 
 You run the Python through `pixi run python` (the default pixi env has codameter
@@ -97,20 +97,24 @@ matter most for this use case.
 
 ## STEP 3: VALIDATE LIVE
 
-Follow `references/validation_loop.md`. Synthesize a matched scenario (reuse a
-golden case when the application maps to one, else `codameter.golden.generate`),
-then run `codameter.deviations.run_pipeline` for the recommended config and for
+Follow `references/validation_loop.md`. Build a public development scenario with `codameter.golden.advisory_case`.
+This supports all six applications independently of the evaluation corpus.
+Run `codameter.golden.recover` for the recommended config and for
 the user's current or a deliberately naive config. Report:
 
 - the RMS error against the known truth for each config,
 - the difference in the recovered signal (for a transient, the recovered drop),
 - for the volcano-style factorial, the first-order variance attribution from
   `codameter.deviations.multiverse` (which choice controls the answer),
-- optionally the marginal measurement covariance `C_d` from
+- optionally the constructed single-member covariance `C_d` from
   `codameter.uq_bayes.bayes_dvv_from_ccfs`.
 
-State the numbers you actually got. If the user's choice is within noise of the
-recommendation, say so; do not manufacture a difference.
+State the numbers you actually got, including availability on fixed support.
+A single seeded RMS comparison cannot establish statistical equivalence.
+Report which elicited properties the scenario represents and which it omits.
+Only the six config axes are executable overrides; site geometry, forcing,
+noise and gaps require explicit scenario construction. Do not imply they were
+modeled merely because the user supplied them.
 
 ## STEP 4: REPORT
 
@@ -126,9 +130,8 @@ caveat that matters most for this use case (the `key_rule`).
 See `references/golden_datasets.md`. The corpus lives in
 `tests/data/golden/manifest.json` (recipes plus expected metrics); arrays are
 regenerated from seeds on demand and cached under `tests/data/golden/cache/`
-(gitignored). To add a case, append a recipe to `codameter.golden.CASES`, then
-run `pixi run golden` to refresh the manifest, and `pixi run -e test pytest
-tests/test_golden.py` to lock it in. Never commit the `.npz` cache.
+(gitignored). For deliberate corpus changes, follow `references/golden_datasets.md`.
+Do not refresh thresholds merely to make a failing check pass. Never commit the `.npz` cache.
 
 ---
 

.claude/skills/codameter-advisor/references/golden_datasets.md

@@ -1,8 +1,10 @@
 # Golden datasets
 
 Seeded synthetic CCF suites with known ground-truth dv/v(t), organised as a
-**graded benchmark**: 30 cases, 10 per difficulty grade, spanning the monitoring
-applications. Two consumers: the pytest regression oracle
+**graded template family**: 30 templates, 10 per difficulty grade.
+The checkout exposes three public cases, one per grade. A separately
+provisioned corpus supplies private cases. Public examples are development
+checks; no completed external-model evaluation is claimed. Two consumers: the pytest regression oracle
 (`tests/test_golden.py`) and this advisor's live validation.
 
 ## Layout
@@ -27,11 +29,12 @@ earthquake/fault, landslide, groundwater, cryosphere, geothermal).
 - **medium** (split `validation`): a transient coseismic-style drop with
   logarithmic partial healing, plus more measurement noise (SNR 3-5).
 - **hard** (split `test`): a **multi-channel** (4-channel) *and*
-  **depth/frequency-dependent** problem. A shallow (high-frequency) layer carries
+  **frequency-component selection** problem. A shallow (high-frequency) layer carries
   a coseismic drop-and-heal plus a full hydrological seasonal cycle; a deep
   (low-frequency) layer carries a long-term trend. Each case targets one depth
-  (`target: shallow|deep`), so the measurement **band selects the depth** and must
-  match the target. Low SNR (2-4) with waveform decorrelation; channels are
+  (`target: shallow|deep`), so the measurement band selects the imposed component and must
+  match the target. This separated-band surrogate does not validate physical
+  depth resolution or kernels. Low SNR (2-4) with waveform decorrelation; channels are
   measured independently and aggregated (`golden.recover`).
 
 The benchmark therefore grades estimator, reference, stacking, aggregation *and*
@@ -51,9 +54,9 @@ for c in m['cases']:
 
 ## Add or change a case
 
-1. Append a recipe dict to `codameter.golden.CASES` (see the docstring there for
-   the fields). Reuse the synthesis geometry from `codameter.use_cases` via the
-   `use_case` key; only add a new ground-truth generator in `golden.TRUTH` if no
+1. Append a recipe dict to the explicit public recipe source or private corpus builder, as appropriate.
+   `CASES` is populated when the module loads; it is not a persistent registry. Reuse the synthesis geometry from `codameter.use_cases` via the
+   `use_case` key; only add a new ground-truth generator in `golden.MOTIF` if no
    existing one fits.
 2. Regenerate the oracle: `pixi run golden`. Review the printed RMS values; they
    should be small for a recovery case and a stable non-zero value for an
@@ -75,3 +78,17 @@ The same cases are exposed as a FrugalMind benchmark through
 regression vs truth). Export with `pixi run frugalmind-export`; the drop-in
 suite is in `integrations/frugalmind/`. Adding or changing a golden case updates
 the FrugalMind rows automatically, since both read `golden.CASES`.
+
+## Evaluation limits
+
+Public templates and application defaults share construction assumptions.
+Hidden amplitudes do not establish a holdout of waveform physics. Difficulty
+and split are confounded in the current template family. Frozen RMS tolerances
+are regression tolerances, not scientific accuracy requirements.
+
+`observed()` removes truth keys from a dictionary; it is not a sandbox.
+An evaluated agent must not access recipe files, truth caches, scorer metadata,
+or generator routes that reconstruct the answer. Process and filesystem
+isolation remain work for the evaluation harness. Scorer support is fixed per
+case; missing predictions are scored as null change and availability is reported.
+No model transcripts or validated agent-performance claims accompany this corpus.

.claude/skills/codameter-advisor/references/report_format.md

@@ -24,25 +24,24 @@ difference or invent one that is not there.
 3. **Why**: one line per axis, each with its driver and a citation from
    `USE_CASES[key]`. Name the one or two axes that matter most for this use case.
 
-4. **Validation**: the table from Step 3:
+4. **Validation**: report measured RMS, fixed-support availability, and the
+   scenario recipe from Step 3. The following values illustrate formatting only:
 
    | Config | RMS vs truth | Recovered signal |
    | --- | --- | --- |
    | recommended | 0.024 % | drop -0.40 % (true -0.40 %) |
    | user / naive | 0.31 % | drop -0.18 % |
 
-   State the ratio and what it means in one sentence. If a factorial was run, add
+   State the difference and availability. Do not infer equivalence from one seed. If a factorial was run, add
    the top variance-driving axis.
 
 5. **Reproduce**: a short snippet the user can paste:
 
    ```python
    from codameter import use_cases as uc, golden
-   from codameter.deviations import run_pipeline
    key = "volcano"
-   d = golden.generate(golden.MAINSTREAM_BY_USE_CASE[key])
-   dvv, valid = run_pipeline(d["ccfs"], d["t"], d["fs"],
-                             uc.recommend(key), eps_max=uc.eps_max(key))
+   d = golden.advisory_case(key, seed=101)
+   dvv, valid = golden.recover(d, uc.recommend(key), uc.eps_max(key))
    ```
 
 6. **One caveat**: the `key_rule` for this use case, verbatim in substance (for

.claude/skills/codameter-advisor/references/validation_loop.md

@@ -1,101 +1,77 @@
-# Live validation
+# Live synthetic assessment
 
-This is the step that makes the advice more than an opinion. Run the recommended
-config and a comparison config on a matched synthetic with known ground truth,
-and report the difference in bias and error bar. The truth is known exactly, so
-every difference is an artifact of the processing choice, not of nature.
+Use public development scenarios for advice. Synthetic recovery measures
+performance conditional on the imposed truth, waveform, noise, and choices.
+It does not prove a recommendation for field observations.
 
-## The primitives
+## Recommended versus comparison
 
-- `codameter.golden.generate(case_id)` returns `{ccfs, t, days, truth, fs, ...}`
-  for a seeded case. `codameter.golden.MAINSTREAM_BY_USE_CASE[use_case]` gives a matched
-  (easy-grade) case id for an application.
-- `codameter.deviations.run_pipeline(ccfs, t, fs, cfg, eps_max=...)` returns
-  `(dvv, valid)` for one config.
-- `codameter.golden._rms(dvv, truth, days, valid)` is the baseline-aligned RMS
-  error (removes the unobservable DC offset of a relative measurement).
-- `codameter.deviations.multiverse(...)` returns the first-order variance
-  attribution (volcano synthetic; use it for the "which choice controls the
-  answer" statement).
-- `codameter.uq_bayes.bayes_dvv_from_ccfs(ccfs, t, fs, truth=truth, days=days)`
-  returns `(BayesResult, EnsembleRun)`; `BayesResult.Cd` is the marginal
-  measurement covariance for a downstream inversion.
+`golden.advisory_case` builds a seasonal example for every application.
+It does not depend on `MAINSTREAM_BY_USE_CASE` or private cases.
+The returned recipe records the seed, duration, application, and noise.
+Application defaults share assumptions with the generator. This is a
+consistency check, not an independent generalization benchmark.
 
-## Recommended vs comparison, on a matched synthetic
-
-Fill `USER_CFG` with the user's current choices (or a deliberately naive config
-if they have none). Run:
-
-```bash
-pixi run python - <<'PY'
+```python
+import numpy as np
 from codameter import use_cases as uc, golden
-from codameter.deviations import run_pipeline
-
-USE_CASE = "volcano"                     # from Step 1
-USER_CFG = {"reference": "moving"}       # the user's current choice(s), as overrides
-
-key = uc.resolve(USE_CASE)
-d = golden.generate(golden.MAINSTREAM_BY_USE_CASE[key])
-eps = uc.eps_max(key)
 
+key = uc.resolve("volcano")
+user_overrides = {"reference": "moving"}
+d = golden.advisory_case(key, seed=101)
 rec = uc.recommend(key)
-usr = uc.recommend(key, **USER_CFG)
-
-for label, cfg in [("recommended", rec), ("user/naive", usr)]:
-    dvv, valid = run_pipeline(d["ccfs"], d["t"], d["fs"], cfg, eps_max=eps)
-    rms = golden._rms(dvv, d["truth"], d["days"], valid)
-    print(f"{label:<12} {cfg}")
-    print(f"{'':<12} RMS vs truth = {rms*100:.4f} %  (valid epochs {int(valid.sum())})")
-PY
-```
-
-Report the two RMS values and the ratio. If the user's config is within ~20 % of
-the recommended RMS, tell them their choice is fine; do not invent a penalty.
-
-## Which choice controls the answer (volcano factorial)
-
-For the ranking of axes by impact, run the one-at-a-time sweep or the multiverse:
-
-```bash
-pixi run python - <<'PY'
-from codameter.deviations import multiverse
-mv = multiverse(years=1.5, cadence=4)
-print("pipelines:", mv["n_pipelines"])
-for axis, frac in sorted(mv["sobol_rms"].items(), key=lambda kv: -(kv[1] or 0)):
-    print(f"  {axis:<11} first-order variance share of RMS = {frac:.2f}")
-PY
-```
-
-`multiverse` is wired to the volcano truth and geometry. For other applications,
-report the OAT contrast from the recommended-vs-comparison run above rather than
-claiming a full factorial you did not run.
-
-## Optional: the marginal covariance C_d
-
-For a user heading into a depth or stress inversion, show that the honest error
-bar comes from marginalising the processing choice:
-
-```bash
-pixi run python - <<'PY'
-from codameter import golden
-from codameter.uq_bayes import bayes_dvv_from_ccfs
-d = golden.generate("easy-volcano-01")
-res, ens = bayes_dvv_from_ccfs(d["ccfs"], d["t"], d["fs"],
-                               truth=d["truth"], days=d["days"], cadence=4)
-import numpy as np
-print("posterior median dv/v std:", float(np.nanmedian(np.sqrt(np.diag(res.Cd))))*100, "%")
-PY
+usr = uc.recommend(key, **user_overrides)
+eps = uc.eps_max(key)
+support = golden.scoring_support(d, rec, eps)
+print("scenario:", d["recipe"])
+for label, cfg in [("recommended", rec), ("comparison", usr)]:
+    dvv, valid = golden.recover(d, cfg, eps)
+    prediction = np.where(valid, dvv, np.nan)
+    rms, availability = golden.rms_on_support(prediction, d["truth"], **support)
+    print(label, cfg, "RMS [%]", rms * 100, "availability", availability)
 ```
 
-This is slower (it runs an ensemble of pipelines plus a Gibbs sampler); only run
-it when the user cares about the propagated uncertainty, and say it is running.
-
-## Reading the numbers honestly
-
-- A lower RMS is better recovery; state the percentage, not an adjective.
-- A `moving` reference erases the slow trend, so on a trend case it shows a large
-  RMS by design. That is the point, not a bug.
-- Harder cases (medium = transient + noise; hard = multi-channel composite)
-  are in the golden manifest; pull one with `golden.generate("<grade>-<app>-<nn>")`
-  when the user's situation is noisier or more complex than a clean seasonal one.
-  Ids are in `golden.CASES_BY_ID`.
+`recover` measures each channel before averaging multi-channel cases.
+Both configurations use the reference configuration's fixed epochs and datum.
+Missing predictions on that support count as zero baseline-relative change.
+Report availability separately. Selective abstention can still improve a poor
+prediction; the score is not evidence of complete temporal recovery.
+A non-finite RMS means the comparison lacks a usable datum.
+
+For a named public evaluation case, inspect `golden.CASES_BY_ID` first.
+Apply the recipe's `config` overrides before creating the reference config.
+A hard groundwater case targets a particular frequency component; its band
+is not the generic groundwater default. Never open private evaluation data
+for routine advice or display scorer truth to an evaluated agent.
+
+## Interpret the comparison
+
+Report RMS in percent, the difference, and both availabilities.
+Do not call choices equivalent because their RMS values differ by 20 percent.
+Repeated independent waveform/noise realizations are needed to quantify the
+uncertainty of that difference. Record all seeds and settings when repeating.
+The default example is seasonal; transient recovery requires a stated
+transient scenario, not a claim inferred from the seasonal run.
+A trailing reference without accumulation measures a different temporal
+quantity. Label that comparison as an ablation.
+
+The six pipeline axes are executable overrides. Other elicited properties
+are context until explicitly encoded in the generator. State unmodeled
+geometry, target depth, forcings, SNR, artifacts, and gaps.
+
+## Optional factorial and covariance
+
+`codameter.deviations.multiverse` uses the volcano scenario. Its sensitivity
+ranking is conditional on that waveform and sampled configuration menu.
+Do not report it as an application-independent ranking or an attribution
+across independent field observations.
+
+`bayes_dvv_from_ccfs` accepts a complete daily CCF grid. It returns a model
+posterior and a separately constructed single-member covariance `Cd`.
+`mu_cov` describes the combined estimate under the conditional-independence
+model. Neither object automatically captures shared artifacts. The locked
+calibration reports poor credible-band coverage on the combined estimate.
+A single-member `Cd` is not a calibrated covariance of that estimate or a
+cross-band covariance for depth inversion. Explain these distinctions before
+using either object downstream. Report standard deviations in fractional
+dv/v or percent explicitly; covariance units are fractional dv/v squared.

.pre-commit-config.yaml

@@ -26,5 +26,5 @@ repos:
     rev: v1.7.0
     hooks:
       - id: mypy
-        additional_dependencies: [numpy, types-PyYAML]
+        additional_dependencies: ["numpy<2.5", types-PyYAML]
         exclude: ^(tests/|examples/|docs/|scripts/)