유명 데이터셋의 true parameter와 추정값 비교해서 알고리즘 교정

[seonghobae]

유명 데이터셋의 true parameter와 추정값 비교해서 알고리즘 교정

대상: 이 라이브러리에서 지원하는 모든 추정 방법
데이터: Full-information item factor analysis 방식에서 지원하는 모든 유명 데이터셋과 그의 true parameter (알려져 있는 데이터를 찾아야 함)
5회 반복의 RMSE 계산

[seonghobae]

Execution checklist

• Build estimator coverage matrix from R/kaefa.R + R/newEngine.R
• Create dataset registry (source/DOI/license/version/checksum/true-parameter citation)
• Define fixed 5-seed RMSE protocol and parameter-alignment rules
• Run smoke subset first, then full matrix runs with failure logging
• Produce correction backlog ranked by RMSE gap magnitude

Acceptance criteria

• Method coverage matrix is complete and posted
• Every included dataset has verifiable true-parameter source
• Exactly 5 repeats per valid method-dataset pair
• Output schema fixed: per-run RMSE, mean/SD, failure log

Risk notes

• True-parameter provenance gaps -> strict inclusion/exclusion log
• Parameterization mismatch -> explicit alignment rules

[seonghobae]

Fleet quality handoff — numerical authority / recovery acceptance

Fresh owner-path evidence on 2026-09-02:

• PR feat(recovery): five-repeat AEFA true-parameter RMSE protocol #79 is Ready at exact head 1c5d9f0491fc178be3f7f307dac521fbcbba6978 on develop@5128d4867e24b5db73e6e3c8652a8dbeabd70aa0. Its own current contract says production/recovery remains R/mirt, with R/recovery.R computing RMSE/summary evidence and Rust/GPU numeric core explicitly excluded.
• PR feat(recovery): nested mixedmirt random-effect RMSE evidence #80 is Draft at exact head 1f05996780cf24ab32c747696764fee096747d64, stacked on feat(recovery): five-repeat AEFA true-parameter RMSE protocol #79. It fits the nested random-intercept Rasch recovery model through mixedmirt, estimates b and tau00, and likewise states that no Rust/GPU numeric core is claimed.
• Protected/default develop still defines kaefa-core as the R/statistical engine around aefa() / engineAEFA(), and R/newEngine.R directly owns full-information item-factor estimation/search through mirt/psych. docs/product-technical-gap-baseline.md is absent on the live default branch.
• A released reusable sibling exists: ContextualWisdomLab/fast-mlsirm v0.9.1. Its Rust core already owns production psychometric likelihood/EM/scoring families, multithreaded CPU execution, and a crossed/multiple-membership person-effect MAP kernel with CPU/GPU score reduction. That kernel explicitly does not estimate variance-component ML, so feat(recovery): nested mixedmirt random-effect RMSE evidence #80's tau00 estimand cannot be silently mapped to it; the missing variance-component/multilevel IRT capability must be implemented at the correct Rust owner boundary before replacing mixedmirt production authority.

This makes the previous “5 repetitions + RMSE” acceptance necessary but not sufficient under the current organization quality contract. Please keep #79/#80 as reference/recovery evidence until the dedicated kaefa writer establishes the production Rust boundary; do not promote R/mirt numerical paths as the long-term production authority merely because recovery tests are green.

Required RED → GREEN acceptance

1. Architectural RED first: add an executable contract that fails while a production aefa()/engineAEFA() path can perform likelihood, quadrature, E/M/MH-RM, vector/matrix algebra, item/person parameter estimation, variance-component estimation, or recovery aggregation without crossing an explicit versioned Rust API boundary. R may validate, marshal, orchestrate, and render results; mirt may remain a scientifically traceable reference/oracle in validation tests.
2. Owner/reuse decision: inventory each estimator family and map it to an existing released fast-mlsirm Rust primitive where semantics really match. For missing AEFA/full-information-factor or multilevel variance-component semantics, open/route the implementation to the actual reusable Rust owner instead of duplicating numerical kernels in R. Record the Context Map/ACL and consumer migration path.
3. Scientific RED: for every migrated estimator, compare the Rust result against known generating parameters and an independent reference implementation across prespecified Monte Carlo conditions. Preserve failed/non-converged replicates in the denominator; report signed bias, RMSE/MAE, convergence/failure rate and interval coverage where uncertainty is claimed. Five repeats may remain a smoke/regression slice, but must not be represented as enough to establish nominal coverage or production scientific validity.
4. Threshold provenance: RMSE < 0.35, cor(a) > 0.90, the feat(recovery): nested mixedmirt random-effect RMSE evidence #80 >60% interval-inclusion guard, and similar constants must each be either (a) justified by the estimand/design and cited evidence, or (b) explicitly labeled engineering regression guards rather than psychometric validity thresholds. Do not turn uncited rule-of-thumb constants into release truth.
5. Multilevel invariant: feat(recovery): nested mixedmirt random-effect RMSE evidence #80's nested random-intercept model must recover both item parameters and the actual group variance component under the intended estimator; a fixed caller-supplied prior precision or centered group-effect MAP estimate is not equivalent to estimating tau00. Multiple-membership/crossed/time-flow extensions remain fail-closed until their own estimands and recovery are implemented.
6. GREEN evidence: CPU multithreading must be deterministic within declared numeric tolerance; any GPU path must demonstrate CPU/GPU parity on the same estimands, convergence semantics, missingness, seeds and recovery protocol. No fallback may silently change the model or evidence semantics.
7. Documentation gate: update ARCHITECTURE.md, ADR/research traceability, CHANGELOG/NEWS as appropriate, and create/refresh docs/product-technical-gap-baseline.md so numerical ownership, reference-oracle status, unsupported models, recovery evidence, and remaining buyer-visible gaps are code-current.

Research basis already cited by the active stack remains appropriate for Monte Carlo/recovery design (Harwell, Stone, Hsu, & Kirisci, 1996; Chalmers, 2015; Chung & Beretvas, 2012; Diez-Roux, 1998). The key correction here is architectural/scientific truthfulness: recovery evidence validates a numerical implementation; it does not make the R/reference implementation the intended production computation boundary.