From 37893647e9fd12b3b142be3d9ca3e422b0d969de Mon Sep 17 00:00:00 2001 From: Rafael Vuijk Date: Fri, 14 Aug 2026 23:45:52 +0000 Subject: [PATCH] Say where the corpus gate is and what to do when it reports an improvement (#529, #500) The corpus now runs in the suite, so it runs on every commit, and a contributor who makes the library solve something new will meet it as a failing test rather than as a green one. AGENTS.md says what to do about that -- record the improvement in Corpus.cs in the same change -- because a gate that fails on getting better is confusing exactly once and then only if nobody wrote it down. It also says what the gate is not. It is small and takes about a second because everything in the suite is paid for on every commit; the harnesses in work/ are where a measurement generates its own inputs and takes minutes. A finding from one of those that is worth keeping belongs in the corpus as a new problem. CLAUDE.md's paragraph on the harnesses said they all live outside the repository, which is now one sentence short of true. https://github.com/asc-community/AngouriMath/issues/529 https://github.com/asc-community/AngouriMath/issues/500 Co-authored-by: Claude Opus 5 --- AGENTS.md | 24 ++++++++++++++++++++++++ CLAUDE.md | 6 ++++++ 2 files changed, 30 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index cb16332a5..87a4d5b20 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -176,6 +176,30 @@ counts alongside the solved count. A change that solves one more problem and int answer is a regression. Compare against SymPy, Mathematica, or a textbook — being different from SymPy is not automatically being wrong, but it is always worth explaining. +### The corpus runs on every commit, and it records what it found + +`Sources/Tests/UnitTests/Corpus` is that measurement, in the suite, so it runs on every commit. Each +problem carries the verdict it currently earns and the gate fails on two things: **any wrong answer**, +and **any case that stops matching its record** — including one that gets *better*, so the record +cannot quietly drift away from the library. + +So when a change makes the corpus solve something it did not: + +1. the gate fails and names the case; +2. **update its `Expect` in `Corpus.cs` in the same change**, which is how the improvement gets + recorded rather than absorbed; +3. and if it is worth a line to a user, `BREAKING-CHANGES.md` too. + +Answers are checked, not compared against stored text — a root is substituted back into its equation, +an antiderivative is differentiated back, a simplification is evaluated against the expression it came +from. A change of *form* is therefore not a failure and only a change of *value* is, which is what +lets the corpus stay useful while printed output moves. + +**It is a gate and not a harness.** It is small and takes about a second, because everything in the +suite is paid for on every commit. The harnesses in `work/` are where a measurement generates its own +inputs, takes minutes, and gets read by a person; the two are not substitutes, and a finding from a +harness that is worth keeping belongs in the corpus as a new problem. + ## One structure under several features Four things in this library are the same shape, and three of them were written separately before diff --git a/CLAUDE.md b/CLAUDE.md index dafb1fbf2..ca667fcfa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,3 +23,9 @@ The measurement harnesses live outside this repository, in the analysis workspac (`work/`): a self-verifying solver corpus, a property checker, root-completeness and simplification sweeps, a boundary checker, a crash harness that survives a stack overflow, and a checker for the documentation's code samples. Run them before claiming anything is fixed. + +Inside the repository there is one measurement, and it is a *gate* rather than a harness: +`Sources/Tests/UnitTests/Corpus` runs forty problems on every commit and reports **solved / unsolved +/ wrong / error / timeout**. It fails on a wrong answer, and on any case that stops matching its +recorded verdict — including one that improves, which is a prompt to record the improvement rather +than a complaint. See *The corpus runs on every commit* in [AGENTS.md](AGENTS.md).