Skip to content

Say where the corpus gate is and what to do when it reports an improvement (#529, #500) #948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Rafael-SOWNet merged 1 commit into from
Aug 15, 2026
Merged

Say where the corpus gate is and what to do when it reports an improvement (#529, #500) #948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,30 @@ counts alongside the solved count. A change that solves one more problem and int
answer is a regression. Compare against SymPy, Mathematica, or a textbook — being different from
SymPy is not automatically being wrong, but it is always worth explaining.

### The corpus runs on every commit, and it records what it found

`Sources/Tests/UnitTests/Corpus` is that measurement, in the suite, so it runs on every commit. Each
problem carries the verdict it currently earns and the gate fails on two things: **any wrong answer**,
and **any case that stops matching its record** — including one that gets *better*, so the record
cannot quietly drift away from the library.

So when a change makes the corpus solve something it did not:

1. the gate fails and names the case;
2. **update its `Expect` in `Corpus.cs` in the same change**, which is how the improvement gets
recorded rather than absorbed;
3. and if it is worth a line to a user, `BREAKING-CHANGES.md` too.

Answers are checked, not compared against stored text — a root is substituted back into its equation,
an antiderivative is differentiated back, a simplification is evaluated against the expression it came
from. A change of *form* is therefore not a failure and only a change of *value* is, which is what
lets the corpus stay useful while printed output moves.

**It is a gate and not a harness.** It is small and takes about a second, because everything in the
suite is paid for on every commit. The harnesses in `work/` are where a measurement generates its own
inputs, takes minutes, and gets read by a person; the two are not substitutes, and a finding from a
harness that is worth keeping belongs in the corpus as a new problem.

## One structure under several features

Four things in this library are the same shape, and three of them were written separately before
Expand Down
6 changes: 6 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,9 @@ The measurement harnesses live outside this repository, in the analysis workspac
(`work/`): a self-verifying solver corpus, a property checker, root-completeness and
simplification sweeps, a boundary checker, a crash harness that survives a stack overflow, and a
checker for the documentation's code samples. Run them before claiming anything is fixed.

Inside the repository there is one measurement, and it is a *gate* rather than a harness:
`Sources/Tests/UnitTests/Corpus` runs forty problems on every commit and reports **solved / unsolved
/ wrong / error / timeout**. It fails on a wrong answer, and on any case that stops matching its
recorded verdict — including one that improves, which is a prompt to record the improvement rather
than a complaint. See *The corpus runs on every commit* in [AGENTS.md](AGENTS.md).
Loading