Compute surface suite

[def-]

Motivation

Compute's rendering path has no generated correctness suite. sqllogictest checks
SQL semantics end to end, and the mz-transform fuzz targets check MIR-to-MIR
equivalence under the optimizer, but nothing draws a plan, renders it on a real
replica, and compares the result against an independent reference. This adds
that, on top of the existing clusterd-test-driver.

The suite draws MIR from the same generator the fuzz targets use, lowers it, and
renders it against a real clusterd with no environmentd in the picture.

Description

Four oracles, layered so each covers where the previous goes blind, all applied
at every timestamp the workload writes:

• FoldConstants, the primary one. Substitute each input's actual contents
  into the plan as literals and evaluate with the optimizer's constant folder, an
  implementation independent of the renderer. Blind to LetRec.
• Export invariance. An index, a materialized view and a subscribe over the
  same computation, rendered as three separate dataflows, must agree. Needs no
  reference implementation.
• Incremental. The maintained collection must equal a dataflow created with
  its as_of already at that timestamp. The only oracle that can judge a
  recursive collection.
• Strategy invariance. A pairwise matrix over the compute strategy dyncfgs
  must not change the answer.

Coverage is measured rather than asserted. surface.rs classifies a lowered
plan into cells, the generator keeps a draw only if it covers something new, and
the runner re-derives the cells at run time and fails if they differ from what
the workload claimed. KNOWN_GAPS records what is not reached and why, and a
test fails if an entry names a cell that is now covered.

src/transform/src/mirgen.rs (behind a mirgen feature) holds the generator and
the fold oracle, extracted from src/transform/fuzz so the main workspace can
use them. FuzzEntropy delegates one-to-one so byte consumption is unchanged:
that is what decides which plan a stored fuzz corpus entry decodes to, and
release qualification carries a minimized corpus between runs.

Design notes and the full findings list are in
doc/developer/compute-surface-suite.md.

What the review of the first draft changed

The suite was green and had found nothing. The reasons were all in the corpus and
the observation points, and the last commit fixes them:

• Nothing was ever maintained incrementally. The exports were created with
  as_of already at the assertion timestamp, so each read a snapshot at the
  final timestamp and stopped. The incremental oracle compared two snapshot
  dataflows. Exports now start at as_of = 0, and every oracle runs at every
  timestamp, so a collection that passes through a wrong intermediate state fails
  even when it converges.
• The corpus computed almost nothing. Full-width integer draws mean two
  values never coincide, so equi-join keys never met, and a fifth of all leaves
  drew zero rows and annihilated everything above them. Of six join workloads,
  four had an empty input, one was a cross product, one had a single row. Values
  are now folded into a small domain, empty leaves are redrawn, and a liveness
  filter rejects a candidate that computes nothing.
• A fixed seed cannot find a new bug. workloads keeps it, because a
  regression suite wants the same plans every night. The new soak step seeds
  from the build number, keeps every draw, and prints the seed to replay it.
• Vacuity was unmeasurable. Two empty collections agree, so a suite that
  stops computing anything reads exactly like a passing one. There is now a test
  that the corpus computes something, a test that every dyncfg in the matrix
  still exists (ConfigUpdates::apply skips unknown names silently, so a rename
  would degenerate the matrix into eight identical runs), and a run reports every
  cell it only ever rendered over an empty collection.
• Retractions now take rows to multiplicity zero rather than only lowering them,
  and eleven shapes cover surfaces nothing reached: a recursion that iterates,
  mutual recursion, an error appearing and being retracted mid-stream, a
  hierarchical reduce losing its running extreme, a collection emptying out, a
  join whose keys meet, a lookup whose key exists, a literal collection, and
  array_agg, which turns out to be the one Basic aggregate expressible over
  integer columns and so closes both Reduce/Basic gaps and the
  render_basic_aggregates path with them.

The finding it then produced

An index peek and a materialized-view read of the same collection at the same
timestamp reported different errors:

Index:            Invalid data in source errors, saw retractions (12) for
                  row that does not exist: Evaluation error: division by zero
MaterializedView: Evaluation error: division by zero

An error is emitted with the multiplicity of the row it was raised on, and
Negate negates its input's rows while passing its errors through, so an
expression evaluated above a Negate raises errors with negative multiplicity.
EXCEPT ALL plans to Threshold(Union(lhs, Negate(rhs))), and predicate
pushdown deliberately leaves a literal-error predicate above the Negate so that
those errors cancel against the other branch's. That cancellation is
load-bearing: it is what keeps the null-extended branch of an outer join from
raising spurious errors, and database-issues#5691's regression test fails if the
predicate is pushed through instead (I tried).

So errors cancel between unrelated rows, and the row counts decide the outcome of

SELECT * FROM (SELECT x FROM a EXCEPT ALL SELECT x FROM b) WHERE 1/0 > 0

• equal contributions: the errors cancel, no error, an empty result
• right side heavier: a negative remainder, reported as Invalid data in source errors and logged at error!, i.e. paging on a plain user query
• left side heavier: the error

None of that is a claim about what should happen. Materialize has no error
semantics, they are a byproduct of render.rs, and STG-54 is the standing
account of what has to be decided. This case is its 1.8.1, which leans towards
errors from different records never cancelling while noting that we rely on the
cancellation internally, and the Negate mechanism is its own entry there, down
to needing error provenance to resolve. test/sqllogictest/error_semantics.slt
now records all three cases rather than asserting one, which is what that file is
for.

What is a defect either way, and all the second commit changes, is narrower: two
read paths over one collection disagreed at one timestamp, and a condition a
plain query can reach was reported in the storage layer's vocabulary (the message
of nearly the same name really does mean an upstream source sent retractions for
rows it never sent, and pg-cdc asserts on it) and logged at error!. It now
reports the error itself, which is what the persist and subscribe paths of the
same collection already do, and logs at warn. No semantics change.

Verification

• cargo nextest run -p mz-clusterd-test-driver: 49 tests, including the corpus
  self-consistency, non-vacuity, gap-list and dyncfg-existence guards.
• The generated corpus against a real clusterd: 31 workloads x 8 configurations
  x up to 4 timestamps each, 38 surface cells, green at one, two and four timely
  workers.
• All 13 hand-written .spec scenarios (SCRIPT=all), which assert on driver
  output including error text.
• bin/sqllogictest -- test/sqllogictest/error_semantics.slt test/sqllogictest/transform/predicate_pushdown.slt, the latter because it
  holds database-issues#5691's regression test.
• Soak: thousands of freshly drawn plans across many seeds and worker widths, at
  two configurations each. This is what found the bug above; the seed that found
  it now runs 163/163 clean.

Nightly runs three steps, separate so one cannot mask another: scripts,
workloads (fixed seed, one and two workers), and soak (new plans each run).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dz1B3CLa3AeEn94Ln2fASF