dogsdogsdogs: one lookup primitive under the delta-join operators

[frankmcsherry]

dogsdogsdogs: one lookup primitive under the delta-join operators

half_join, lookup_map, count, propose and validate were five hand-rolled
binary_frontier operators, each carrying its own copy of the stash-by-capability, frontier,
cursor-seek and compaction logic. They are one operator with five output functions:

for each (request, matching value): here are the consolidated (time, diff) arrangement
updates the cut admits. Do what you like with them.

operators::lookup is that operator. Two behaviours sit on it — lookup_map sums the admitted
updates into one diff and emits once (count), and lookup_join visits them individually,
joining each time into a time the request carries (propose, validate). half_join2 is
promoted to half_join, whose 366-line body becomes a 155-line derivation.

Two commits: the operator unification, then retiring AltNeu now that the cut is explicit.

Cuts are derived, not chosen

Cut::for_positions(seed, atom) returns the strict or lax comparison from the two atoms'
positions in the query. Rule seed claims a combination exactly when it is the largest index
achieving the Ord-maximum of the times, so exactly one rule claims each. Ordering by Ord
rather than PartialOrder is what makes that maximum exist — under the partial order two
incomparable times have no maximum, no rule claims the pair, and the match is silently lost.
Every other value a caller could pass either double counts or drops matches, so passing one is
now impossible.

The order time stays the timestamp; the join time rides in the payload

Progress tracking has to see the time the cuts compare against. Since order ⪯ join, a frontier
over join times would not bound the order times still in flight, and a downstream operator could
not set its compaction bound. So propose and validate emit per admitted update rather than
accumulating — which matters as soon as a cut admits an update incomparable to the request, i.e.
inside a nested scope.

AltNeu retired

AltNeu<T> encoded the same strict/lax choice in the timestamp: an atom read "before" the seed
was entered at neu so a plain less_equal excluded ties. With Cut carrying it directly the
encoding is redundant — callers were passing a cut and relying on alt/neu to do the real work.

Retiring it removes a scope. Callers ran inside scope.scoped::<AltNeu<_>> and maintained an
"old" and a "new" copy of every arrangement, since alt and neu are separate delayed
collections. On plain usize times the triangle query in delta_query_wcoj goes from four
CollectionIndexes to two, and lookup_map_regression likewise. calculus
(Differentiate/Integrate) is defined over AltNeu by construction, so it goes too.

Behaviour changes worth a look

• validate multiplies by the matched diff instead of passing the prefix diff through. An
  atom contributes to the product whether it proposed an extension or merely validated it. For
  set-valued relations the matched diff is one, so nothing observable moves.
• lookup_map prunes only when the cut admits nothing, never when the admitted diffs sum to
  zero. Those updates can carry different join times, so their sum is not the count at any one
  of them; dropping on it removes the request from the collection outright, so no atom proposes
  it and the extension is lost. An inaccurate non-zero count merely picks a worse proposer.
• CollectionIndex holds local Arranged values rather than trace handles. Re-importing an
  arrangement the same dataflow produced forfeits the scope's progress tracking; inside a
  recursive scope the timestamp then counts upward without end. Documented on
  TraceAgent::import — the only change outside dogsdogsdogs/, and it stands alone if you'd
  rather it were separate.
• extend_using gains two parameters (the cut and its compaction bound).

Tests

11, every case at one and four workers.

• delta_join_property, wcoj_triangle_property — two- and three-atom joins against
  brute-force oracles over random Product<usize, usize> updates. The three-atom one caught a
  real bug during development: lookup sorted its ready prefix by payload, but seek_key only
  advances, so a key selector extracting anything other than a payload prefix could send the
  cursor backwards and silently see no match.
• half_join_total_order — exactly-once pinned from all three sides, on incomparable and on
  equal times.
• ktruss_iterative — a worst-case-optimal join inside a recursive scope, maintained
  incrementally across insertions and deletions. This caught the zero-count prune above.
• lookup_map_regression — ported off AltNeu; it asserts on its output, so it is the evidence
  that derived cuts and plain timestamps find the same triangles the alt/neu encoding did.

Deliberately left

count still reports distinct-extension cardinality. That tracks changes and no longer reports
a false zero, but it measures the wrong thing — the work is the post-consolidation updates a
walk encounters, and reporting that needs a per-key update-count hook on the cursor that does
not exist. Left alone rather than swapped for a proxy that saturates.

There is no tactic trait here yet; lookup takes a closure-based probe. The natural boundary
(cursor at a key, request's time in, admitted updates out) would own the admits test, the
release gate and the compaction bound together, and is where a columnar or int-proxy backend
would plug in.

Diffstat

22 files changed, 1876 insertions(+), 1223 deletions(-)

dogsdogsdogs/src          903 insertions, 1033 deletions   (net -130)
dogsdogsdogs/tests        838 insertions,    39 deletions
dogsdogsdogs/examples     117 insertions,   151 deletions
differential-dataflow      18 insertions                   (import doc)

Source shrinks modestly; the additions are tests and doc comments — a little over half of the
new lookup.rs is documentation.