perf: use the latest multi-head critical version as the replay base - #1084
Open
rexikan wants to merge 7 commits into
Open
perf: use the latest multi-head critical version as the replay base#1084rexikan wants to merge 7 commits into
rexikan wants to merge 7 commits into
Conversation
…uplicate A persisted RichtextDiffCalculator (LoroDoc::checkout reuses one) replays ops its tracker has already applied whenever the tracker was rebuilt past `from` — a retreat rebuilds it over the whole history, so the forward walk that follows re-replays every op. The tracker skips those inserts (`skip_applied`), but the StyleStart arm pushed a fresh entry into the style table unconditionally, so every re-replayed mark added an unreferenced duplicate: 10 entries for 5 marks after one retreat and a forward walk. StyleEnd already looks its entry up before pushing. Gate the push on the tracker's applied version and reuse the existing entry. All entries for one op id are content-equal (seeded entries only cover ops below the shallow root, which are trimmed from the oplog and never replayed), and StyleStart is a single atom, so applied-vv membership is exact.
…llback find_common_ancestor conflated two decisions: computing the meet (with diff mode), and retreating to a safe replay base when the meet is not a valid one. Split it into find_meet_and_mode, which returns MeetAsBase::NeedsCriticalRetreat instead of resolving the retreat itself, plus a crate-visible latest_single_head_critical_version, and recombine them in find_common_ancestor so every existing caller keeps byte-identical behavior. The nested helpers the fallback shares with the main walk (ids_to_ord_id_spans, deps_to_ord_id_spans) move to module scope. This lets a caller that can find a better conservative base than the single-head descent (which walks the whole DAG) try its own first. No behavior change; the four common_ancestor_valid_against_slow_oracle_* random-oracle tests and the directed regression tests pin it.
…he single-head one When the meet of two versions is not a valid replay base, find_common_ancestor retreats to the latest *single-head* critical version. On the criss-cross DAG that continuous two-peer sync produces, every synced version has two heads, so that descent stays stuck at the initial fork point forever: the replay region for every concurrent import grows with the whole session, and the causal replay in calc_diff_internal degrades to O(total history) per import. Add OpLog::latest_critical_version_below_meet: a fixpoint that starts at the meet min(from, to) and lowers it until, for every change of the region above it, the context of the change's first op above the candidate covers the candidate. The context is the one the replay itself hands the diff calculators — the change's recorded deps plus the author's own earlier ops — so the certified property is exactly the one a calculator trusting the base consumes. Each lowering preserves every critical cut below the candidate, so the fixpoint is the greatest critical version <= the meet (spec L13); a candidate that bottoms out at the empty version proves the single-head descent cannot do better, so that whole-DAG walk is skipped. Lowering cannot change a verdict already reached, it only exposes the spans between the old and the new candidate, which go on a worklist; the scan is bounded by the replay it enables, which walks the same region and computes the same contexts, so it needs no budget of its own. The caller replays only from the version whose criticality was proved: a shallow doc cannot replay from below its seed version, and a cut assembled from recorded deps plus the author's own prefix is causally closed only when those deps cover that prefix (spec axiom A6, which locally created changes satisfy and imported data is not checked against), so a cut below the seed or one that does not round-trip through frontiers is left to the single-head descent, as before. Wire it into iter_from_replay_base_causally ahead of the descent, which now returns a ReplayBase struct carrying an is_critical flag that downstream calculators can rely on. Nothing relaxes on it yet; a debug assertion in the replay loop checks every replayed op's context against a base claimed critical. This alone is a 2x win: it shrinks the causal replay region that feeds all diff calculators, but richtext/list still rebuild their trackers from empty on every conservative diff. The order-of-magnitude change comes from the follow-up commits that let them trust a critical base.
…e tracker RichtextDiffCalculator rebuilt its CRDT tracker by replaying the entire container history whenever the replay base differed from `from` — which is every concurrent import. With the previous commit the base is usually the last fully synced version and certified critical, so the rebuild's reason evaporates: the tracker's opaque unknown span (everything below the base) is sound exactly when no replayed op is concurrent with anything below the base, which is the definition of a critical version (spec D8b). Every Fugue-concurrent pair is then inside the replayed region, and the alive-set of below-base content is identical at every checkout the replay performs, so author-frame positions resolve to the same offsets as in a full rebuild. Relax should_rebuild accordingly: rebuild on retreat (which stays load-bearing — a retreat is the only way a diff could emit the opaque span, now pinned by a debug assertion in the diff loop), on shallow docs, and on untrusted bases; trust a certified-critical base on a non-shallow doc (source_not_in_op_context is subsumed: it flags replayed ops whose context misses the source state, which criticality rules out below the base and the tracker handles above it). An empty base is trivially critical, which also removes a pure-waste double replay: the calc loop had already fed this tracker the full history before the old code rebuilt an identical tracker from empty. A critical cut is counter-granular and can land between the StyleStart and StyleEnd ops of a single change; that reaches the pre-existing unpaired-end recovery path (style_for_end_anchor), which reconstructs the StyleOp from the oplog with the same end position the full rebuild uses. Pinned by tests that also compare the trusted path against a forced full rebuild (checkout round-trip through the empty frontiers) rather than only against itself, and a test-only rebuild counter proving concurrent imports stop rebuilding (0 rebuilds across 40 criss-cross rounds and across a randomized 600-step three-peer workload; previously 2 per round). Trusting the base also means a persisted calculator (LoroDoc::checkout reuses one) re-replays already-applied ops on every forward step, not only after a retreat; two persisted-walk tests pin that the style table stays at one entry per mark through that, including across the StyleEnd-fallback and after-rebuild reuse paths.
Same relaxation as the previous commit, with less machinery: list trackers are plain RichtextTrackers with no style anchors, so the only list-specific concern is shallow docs. Unlike richtext, list trackers have no shallow-root seeding, so the new critical arm requires a non-shallow doc; the pre-existing `base == from` arm keeps its exact old behavior (it never had a shallow term). MovableList keeps its rebuild condition un-relaxed: its element phase reads positions from the freshly rebuilt tracker, and trusting an unknown-span tracker there needs its own argument about element/position state — deliberately out of scope, now documented at the site.
Black-box tests in crates/loro pinning the whole change rather than any one commit: two-peer ping-pong and three-peer asymmetric criss-cross convergence against fresh replicas, checkout/retreat through concurrent history, concurrent styles, mixed and concurrently-created containers, detached imports, and cursor resolution on a deleted target (which now runs against a non-rebuilt tracker; safe because the replayed delete stamps the target's real id onto the opaque span). perf_concurrent_import.rs holds the two measured scenarios (timings printed, never asserted): the live ping-pong session whose per-import cost used to grow without bound (round-399 import ~71 ms -> 0.11 ms, session 13.7 s -> 58 ms, one-way control unchanged), and the single-concurrent-edit-after-long-history shape that shows both the one-time O(divergence) cost at an old fork and the flat cost after a fresh sync point (~46 ms -> 0.06 ms).
The spec's §4.1 listed the single-head-only fallback as a known trade-off against Eg-walker's V_crit; that gap is now closed by the bounded multi-head fixpoint. Document the two-tier search, add lemma L13 (soundness, greatest-cut preservation, termination) and axiom A6 (deps are the author's complete frontier — load-bearing for maximality only), update Q7's tree-window argument to name both fallbacks and the new base-later-than-window direction, add Q10 for the budget constants, and refresh the glossary and the AGENTS.md working rule. Changeset for a loro-crdt patch release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repeated concurrent imports — the shape produced by any two devices editing
and syncing continuously — went quadratic: the replay base for every import
stayed pinned at the initial fork point, so the richtext/list diff calculators
replayed (and rebuilt their trackers from) the whole accumulated history each
time. At round 400 of a realistic ping-pong session one import costs ~68 ms
and the session has spent ~12.5 s in imports.
This PR computes the latest multi-head critical version as the replay
base, and lets the text and list calculators trust a certified-critical base
instead of rebuilding. Round-399 import drops to 0.11 ms; the session to
55 ms. Stateless: no cache, no lifecycle, no rollback interaction.
Cause
docs/critical-version-spec.md§4.1 already documents the gap: theconservative fallback from #1058 finds only single-head critical cuts.
Continuous two-peer sync produces a criss-cross DAG in which every synced
version has two heads, so the descent walks past all of them to genesis and
returns ∅ — forever. Each concurrent import then replays the entire session,
and
RichtextDiffCalculator/ListDiffCalculator, seeingreplay_base != from, rebuild their CRDT trackers from empty on top of that.Changes (one commit each)
fix(richtext)— pre-existing: a persistedRichtextDiffCalculator(
LoroDoc::checkoutreuses one) pushed a duplicatestylesentry forevery re-replayed StyleStart (10 entries for 5 marks after one retreat and
a forward walk, on plain
main). Trusting the base below would make thatgrow on every step, so it is fixed first, on its own.
refactor(dag)— no behavior change:find_common_ancestorsplitinto
find_meet_and_mode(returnsMeetAsBase::NeedsCriticalRetreatinstead of resolving the retreat) plus a crate-visible
latest_single_head_critical_version.perf(oplog)—OpLog::latest_critical_version_below_meet: afixpoint from the meet downward, lowering it until every change of the
region above it is causally after all of it — "causally after" measured
in the exact per-op context the replay hands the calculators (recorded
deps plus the author's own earlier ops). Each lowering preserves every
critical cut below the candidate, so the fixpoint is the greatest
critical version ≤ the meet (spec L13); a candidate that bottoms out at ∅
proves the single-head descent cannot do better, so that whole-DAG walk
is skipped. Lowering only exposes new spans (worklist), and the scan is
bounded by the replay it enables, which walks the same region and
computes the same contexts — no budget. The result is a
ReplayBase { vv, diff_mode, is_critical }.perf(richtext)—should_rebuildbecomeshas_retreat || !base_is_trustworthy || shallow; a certified-criticalbase is trustworthy. The tracker's opaque below-seed span can never be
retreated and has no interior ids; that is sound exactly when no replayed
op is concurrent with anything below the base. A cut between the
StyleStart/StyleEnd of one change reaches the pre-existing
style_for_end_anchorrecovery, which resolves the position a fullrebuild would.
perf(list)— the same relaxation; the new arm requires anon-shallow doc, the old
base == fromarm is untouched, MovableListkeeps its rebuild (documented at the site).
test— black-box convergence probes and a perf guard (timingsprinted, never asserted).
docs— spec draft v3: L13, axiom A6, Q7/Q10; changesets.Diff modes,
origin_diff_mode,changed_containersfiltering, the treecalculator's lamport windows, and event emission are untouched; the tree
calculator keeps computing its own (single-head) window base, so its cost is
unchanged. Nothing public changes.
Measured
crates/loro/tests/perf_concurrent_import.rs(Apple M-series,--release,-- --ignored --nocapture):Commits 2–3 alone (better base, still rebuilding) halve the constant but
leave the per-import cost linear in session length; commits 4–5 make it flat.
Trade-offs
O(divergence) per import until the exchange completes: there the fork
point genuinely is the latest critical version, and no stateless choice
of replay base can do better (spec Q10).
where the alternative was a whole-DAG walk plus a full-history replay.
own ancestry. Skipping the single-head descent on an empty fixpoint, and
the cut being a causally closed version, rely on a change's recorded
depscovering the author's own earlier ops (spec axiom A6): locallycreated changes satisfy it, imported data is not checked, and a cut that
fails to round-trip through frontiers is left to the descent as before.
Validation
cargo test -p loro-internal/-p loro/-p fuzzgreen (debug; in--releasethe pre-existingop_count.rslock-order test fails onmainas well); every commit builds and passes the loro-internal lib suite on its
own; no new clippy or rustfmt findings on touched code;
Cargo.lockuntouched.
src/tests/replay_base.rs: greatest-cut fixpoint on criss-cross sync (thesingle-head descent shown stuck at the fork), disjoint histories, a base
cutting between the two ops of one mark converging in every import order,
a randomized 600-step 3-peer workload asserting zero fallbacks and zero
rebuilds, persisted checkout walks pinning the exact style-table size, and
the trusted incremental result compared against a forced full rebuild.
crates/loro/tests/concurrent_import.rs(13): ping-pong and 3-peercriss-cross vs fresh replicas, checkout/retreat through concurrent
history, concurrent styles, mixed containers, detached imports, cursor
resolution on a deleted target.
against a base claimed critical; the non-rebuild diff loop asserts the
opaque span never leaks into output.
against every change of its region across all suites, and every
incremental richtext diff byte-compared against a from-empty rebuild —
zero mismatches.
Disclosure
This change was produced with substantial AI assistance (Claude): the
profiling, root-cause analysis, the spec lemma, the patch, tests, and the
review rounds (three independent agent reviews of the first version, which
led to this rewrite, then three more of this one, plus differential audits
against the previous algorithm) were AI-driven, directed and reviewed by me.