diff --git a/docs/design/2026_02_18_partial_hotspot_shard_split.md b/docs/design/2026_02_18_partial_hotspot_shard_split.md index 55ecfe1b0..ee8a0b5b8 100644 --- a/docs/design/2026_02_18_partial_hotspot_shard_split.md +++ b/docs/design/2026_02_18_partial_hotspot_shard_split.md @@ -293,12 +293,13 @@ Add RPCs: 2. Job phases: BACKFILL/FENCE/DELTA/CUTOVER 3. Manual split with target-group relocation -Status: partial. SplitJob catalog/codec, MVCC export/import primitives, staged -visibility, write-fence checks, target readiness, and target-promotion catalog -components exist in the M2 stack, but no production runner advances cross-group -jobs to `DONE` yet. M2 remains open in -[`2026_06_11_partial_hotspot_split_milestone2_migration.md`](2026_06_11_partial_hotspot_split_milestone2_migration.md) -until the cross-group acceptance criteria and Jepsen workload pass. +Status: implemented. The production runner advances durable cross-group jobs +through BACKFILL, FENCE, DELTA_COPY, CUTOVER, CLEANUP, and `DONE`; current-voter +readiness and cleanup barriers protect leadership changes; and the deterministic +Jepsen workload covers the split alongside leader-kill and partition packages. +The completed M2 contract is recorded in +[`2026_06_11_implemented_hotspot_split_milestone2_migration.md`](2026_06_11_implemented_hotspot_split_milestone2_migration.md) +and its final runner/lifecycle slice landed in PR #1096. ### Milestone 3: Automation diff --git a/docs/design/2026_06_11_partial_hotspot_split_milestone2_migration.md b/docs/design/2026_06_11_implemented_hotspot_split_milestone2_migration.md similarity index 99% rename from docs/design/2026_06_11_partial_hotspot_split_milestone2_migration.md rename to docs/design/2026_06_11_implemented_hotspot_split_milestone2_migration.md index dc13a3044..80b7716dd 100644 --- a/docs/design/2026_06_11_partial_hotspot_split_milestone2_migration.md +++ b/docs/design/2026_06_11_implemented_hotspot_split_milestone2_migration.md @@ -1,13 +1,13 @@ # Hotspot Shard Split — Milestone 2: Migration Plane -Status: Partial +Status: Implemented Author: bootjp Date: 2026-06-11 Parent: [2026_02_18_partial_hotspot_shard_split.md](2026_02_18_partial_hotspot_shard_split.md). M1 (control plane) is as-built in [2026_02_18_implemented_hotspot_split_milestone1_pr.md](2026_02_18_implemented_hotspot_split_milestone1_pr.md). -Current implementation status: partial. The SplitJob catalog/codec, versioned route descriptor storage, range export/import primitives, staged-read visibility, write-fence checks, target readiness, and target-promotion catalog pieces have landed or are covered by the active M2 stack. The feature is not implemented end to end until a production runner advances cross-group jobs to `DONE`, `ListRoutes` exposes the moved child on its target group after cutover, and the Jepsen cross-group split workload passes the acceptance criteria in §13. +Current implementation status: implemented. The M2 stack provides the durable SplitJob catalog, versioned route descriptors, resumable range export/import, staged/live visibility, source write and read fences, current-voter readiness and cleanup barriers, constant-time route cutover, incremental target promotion, bounded source/target cleanup, terminal `DONE` history, and the deterministic cross-group Jepsen workload. PR #1096 completed the production runner and lifecycle wiring. Automatic hotspot detection and split scheduling remain M3 scope; the broader route-shuffle and fault campaign remains M4 scope. ## 1. Background @@ -1243,7 +1243,7 @@ Phased into reviewable PRs, each lands behind its own doc-or-test gate: | M2-PR5 | Coordinator + FSM FENCE rejection (`ErrRouteWriteFenced`) with point and `DEL_PREFIX` range-footprint checks + route-faithful txn-lock drain (§3.2a.0a) + same-group `SplitRange` overlap rejection while a SplitJob is live | fsm + coordinator unit + `migrator_lock_drain_test` + `catalog_test` overlap red controls | | M2-PR6 | Cross-group end-to-end: ExportRangeVersions / ImportVersions server-side handlers + migrator BACKFILL/DELTA_COPY + raw-candidate staged/live merge read path in both scan directions and `LatestCommitTS` + §7.2.2e source-side cutover read-fence arm with every-current-source-voter ACK, membership-epoch re-ACK, catalog-version waiter, route-key-normalized scan ownership checks, and server-stamped RawKV read versions | integration incl. delete/TTL DELTA_COPY, Redis list/hash/set/zset/stream migration, HLC restart/fence-floor cases + `kv/fsm_cutover_read_fence_test.go` | | M2-PR7 | Target-local `PromotionState` + background promoter + ordered default-group promotion-complete CAS retaining `min_write_ts_exclusive` + source/target cleanup before DONE history move; readiness guard accepts matching cleared descriptors while retained; `AbandonSplitJob` with durable `ABANDONING` cleanup witness + CLEANUP GC + Jepsen split workload | `kv/fsm_promote_staged_test.go` raw hidden-version/LatestCommitTS merge + jepsen suite | -| M2-PR8 | Rename `*_proposed_*` → `*_partial_*` after PR1 ships; update parent partial doc M2 status; rename to `*_implemented_*` after PR7 | | +| M2-PR8 | Rename `*_proposed_*` → `*_partial_*` after PR1 ships; update parent partial doc M2 status; rename to `*_implemented_*` after PR7 | Completed after the runner, voter barriers, cleanup lifecycle, route publication, and Jepsen workload landed in PR #1096. | Each PR follows the five-lens self-review and is gated by its tests + `make lint`. @@ -1283,9 +1283,12 @@ This is independent of the existing rolling-upgrade protocol for unrelated subsy ## 14. Lifecycle -This document is `*_partial_*` because M2-PR1 and later component slices have landed, but the cross-group migration plane is not complete end to end. +This document is `*_implemented_*` because the M2 cross-group migration plane is complete as a central subsystem. The production runner now resumes durable jobs across leadership changes, drives every phase through `CLEANUP`, waits for current source and target voter proofs, publishes the target route, removes bounded migration state, and moves the job to `DONE` history. The deterministic Jepsen split workload exercises the cross-group operation alongside leader-kill and partition fault packages. -- Track per-PR landing under §11. -- Rename to `*_implemented_*` only after the acceptance criteria in §13 pass: cross-group `StartSplitMigration` reaches `phase=DONE`, `ListRoutes` shows the target-group child after cutover, leader-kill recovery is automatic, and the Jepsen split workload is green. +The remaining work is intentionally outside M2's central scope: + +- M3 owns automatic hotspot detection, target selection, and split scheduling. +- M4 owns the broader route-shuffle nemesis matrix and production-scale fault campaigns. +- Reverse migration after CUTOVER and concurrent migration jobs remain future extensions. `git mv` is used so history follows. diff --git a/docs/design/2026_06_12_proposed_scaling_roadmap.md b/docs/design/2026_06_12_proposed_scaling_roadmap.md index 11936ebd1..6d3fa34ac 100644 --- a/docs/design/2026_06_12_proposed_scaling_roadmap.md +++ b/docs/design/2026_06_12_proposed_scaling_roadmap.md @@ -343,9 +343,9 @@ control-plane (`*_proposed_*` doc TBD).** - M1 standalone but doesn't enable cross-region writes — it just makes Raft survive cross-WAN partition. -- M2 depends on the M2 hotspot-split migration contract - (`2026_06_11_partial_hotspot_split_milestone2_migration.md`) - being implemented so the monotone-merge primitive exists. +- M2's hotspot-split migration dependency is implemented in + [2026_06_11_implemented_hotspot_split_milestone2_migration.md](2026_06_11_implemented_hotspot_split_milestone2_migration.md), including + the monotone-merge primitive. - M3 depends on M1's region-aware membership and M2's per-region ceiling. - M4 depends on M2 and M3. @@ -538,7 +538,7 @@ the ceiling shape: Composability invariant: **every monotone-merge happens via the same `SetPhysicalCeiling` + `Observe` primitive**. The M2 hotspot-split contract (§6.2.1 of -`2026_06_11_partial_hotspot_split_milestone2_migration.md`) is the +[2026_06_11_implemented_hotspot_split_milestone2_migration.md](2026_06_11_implemented_hotspot_split_milestone2_migration.md)) is the reference implementation; per-region and per-group merges reuse it. ### 7.2 Capability bits diff --git a/docs/design/2026_06_23_proposed_scaling_roadmap.md b/docs/design/2026_06_23_proposed_scaling_roadmap.md index 6e2d11628..210d52c14 100644 --- a/docs/design/2026_06_23_proposed_scaling_roadmap.md +++ b/docs/design/2026_06_23_proposed_scaling_roadmap.md @@ -98,10 +98,10 @@ memory each group's private cache/memtable pins. - **Range split — distribute a range across groups.** Same-group split shipped in M1 (`distribution/`). Cross-group migration (the part that - actually relocates data and reduces per-node volume) is tracked by **PR #945** - and the active M2 stack - (`docs/design/2026_06_11_partial_hotspot_split_milestone2_migration.md`, - branch `docs/hotspot-split-m2-proposal`): a resumable `SplitJob` with + actually relocates data and reduces per-node volume) is implemented by the + M2 stack recorded in + [2026_06_11_implemented_hotspot_split_milestone2_migration.md](2026_06_11_implemented_hotspot_split_milestone2_migration.md): + a resumable `SplitJob` with `PLANNED → BACKFILL → FENCE → DELTA_COPY → CUTOVER → CLEANUP → DONE` phases driven by a migrator on the default-group leader. M2 is the required ownership-migration mechanism, but it reduces per-node bytes only when the