Skip to content

Add durable route catalog delta watch#1117

Open
bootjp wants to merge 4 commits into
mainfrom
feature/route-catalog-delta-watch
Open

Add durable route catalog delta watch#1117
bootjp wants to merge 4 commits into
mainfrom
feature/route-catalog-delta-watch

Conversation

@bootjp

@bootjp bootjp commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • persist one validated route-catalog delta for every catalog version and retain a bounded reconnect window
  • publish local and remote mirrors atomically through bounded delta batches with snapshot reset fallback
  • add capability-negotiated gRPC streaming, catalog-group leader re-resolution, and legacy ListRoutes polling
  • emit the same delta from SplitRange in its catalog transaction
  • document the completed focused design

Safety and compatibility

  • retained-history gaps and invalid mutation operations fail closed
  • stream disconnect resumes from the last atomically published version
  • old clients keep using ListRoutes and new clients fall back when the server does not implement the watch protocol
  • the existing local durable watcher remains active as a fallback

Verification

  • go test ./distribution -count=1
  • focused adapter and same-group split tests
  • go test -race ./distribution ./adapter with focused route-catalog coverage
  • go test . -run '^$' -count=1
  • golangci-lint run ./distribution ./adapter . --timeout=5m
  • make -C proto gen
  • git diff --check

Summary by CodeRabbit

  • 新機能

    • カタログ変更を差分(デルタ)として継続配信し、効率的に同期できるようになりました。
    • サーバー切断やリーダー変更後も、自動再接続して同期を継続します。
    • 差分を取得できない場合は、スナップショットから自動的に再同期します。
    • 差分のバッチサイズを設定できるようになりました。
    • 対応していない環境では、従来のポーリング方式へ自動的に切り替わります。
  • 改善

    • カタログ更新時の差分整合性検証を強化し、不正な変更やバージョン不整合による部分反映を防止します。
  • ドキュメント

    • カタログ差分配信の設計と動作仕様を追加しました。

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (18e0a1d):

  • distribution/catalog.go
  • distribution/catalog_delta.go
  • distribution/catalog_delta_test.go
  • distribution/engine.go
  • distribution/grpc_watcher.go
  • distribution/watcher.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f85421e2-4195-4ba5-bf7f-f9f927aecbb0

📥 Commits

Reviewing files that changed from the base of the PR and between 18e0a1d and 2622b96.

⛔ Files ignored due to path filters (1)
  • proto/distribution.pb.go is excluded by !**/*.pb.go
📒 Files selected for processing (9)
  • adapter/distribution_server.go
  • adapter/distribution_server_delta_test.go
  • adapter/distribution_server_test.go
  • distribution/catalog.go
  • distribution/catalog_delta.go
  • distribution/catalog_delta_test.go
  • distribution/grpc_watcher.go
  • main.go
  • proto/distribution.proto
📝 Walkthrough

Walkthrough

カタログ変更を永続デルタとして保存・取得し、Engineへ段階適用する機能を追加した。Distributionサーバーは能力取得とストリーミング監視を提供し、gRPCウォッチャーは再接続、リセット、旧RPCへのフォールバックに対応する。

Changes

カタログデルタの契約と永続化

Layer / File(s) Summary
デルタ契約と永続化
proto/distribution.proto, distribution/catalog_delta.go, distribution/catalog.go, distribution/catalog_delta_test.go, docs/design/...
デルタのprotobuf表現、エンコード・デコード、連続性検証、保持範囲、ChangesSince、カタログ保存時のデルタミューテーション生成を追加した。

Engineとローカル同期

Layer / File(s) Summary
Engineとローカルウォッチャーの差分適用
distribution/engine.go, distribution/watcher.go, distribution/catalog_delta_test.go
Engine.ApplyDeltaとバッチサイズ設定を追加し、ローカルウォッチャーがリセットまたはデルタ列を適用するよう変更した。

Distributionサーバー

Layer / File(s) Summary
Distributionサーバーの監視RPC
adapter/distribution_server.go, adapter/distribution_server_delta_test.go, adapter/distribution_milestone1_e2e_test.go
GetCatalogCapabilitiesWatchCatalogを追加し、リーダー検証、バッチ制限、リセット・デルタ送信、SplitRangeのデルタコミットを実装した。

gRPC統合

Layer / File(s) Summary
gRPCウォッチャーと実行時配線
distribution/grpc_watcher.go, main.go, adapter/distribution_server_delta_test.go
gRPC能力交渉、ストリーム適用、再接続、エンドポイント再解決、旧サーバーへのポーリングフォールバックを追加し、実行時に監視を起動するよう配線した。

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Runtime
  participant GRPCCatalogWatcher
  participant DistributionServer
  participant CatalogStore
  participant Engine
  Runtime->>GRPCCatalogWatcher: リーダー接続を解決
  GRPCCatalogWatcher->>DistributionServer: GetCatalogCapabilities
  GRPCCatalogWatcher->>DistributionServer: WatchCatalog
  DistributionServer->>CatalogStore: ChangesSince
  CatalogStore-->>DistributionServer: ResetまたはDeltas
  DistributionServer-->>GRPCCatalogWatcher: CatalogWatchEvent
  GRPCCatalogWatcher->>Engine: ApplySnapshotまたはApplyDelta
Loading

Possibly related PRs

  • bootjp/elastickv#1084: SplitRangeのカタログミューテーション生成経路とbuildCatalogSplitOpsを変更しており、本変更と同じコード領域に関連する。
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed タイトルは耐久的なルートカタログのデルタ監視追加という主要変更を端的に表しています。

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a versioned route catalog delta and streaming watch protocol to optimize route propagation. It adds a durable delta log, supports atomic delta publication in the route engine, and implements new gRPC endpoints (GetCatalogCapabilities and WatchCatalog) along with a client-side GRPCCatalogWatcher. A potential issue was identified in the watcher's fallback mechanism: when the watch stream is unimplemented, the watcher successfully falls back to snapshot synchronization but still returns an Unimplemented error, which could flood logs with warnings and trigger unnecessary reconnect loops.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread distribution/grpc_watcher.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@distribution/catalog_delta.go`:
- Around line 237-241: DELETE payload validation must reject route information
in both input paths: update the CatalogMutationDelete handling in
distribution/catalog_delta.go (lines 237-241) to reject any non-zero
RouteDescriptor, and update the DELETE protobuf handling in
distribution/grpc_watcher.go (lines 281-289) to reject events when route is set.

In `@distribution/grpc_watcher.go`:
- Around line 120-127: Update the fallback flow around consumeStream and
syncSnapshot so that when the watch returns codes.Unimplemented and snapshot
synchronization succeeds, the method returns nil instead of the original err;
continue returning the joined error when syncSnapshot fails and preserve direct
returns for other watch errors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6484f689-e498-46aa-a4bb-7e69a49f14d6

📥 Commits

Reviewing files that changed from the base of the PR and between 5d81bfa and 18e0a1d.

⛔ Files ignored due to path filters (2)
  • proto/distribution.pb.go is excluded by !**/*.pb.go
  • proto/distribution_grpc.pb.go is excluded by !**/*.pb.go
📒 Files selected for processing (12)
  • adapter/distribution_milestone1_e2e_test.go
  • adapter/distribution_server.go
  • adapter/distribution_server_delta_test.go
  • distribution/catalog.go
  • distribution/catalog_delta.go
  • distribution/catalog_delta_test.go
  • distribution/engine.go
  • distribution/grpc_watcher.go
  • distribution/watcher.go
  • docs/design/2026_07_18_implemented_route_catalog_delta_watch.md
  • main.go
  • proto/distribution.proto

Comment thread distribution/catalog_delta.go
Comment thread distribution/grpc_watcher.go Outdated
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (2e86ebc):

  • distribution/catalog.go
  • distribution/catalog_delta.go
  • distribution/catalog_delta_test.go
  • distribution/engine.go
  • distribution/grpc_watcher.go
  • distribution/watcher.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Current HEAD: 2e86ebc76102f54e9fad4e2f53a268caca85fb3c

Resolved the current-head findings:

  • successful WatchCatalog Unimplemented fallback returns nil after snapshot synchronization; genuine stream/snapshot failures remain errors
  • DELETE delta payloads are rejected for any non-zero Go descriptor (including non-nil empty bounds) and for any present protobuf route
  • merged current main (910a97e0), removing the conflict and inheriting the stable TLA tools pin

Caller audit:

  • runAttempt has one caller, Run; only the false warning path changes
  • delta validation callers include codec encode/decode, builder, Engine apply, and gRPC decode; normal DELETE producers emit zero/no payload

Verification:

  • go test ./distribution ./adapter -run "TestCatalogDelta|TestGRPCCatalogWatcher" -count=1
  • go test ./... -run "^$" -count=1
  • make tla-check

All pass locally. The two new merge commits are signed by bootjp <contact@bootjp.me>.

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e86ebc761

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread adapter/distribution_server.go
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (2622b96):

  • distribution/catalog.go
  • distribution/catalog_delta.go
  • distribution/catalog_delta_test.go
  • distribution/engine.go
  • distribution/grpc_watcher.go
  • distribution/watcher.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

bootjp commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Current HEAD: 2622b962d3459bb4bc471940529f2ed6b6b90877

Resolved the current split-delta timestamp finding. SplitRange now allocates and pins one coordinator commit timestamp before serialization, so route KV records, durable delta upserts, snapshots, and the RPC response expose the same split_at_hlc.

Caller audit: saveSplitResultViaCoordinator has one caller (SplitRange); NextTimestampAfterThrough guarantees a timestamp greater than the catalog read timestamp, and the existing coordinator CommitTS path remains authoritative.

Verification:

  • go test ./adapter -run "TestDistributionServerSplitRange_UsesCoordinatorForCatalogWrites|TestDistributionServerSplitRange_" -count=1
  • make tla-check
  • git diff --check

All pass locally. Commit is signed by bootjp <contact@bootjp.me>.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 2622b962d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant