Tune redis proxy connection capacity by bootjp · Pull Request #1147 · bootjp/elastickv

bootjp · 2026-07-22T18:11:24Z

Author: bootjp

Summary

raise the default ElasticKV redis-proxy backend pool from 4 to 16 so derived secondary write concurrency increases from 2 to 8
raise the Redis adapter per-peer connection cap default from 8 to 64 and update HA compose/docs accordingly
shorten BZPOP secondary replay no-effect retry window from 500ms to 100ms to reduce long ZREM miss worker occupancy while keeping the initial ordering delay

Production observation

192.168.0.64 production redis-proxy currently reports zero async drops and zero backend pool pending requests
ElasticKV ZREM replay metrics show mostly miss results with high cumulative latency, matching the blocking replay retry overhead addressed here

Tests

go test ./proxy ./cmd/redis-proxy -count=1
go test ./proxy -run 'TestDualWriter_Blocking|TestNoEffectReplayRetryLimit|TestDualWriter_GoAsync|TestDefaultElasticKVBackendOptions' -count=1
go test ./cmd/redis-proxy -run 'TestDeriveSecondaryConcurrency|TestValidateSecondaryConcurrency|TestParseRuntimeOptions' -count=1
go test ./adapter -run 'TestRedisPeerLimiter|TestRedisLeaderClient|TestRedisBlockingLeaderClient' -count=1
git diff --check

Note

go test ./proxy ./cmd/redis-proxy ./adapter reached the adapter package timeout after proxy and cmd completed; targeted adapter tests for the touched limiter paths passed.

coderabbitai · 2026-07-22T18:11:34Z

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 23 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 25bf1afb-3164-4c50-91fc-359aa6c50d39

📥 Commits

Reviewing files that changed from the base of the PR and between 5d4c5ca and fa52ca7.

📒 Files selected for processing (6)

adapter/redis_peer_limiter_test.go
cmd/redis-proxy/main_test.go
docs/design/2026_04_24_implemented_workload_isolation.md
docs/redis-proxy-deployment.md
proxy/backend.go
proxy/proxy_test.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gemini-code-assist · 2026-07-22T18:13:08Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30004dfa7d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-22T18:23:00Z

 	// BZPOP load from turning into a long retry backlog.
 	blockingReplayInitialDelay        = 250 * time.Millisecond
-	blockingReplayNoEffectRetryWindow = 500 * time.Millisecond
+	blockingReplayNoEffectRetryWindow = 100 * time.Millisecond


Restore enough BZPOP replay retry budget

When a client does ZADD followed quickly by BZPOPMIN and the ZADD secondary replay is still queued or committing for more than about 300ms, this 100ms window makes noEffectReplayRetryLimit equal 2, so the ZREM replay gives up after only two backoffs and records a permanent miss while the item can later appear on the secondary. The async queues still allow work to remain within the 5s SecondaryTimeout, so this narrows a previously recoverable 300–600ms ordering delay into a divergence for mutating blocking pops.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T18:23:00Z

 const (
 	redisPerPeerLimitEnv             = "ELASTICKV_REDIS_PER_PEER_CONNECTIONS"
-	defaultRedisPerPeerConnectionCap = 8
+	defaultRedisPerPeerConnectionCap = 64


Keep the default peer cap below noisy-client levels

With this default, the per-peer limiter no longer constrains the noisy-client shape this limiter was added for: one host can open 37 heavy-command connections and all are accepted until 64, while the heavy-command worker pool is global (2*GOMAXPROCS) rather than per-peer. On production-sized nodes with fewer than 32 Ps, that single peer can again occupy every heavy-command slot and starve other clients unless operators override ELASTICKV_REDIS_PER_PEER_CONNECTIONS, so the raised cap should be scoped to proxy deployments or kept below the noisy-client range.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T18:23:00Z

 const (
 	defaultPoolSize          = 128
-	defaultElasticKVPoolSize = 4
+	defaultElasticKVPoolSize = 16


Guard proxy default against old per-peer caps

When the redis-proxy image is rolled before ElasticKV nodes that still run the previous 8-connection per-peer default, this new default pool can open 16 leader connections from the same proxy peer. Those older servers reject connections above 8 with ERR max connections per client exceeded, so secondary concurrency above the old cap turns into write errors during a mixed-version rollout unless the server cap is raised first or the proxy default stays within the old limit.

Useful? React with 👍 / 👎.

bootjp · 2026-07-22T18:58:56Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4ce5d038c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-22T19:06:55Z

 const (
 	defaultPoolSize          = 128
-	defaultElasticKVPoolSize = 4
+	defaultElasticKVPoolSize = 8


Keep proxy pool below the server connection cap

When the proxy targets ElasticKV nodes that still use the default per-peer cap of 8, this new default regular pool can grow to all 8 TCP sessions and keep them idle. Pub/Sub uses a separate dedicated go-redis connection rather than this pool, so after normal traffic has filled the pool a later SUBSCRIBE/Pub/Sub shadow connection from the same proxy IP becomes the 9th server-side connection and is rejected with ERR max connections per client exceeded. Please leave headroom under the default cap or require the server cap to be raised before increasing this default.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T19:06:55Z

-  -secondary-write-concurrency 2 \
-  -secondary-script-concurrency 1 \
+  -elastickv-pool-size 8 \
+  -secondary-write-concurrency 8 \


Leave capacity for blocking-command replay

In dual-write/shadow deployments that copy this example, -secondary-write-concurrency now consumes the entire ElasticKV pool. The runtime derivation sets blocking replay concurrency to poolSize - writeConcurrency, and goBlockingReplay returns immediately when that capacity is 0, so mutating blocking commands such as BZPOPMIN/BLPOP will not be replayed to ElasticKV. Please keep the write limit below the pool size or set an explicit -secondary-blocking-replay-concurrency when documenting this migration mode.

Useful? React with 👍 / 👎.

bootjp · 2026-07-22T19:08:55Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41be10a1fd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-22T19:14:52Z

 ### Recommended v1 shape

-**Per-peer-IP connection cap, default `N=8`, env-configurable,
+**Per-peer-IP connection cap, default `N=64`, env-configurable,


Correct the documented per-peer default

The implementation still sets defaultRedisPerPeerConnectionCap to 8, and the new limiter test in this commit also asserts the default is 8, so documenting the implemented v1 shape as default N=64 makes the design doc contradict the actual server behavior. Operators who rely on this line without setting ELASTICKV_REDIS_PER_PEER_CONNECTIONS would still have connections from the same proxy peer rejected after 8 sockets, including a 16-connection proxy pool.

Useful? React with 👍 / 👎.

bootjp · 2026-07-22T19:17:02Z

@codex review

chatgpt-codex-connector · 2026-07-22T19:21:59Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: fa52ca709b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Tune redis proxy connection capacity

30004df

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

Tune redis proxy capacity defaults

e4ce5d0

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

Leave redis proxy pool headroom

41be10a

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

Fix redis peer cap documentation

fa52ca7

Conversation

bootjp commented Jul 22, 2026

Summary

Production observation

Tests

Note

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

gemini-code-assist Bot commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

bootjp commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

bootjp commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

bootjp commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading