Stop Envoy reusing upstream connections to actors#549
Open
Chuang Wang (chuangw6) wants to merge 1 commit into
Open
Stop Envoy reusing upstream connections to actors#549Chuang Wang (chuangw6) wants to merge 1 commit into
Chuang Wang (chuangw6) wants to merge 1 commit into
Conversation
Envoy pools HTTP/1.1 connections by destination address, which is safe when an address means one stable server. For actors it does not. A worker pod keeps its IP for its whole lifetime, but the actor sandbox behind port 80 is torn down on every Suspend and a different actor takes the slot. A pooled connection therefore belongs to an actor that may already be gone, and reusing it races the far end's close: the request goes into a dead socket and Envoy synthesizes a 503. At 50 actors churning across 50 workers this was ~42% of pings failing, with Envoy reporting 450 upstream_cx_destroy_remote_with_active_rq. Setting max_requests_per_connection to 1 took it to 0% and 0. The handshake this costs is sub-millisecond in-cluster against pings that take ~22ms, and it removes the race rather than recovering from it after the fact. Carrying HttpProtocolOptions on a dynamic_forward_proxy cluster also needs allow_insecure_cluster_options. Envoy guards that because a DFP cluster forwards to whatever host the request names, so a TLS upstream must validate against it; this cluster has no transport socket at all and the authority is an IP literal. Without the flag Envoy NACKs every CDS push and drops the cluster, which looks like "all actor traffic 503s" rather than a config error, so there is a test pinning it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Envoy pools HTTP/1.1 connections by destination address. That is safe when an address means one stable server, and for actors it does not. A worker pod keeps its IP for its whole lifetime, but the actor sandbox behind port 80 is torn down on every Suspend and a different actor takes the slot. A pooled connection therefore belongs to an actor that may already be gone, and reusing it races the far end's close.
Symptom
The request goes into a dead socket and Envoy synthesizes a 503. At 50 actors churning across 50 workers, ~42% of pings failed, with Envoy reporting 450
upstream_cx_destroy_remote_with_active_rq.Solution
Set
max_requests_per_connectionto 1 on the dynamic forward proxy cluster. Failures and the reset counter both went to 0.Notes
The handshake this costs is sub-millisecond in-cluster, against pings that take ~22ms. It removes the race rather than recovering from it after the fact, which is why this is preferred over a route-level retry policy.
Carrying
HttpProtocolOptionson adynamic_forward_proxycluster also requiresallow_insecure_cluster_options. Envoy guards that because a DFP cluster forwards to whatever host the request names, so a TLS upstream must validate against it. This cluster has no transport socket at all and the authority is an IP literal, so there is nothing to validate. Without the flag Envoy NACKs every CDS push and drops the cluster, which presents as "all actor traffic 503s" rather than as a config error, so there is a test pinning it.Independent of the benchmarking stack (#542 / #543 / #545); this only touches
cmd/atenet/internal/router.