Repository navigation
fix(zombie): cap the zombie-detection threshold - #71
Merged
Merged
Conversation
ExpireAt was an absolute deadline computed in the caller. It becomes
{CalledAt, Timeout}, so the worker can read the request timeout
itself. The zombie-detection threshold needs that value to clamp
detection time for long timeouts.
deadline/1, timeout/1 and is_expired/2 interpret the tuple. The sites
that only pass ExpireAt through are unchanged. No behaviour change.
Zombie detection fired at request timeout + max_inactive after the
newest send, so it scaled with the caller's timeout. With a 600 s
request timeout a dead connection was reconnected only after 610 s,
and lowering the timeout would break the long requests it was chosen
for.
The threshold is now
max(MaxInactive, min(MaxInactiveCap, MaxTimeout + MaxInactive))
MaxTimeout is the largest request timeout among the in-flight
requests. MaxInactiveCap defaults to 60 s and is a new pool option,
max_inactive_cap. For timeouts up to 50 s with the default
max_inactive the result is unchanged. A raised max_inactive stays
authoritative through the outer max.
The reference timestamp is now the latest send time (max_sent_at),
not the latest deadline. The timeout is added by the threshold, so a
deadline reference would count it twice. The send time is taken with
now_() in put_sent_req/3, when the request goes to gun, rather than
the caller's call time. The two differ by the time spent in the
pending queue, and the send time is what "silent since the last send"
means. It also needs no special case for infinity.
max_expire/2 skipped infinity deadlines, so a pool whose requests all
used an infinity timeout never recorded one and zombie detection never
fired. infinity now wins the timeout maximum, and the threshold for it
is the cap.
max_sent_expire is renamed to max_sent_at, and max_sent_timeout is
tracked next to it. Both reset to 0 when no request is in flight.
The force_reconnecting_zombie_http_connection log reports the computed
threshold, the latest send time and the largest request timeout.
Incompatible case: with a request timeout above the cap minus
max_inactive, a response that takes longer than the cap now gets its
connection killed at the cap. Received data does not count as
activity. Raise max_inactive to the slowest expected response.
Refs #70
zmstone
marked this pull request as ready for review
September 25, 2026 20:24
thalesmg
reviewed
Sep 25, 2026
| reset_sent(R). | ||
|
|
||
| reset_sent(Requests) -> | ||
| Requests#{sent => #{}, max_sent_at => 0, max_sent_timeout => 0}. |
Contributor
There was a problem hiding this comment.
Suggested change
| Requests#{sent => #{}, max_sent_at => 0, max_sent_timeout => 0}. | |
| Requests#{sent := #{}, max_sent_at := 0, max_sent_timeout := 0}. |
| detection entirely. | ||
| - Behaviour change: with a request timeout above about 50 seconds and the default | ||
| `max_inactive`, a connection whose response takes longer than 60 seconds is now killed at | ||
| 60 seconds. Set the `max_inactive` pool option to the slowest expected response time to |
The cap is a pool option, so the changelog names it and lists it as a way to raise the bound, next to max_inactive.
hjianbo
approved these changes
Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #70
Why
Zombie-connection detection fires at
request timeout + max_inactiveafter the latest send, so it scales with the caller's timeout. A production deployment runs with a 600 s request timeout. With the default 10 smax_inactive, a dead connection is reconnected 610 s after the send. The operator cannot shorten that without breaking the long requests the 600 s timeout was chosen for.What
The threshold becomes
MaxTimeoutis the largest request timeout among the in-flight requests.MaxInactiveCapdefaults to 60 s. It is a new pool option,max_inactive_cap, next to the existingmax_inactiveoption, so tests can exercise the cap in seconds.max_inactiveinfinityThe outer
maxkeeps a raisedmax_inactiveauthoritative. Whenmax_inactive >= 60s, the threshold is exactlymax_inactive.Commits
refactor:ExpireAtbecomes{CalledAt, Timeout}instead of a precomputed deadline, so the worker can read the request timeout.deadline/1,timeout/1andis_expired/2interpret it. The pass-through sites are unchanged. No behaviour change.fix(zombie): the threshold change, themax_inactive_capoption, tests, version bump to 0.7.6 and changelog.Reference timestamp
The old code stored the latest deadline (
Now + Timeout) and comparednow - MaxDeadline > MaxInactive. The new threshold addsMaxTimeoutitself, so the reference must be a send-side timestamp. Otherwise the timeout is counted twice.max_sent_expireis renamed tomax_sent_at, andmax_sent_timeoutsits next to it. Both reset to 0 whensentempties.max_sent_atisnow_()taken input_sent_req/3, when the request goes to gun, not the caller's call time. The two differ by the time the request spent in the pending queue. The send time answers "how long since we sent something and heard nothing", and it needs no special case forinfinity. With a uniform timeout under the cap, the new rule fires at the same point as the old one, apart from that queue time.The
infinityholemax_expire/2skippedinfinitydeadlines. A pool whose requests all usedTimeout = infinitynever recorded a deadline, so zombie detection never fired.infinitynow wins the timeout maximum, and the threshold for it is the cap.Incompatible case
With a request timeout above
cap - max_inactive(50 s by default), a response that legitimately takes longer than the cap now has its connection killed at the cap. Received data does not count as activity: the reference timestamp is written only at send time, so a response that streams for 90 s looks the same as a jammed connection. Such deployments must raisemax_inactive_capormax_inactiveto the slowest expected response. The changelog states this. Counting received data as activity is a possible follow-up in #70 and is out of scope here.Log
force_reconnecting_zombie_http_connectionnow reportsinactive_duration_thresholdas the computed threshold,last_request_sent_atinstead oflast_request_expire, andmax_request_timeout.Tests
inactive_threshold_test_(insrc/ehttpc.erl): table test of the threshold function, covering every row above,infinity, and themax_inactive >= capfloor.zombie_detect_long_timeout_test:max_inactive = 1s,max_inactive_cap = 2s, request timeout 30 s, server never answers.reconnectfires between 2 s and 3.5 s.zombie_detect_infinity_timeout_test: same, with request timeoutinfinity.zombie_detect_inflight_*tests pass with only the field rename.Both new tests fail against the
mainworker:reconnectdoes not arrive within 3.5 s.Run locally on OTP 28:
make fmt-check,./check-style.sh,make compile,make xref,make dialyzer,make eunit.make eunithas 9 failures, all inehttpc_google_tests:proxy_test_, becausetinyproxyis not installed on the host.mainhas the same 9 failures. The first commit alone passesrebar3 eunit --module=ehttpc,ehttpc_tests. The proxy tests and OTP 25/26/27 need CI.