Skip to content

Add CircuitBreaker + AsyncTimeout (0.10.0)#51

Merged
lesnik512 merged 16 commits into
mainfrom
feat/circuit-breaker-timeout
Jun 13, 2026
Merged

Add CircuitBreaker + AsyncTimeout (0.10.0)#51
lesnik512 merged 16 commits into
mainfrom
feat/circuit-breaker-timeout

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Summary

Completes the Polly-shape resilience suite by adding the two missing strategies, both pure-stdlib and additive (no new optional extra):

  • CircuitBreaker / AsyncCircuitBreaker — classic consecutive-failure circuit breaker (Polly pre-v8 default). Opens after failure_threshold consecutive counted failures, probes after reset_timeout, closes after success_threshold consecutive half-open successes. A counted failure = NetworkError, httpware TimeoutError, or a StatusError whose status is in failure_status_codes (default: all 5xx). 4xx including 429 count as successes (429 = healthy-but-throttling). The transition logic lives once in a lock-free _CircuitBreakerState; the async wrapper relies on asyncio atomicity + a single-event-loop guard (carried from AsyncBulkhead), the sync wrapper serializes with a threading.Lock.
  • AsyncTimeout — async-only overall wall-clock deadline across the whole inner pipeline (most importantly across an AsyncRetry loop, which httpx2 can't bound). Uses asyncio.timeout(...).expired() to re-wrap only its own deadline as httpware.TimeoutError, leaving inner timeouts untouched. No sync Timeout (sync Python can't cancel a blocking call mid-flight; httpx2 covers sync per-call timeouts).
  • CircuitOpenErrorClientError subclass with retry_after: float | None, raised when the breaker refuses a request.

New observability events (stable surface): circuit.opened, circuit.rejected, circuit.half_open, circuit.closed (logger httpware.circuit_breaker), timeout.exceeded (logger httpware.timeout).

Docs (resilience.md, errors.md, README) document the recommended (not enforced) chain order: AsyncTimeout → AsyncCircuitBreaker → AsyncBulkhead → AsyncRetry → terminalAsyncBulkhead stays outside AsyncRetry (one slot per logical call), consistent with the existing tested guidance.

Spec: planning/specs/2026-06-13-circuit-breaker-and-timeout-design.md · Plan: planning/plans/2026-06-13-circuit-breaker-and-timeout.md · Release notes: planning/releases/0.10.0.md.

Test Plan

  • just lint-ci clean (ruff format/check, ty)
  • just test — 555 passed, 100% coverage (verified stable across 10 runs; property-test coverage made deterministic)
  • Full state machine covered (CLOSED→OPEN→HALF_OPEN→CLOSED, probe re-open, concurrent-probe rejection) in both sync and async
  • Hypothesis property test: while OPEN and pre-reset_timeout, next is never forwarded
  • CircuitOpenError pickle round-trip; cross-event-loop guard; 429/4xx-as-success; custom failure_status_codes
  • No production code paths exercise networking beyond httpx2.MockTransport

🤖 Generated with Claude Code

lesnik512 and others added 16 commits June 13, 2026 12:21
Spec for the two resilience primitives that complete the Polly-shape
suite: classic consecutive-failure CircuitBreaker/AsyncCircuitBreaker
and an async-only AsyncTimeout (overall pipeline deadline).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six-task TDD plan: CircuitOpenError, AsyncTimeout, shared
_CircuitBreakerState + AsyncCircuitBreaker, sync CircuitBreaker,
property test, docs/release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e breaker)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ers on open

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rity

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…syncRetry

The recommended chain wrongly placed AsyncBulkhead inside AsyncRetry,
contradicting the tested [AsyncBulkhead, AsyncRetry] guidance (one slot
per logical call). Correct order: AsyncTimeout -> AsyncCircuitBreaker ->
AsyncBulkhead -> AsyncRetry -> terminal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The accumulate-advances-with-break loop only exercised the `break`
branch when generated advances happened to sum past reset_timeout,
which Hypothesis did not hit on every run — intermittently dropping
coverage to 99.98% and failing --cov-fail-under=100. Map each clock
step to a fraction strictly below reset_timeout instead; no branch,
100% every run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…codes

Requiring frozenset[int] made the natural call site
`failure_status_codes={500, 503}` (a set literal) a ty error. Accept
Collection[int] and freeze internally — membership semantics unchanged,
no friction pushed onto callers. Tests now pass a plain set and a list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lesnik512

Copy link
Copy Markdown
Member Author

Addressed in c52d343.

1. failure_status_codes ergonomics — fixed. Relaxed the param from frozenset[int] to Collection[int] on both wrappers and _CircuitBreakerState, freezing internally (frozenset(failure_status_codes)). Confirmed the prior friction was real: AsyncCircuitBreaker(failure_status_codes={500, 503}) (a plain set literal) was a ty diagnostic before and type-checks clean now. Membership semantics unchanged. The custom-status tests now pass a plain set (async/sync trips_on_member) and a list (excludes_other_5xx) to lock in the generality.

2. _now test seam docstring — leaving as-is. The existing precedent is AsyncRetry._sleep: a leading-underscore seam with no docstring mention (the underscore is the signal). AsyncCircuitBreaker/CircuitBreaker already match that. Adding a note here would diverge from the convention you asked me to mirror, so I'd rather keep it consistent across the resilience modules.

3. Logging under the lock (sync) — leaving as-is, per your note. It matches the Bulkhead approach, and moving the _emit calls outside the critical section would risk reordering events relative to the state transition they describe. Not worth the trade for a handler-latency edge case.

Re-verified: just lint-ci clean, full suite 555 passed at 100% coverage.

@lesnik512 lesnik512 merged commit 2a2b541 into main Jun 13, 2026
5 checks passed
@lesnik512 lesnik512 deleted the feat/circuit-breaker-timeout branch June 13, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant