Skip to content

perf(decoders): memoize can_decode verdict per model (0.9.2)#44

Merged
lesnik512 merged 1 commit into
mainfrom
perf/can-decode-result-cache
Jun 13, 2026
Merged

perf(decoders): memoize can_decode verdict per model (0.9.2)#44
lesnik512 merged 1 commit into
mainfrom
perf/can-decode-result-cache

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Summary

can_decode runs on every dispatch (the client loops registered decoders calling it until one claims the response_model). After the 0.9.1 nested-CustomType fix, MsgspecDecoder.can_decode cost ~30 µs/call for a type like list[SomeStruct] — an uncached msgspec.inspect.type_info() plus a recursive dir()-based tree walk, repeated every request. PydanticDecoder.can_decode had the milder version: it re-probed TypeAdapter construction each call and never cached a rejection.

Both decoders now memoize the per-model bool verdict in a per-instance dict[type, bool], so the probe runs once per response_model type instead of once per request.

Measured (repeat call):

decoder before after
MsgspecDecoder.can_decode(list[Struct]) ~30.6 µs ~0.15 µs (~200×)
PydanticDecoder.can_decode(BaseModel) re-probed each call ~0.07 µs

Unhashable models skip the cache and probe fresh (preserves the existing fallback). The cache is bounded by the set of response_model types an app actually uses, mirroring the existing per-instance decoder/adapter caches.

No behavior change: can_decode's signature, the ResponseDecoder protocol, and every routing verdict are unchanged — only repeated computation is removed.

Ships as 0.9.2 (release notes in planning/releases/0.9.2.md; tagging is a separate manual step).

Test Plan

  • just lint clean
  • just test — 507 passed, 100% coverage
  • Cache behavior (TDD red→green): repeat can_decode calls hit the cache (probe call_count == 1 across two calls), for both positive and negative verdicts, both decoders
  • Unhashable model does not propagate TypeError from the cache lookup (falls back to fresh probe)
  • All pre-existing decoder/dispatch tests still green

🤖 Generated with Claude Code

can_decode runs on every dispatch. After the 0.9.1 nested-CustomType fix,
MsgspecDecoder.can_decode cost ~30us/call (uncached type_info + recursive
walk); PydanticDecoder re-probed TypeAdapter construction and never cached
rejections. Both now memoize the per-model bool verdict in a per-instance
dict, so the probe runs once per response_model type instead of per request
(~200x faster on repeat calls). Unhashable models skip the cache and probe
fresh. No public-API or routing-behavior change.

Drafts 0.9.2 release notes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lesnik512 lesnik512 merged commit ef9cbb5 into main Jun 13, 2026
5 checks passed
@lesnik512 lesnik512 deleted the perf/can-decode-result-cache branch June 13, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant