perf(decoders): memoize can_decode verdict per model (0.9.2)#44
Merged
Conversation
can_decode runs on every dispatch. After the 0.9.1 nested-CustomType fix, MsgspecDecoder.can_decode cost ~30us/call (uncached type_info + recursive walk); PydanticDecoder re-probed TypeAdapter construction and never cached rejections. Both now memoize the per-model bool verdict in a per-instance dict, so the probe runs once per response_model type instead of per request (~200x faster on repeat calls). Unhashable models skip the cache and probe fresh. No public-API or routing-behavior change. Drafts 0.9.2 release notes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
can_decoderuns on every dispatch (the client loops registered decoders calling it until one claims theresponse_model). After the 0.9.1 nested-CustomTypefix,MsgspecDecoder.can_decodecost ~30 µs/call for a type likelist[SomeStruct]— an uncachedmsgspec.inspect.type_info()plus a recursivedir()-based tree walk, repeated every request.PydanticDecoder.can_decodehad the milder version: it re-probedTypeAdapterconstruction each call and never cached a rejection.Both decoders now memoize the per-model
boolverdict in a per-instancedict[type, bool], so the probe runs once perresponse_modeltype instead of once per request.Measured (repeat call):
MsgspecDecoder.can_decode(list[Struct])PydanticDecoder.can_decode(BaseModel)Unhashable models skip the cache and probe fresh (preserves the existing fallback). The cache is bounded by the set of
response_modeltypes an app actually uses, mirroring the existing per-instance decoder/adapter caches.No behavior change:
can_decode's signature, theResponseDecoderprotocol, and every routing verdict are unchanged — only repeated computation is removed.Ships as 0.9.2 (release notes in
planning/releases/0.9.2.md; tagging is a separate manual step).Test Plan
just lintcleanjust test— 507 passed, 100% coveragecan_decodecalls hit the cache (probe call_count == 1 across two calls), for both positive and negative verdicts, both decodersTypeErrorfrom the cache lookup (falls back to fresh probe)🤖 Generated with Claude Code