Add provider error taxonomy, model capabilities gating, and telemetry fixes - #350
Merged
Merged
Conversation
Audit of the ActiveAgents platform's app-level code that turned out to be framework-shaped: per-model capability gating, provider model catalogs, tool-loop turn/budget limits, agent-to-agent delegation, provider error taxonomy + fallback chains, a real MCP client/server story, the tool DSL, and server-side tool implementations. Also records the dead seams and instrumentation bugs found during the audit (generation_provider / around_generate / messages in telemetry instrumentation, the vestigial Observers seam, unimplemented telemetry redaction, orphaned dashboard models) and what deliberately stays platform-side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Instrumentation called self.class.generation_provider — a method that doesn't exist (the accessor is prompt_provider_klass) — so agent.provider and llm.provider were always rescued to "unknown", and the respond_to?(:provider_name) guards were false anyway because the helpers are private, so the attributes were never set at all. Same for messages.count, which read a nonexistent Base#messages instead of prompt_options[:messages]. Also removes the permanently-dead around_generate registration (the macro is around_generation, and no trace_generation method exists). Root and llm spans now carry provider tag_name + model; the prompt span carries the message count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Thinking-first models reject sampling parameters with an API 400 — Anthropic's Opus 4.7+/Opus 5/Sonnet 5/Fable 5/Mythos 5 and OpenAI's o-series/GPT-5 family. ActiveAgent::ModelCapabilities strips the parameters a model rejects from the prepared prompt parameters before they reach the provider, so an agent configured with a shared temperature keeps working across model switches instead of erroring. Registry is extensible (ModelCapabilities.register(pattern, unsupported:)) with app rules consulted ahead of the built-ins, and can be disabled outright (ModelCapabilities.enabled = false) to defer to vendor enforcement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
process_prompt_finished re-entered resolve_prompt with no cap — a model that keeps emitting tool calls recursed until the provider stopped it. Each generation now counts tool round-trips against max_tool_turns (default 25, configurable per agent/prompt via generate_with/prompt options). Hitting the cap emits a tool_turns_exceeded.active_agent notification and finishes cleanly with the messages gathered so far instead of raising or recursing on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Vendor SDKs raise their own classes for the same failures, so retry/backoff/fallback policy couldn't be written portably. API-shaped exceptions are now normalized in with_exception_handling into a small taxonomy — RateLimited, ContextLengthExceeded, AuthenticationFailed, ContentFiltered, ServiceUnavailable, InvalidRequest — classified by vendor class name, HTTP status, and message heuristics (context overflow and content-filter 400s classify specifically). The original exception is preserved as #cause and the typed error carries status + provider tag; ordinary Ruby errors pass through untouched. rescue_from in agents now works identically across providers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
redact_attributes was documented configuration the tracer never consumed. Span and span-event attribute keys matching any configured pattern (case-insensitive substring; defaults cover password/secret/ token/key/credential/api_key) now have their values replaced with [REDACTED] in build_trace_payload — the shared choke point for both transmission and local storage — so secrets can't leave the process in traces. Matching is deliberately over-broad: better a redacted max_tokens than a shipped api_key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Capability gating, tool-loop turn cap, error taxonomy, telemetry redaction, and the instrumentation fixes are implemented; agent_runs landed in solid_agent. Remaining v2 scope updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
The gpt-5 / claude-sonnet-5 preview fixtures configure temperature: 0.7 — a combination those models reject with a 400 at the real API. ModelCapabilities now strips it before the request is built, so the preview (which renders the request that will actually be sent) no longer shows it. Assert the strip instead of the stale passthrough. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
TonsOfFun
marked this pull request as ready for review
August 1, 2026 22:24
TonsOfFun
added a commit
that referenced
this pull request
Aug 1, 2026
…ent capture and tool spans Resolves the overlap with #350 in favor of the richer branch instrumentation (content attributes, timed tool spans via tools_function, served-model stamping) while keeping #350's messages.count fix, provider tag_name attribution, dead-hook removal, error taxonomy, model capabilities gating, and max_tool_turns. #350's payload-time redaction is ported to the shared-core report path (Tracer#redact_trace!) since the in-repo payload builder no longer exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements three major framework improvements extracted from the ActiveAgents platform: a portable provider error taxonomy for cross-vendor exception handling, per-model capability gating to strip unsupported parameters before API calls, and fixes to telemetry instrumentation and redaction.
Key Changes
1. Provider Error Taxonomy (
lib/active_agent/providers/errors.rb)ActiveAgent::Providers::Errorsmodule with normalized exception classes:RateLimited,ContextLengthExceeded,AuthenticationFailed,ContentFiltered,ServiceUnavailable, andInvalidRequestTaxonomy.normalize()that classifies vendor SDK exceptions by:RateLimitError→RateLimited)RateLimited, 401/403 →AuthenticationFailed, etc.)#causefor debuggingrescue_frompolicies across different LLM providerswith_exception_handlingin the exception handler concern2. Model Capabilities Gating (
lib/active_agent/model_capabilities.rb)ActiveAgent::ModelCapabilitiesregistry to strip unsupported parameters before API callsModelCapabilities.register(pattern, unsupported:)for custom/self-hosted modelssanitize!method removes rejected parameters in-place and reports what was removedModelCapabilities.enabled = falseprepare_prompt_parametersinActiveAgent::Base3. Telemetry Improvements
redact_spansinTracer#build_trace_payloadto scrub span and span-event attributes matching configured patterns before transmission/storageconfiguration.redact_attributes(defaults to common sensitive keys)around_generatehook registration that was never calledprovider_nameandmodel_nameattribute recording (now always set, not conditional onrespond_to?)messages.countrecording by reading fromprompt_options[:messages]instead of nonexistentmessagesmethodllm.providerandllm.modelattributes to LLM spans4. Tool-Loop Safety
max_tool_turns(default 25) to bound tool-calling recursion inBaseProvidertool_turnscounter and stops processing when limit is reachedtool_turns_exceeded.active_agentnotification when cap is hitmax_tool_turns:parameter5. Documentation
docs/framework/v2-extraction-roadmap.mddocumenting the three-layer architecture and what belongs in each layerTesting
Comprehensive test coverage added:
test/providers/errors_taxonomy_test.rb— classification by name, status, message patterns; vendor exception wrappingtest/model_capabilities_test.rb— parameter stripping, custom rules, sampling support detectiontest/providers/base_provider_tool_turns_test.rb— recursion bounding and notification emissiontest/dashboard/telemetry_redaction_test.rb— attribute scrubbing in spans and eventstest/dashboard/telemetry_correlation_test.rb— provider/model attribute recording in instrumented generationsNotable Implementation
https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP