Skip to content

Add provider error taxonomy, model capabilities gating, and telemetry fixes - #350

Merged
TonsOfFun merged 8 commits into
mainfrom
claude/analytics-api-key-encryption-9kwpsc
Aug 1, 2026
Merged

Add provider error taxonomy, model capabilities gating, and telemetry fixes#350
TonsOfFun merged 8 commits into
mainfrom
claude/analytics-api-key-encryption-9kwpsc

Conversation

@TonsOfFun

Copy link
Copy Markdown
Contributor

Summary

This PR implements three major framework improvements extracted from the ActiveAgents platform: a portable provider error taxonomy for cross-vendor exception handling, per-model capability gating to strip unsupported parameters before API calls, and fixes to telemetry instrumentation and redaction.

Key Changes

1. Provider Error Taxonomy (lib/active_agent/providers/errors.rb)

  • Introduces ActiveAgent::Providers::Errors module with normalized exception classes: RateLimited, ContextLengthExceeded, AuthenticationFailed, ContentFiltered, ServiceUnavailable, and InvalidRequest
  • Implements Taxonomy.normalize() that classifies vendor SDK exceptions by:
    • Demodulized class name (e.g., RateLimitErrorRateLimited)
    • HTTP status code mapping (429 → RateLimited, 401/403 → AuthenticationFailed, etc.)
    • Message pattern matching for context overflow and content filter errors
  • Preserves original exception as #cause for debugging
  • Enables portable rescue_from policies across different LLM providers
  • Integrated into with_exception_handling in the exception handler concern

2. Model Capabilities Gating (lib/active_agent/model_capabilities.rb)

  • Adds ActiveAgent::ModelCapabilities registry to strip unsupported parameters before API calls
  • Built-in rules for thinking-first Claude models (Opus 4.7+, Opus 5, Sonnet 5) and OpenAI reasoning models (o-series, GPT-5) that reject sampling parameters
  • Extensible via ModelCapabilities.register(pattern, unsupported:) for custom/self-hosted models
  • sanitize! method removes rejected parameters in-place and reports what was removed
  • Can be disabled globally with ModelCapabilities.enabled = false
  • Integrated into prepare_prompt_parameters in ActiveAgent::Base

3. Telemetry Improvements

  • Redaction: Implements redact_spans in Tracer#build_trace_payload to scrub span and span-event attributes matching configured patterns before transmission/storage
    • Uses case-insensitive substring matching on attribute keys
    • Controlled by configuration.redact_attributes (defaults to common sensitive keys)
    • Applied to both span attributes and nested event attributes
  • Instrumentation fixes:
    • Removes dead around_generate hook registration that was never called
    • Fixes provider_name and model_name attribute recording (now always set, not conditional on respond_to?)
    • Fixes messages.count recording by reading from prompt_options[:messages] instead of nonexistent messages method
    • Adds llm.provider and llm.model attributes to LLM spans

4. Tool-Loop Safety

  • Adds max_tool_turns (default 25) to bound tool-calling recursion in BaseProvider
  • Tracks tool_turns counter and stops processing when limit is reached
  • Emits tool_turns_exceeded.active_agent notification when cap is hit
  • Allows per-agent/prompt override via max_tool_turns: parameter

5. Documentation

  • Adds docs/framework/v2-extraction-roadmap.md documenting the three-layer architecture and what belongs in each layer
  • Clarifies which platform features are being absorbed into the framework vs. staying in the platform

Testing

Comprehensive test coverage added:

  • test/providers/errors_taxonomy_test.rb — classification by name, status, message patterns; vendor exception wrapping
  • test/model_capabilities_test.rb — parameter stripping, custom rules, sampling support detection
  • test/providers/base_provider_tool_turns_test.rb — recursion bounding and notification emission
  • test/dashboard/telemetry_redaction_test.rb — attribute scrubbing in spans and events
  • test/dashboard/telemetry_correlation_test.rb — provider/model attribute recording in instrumented generations

Notable Implementation

https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP

claude added 8 commits August 1, 2026 15:47
Audit of the ActiveAgents platform's app-level code that turned out to be
framework-shaped: per-model capability gating, provider model catalogs,
tool-loop turn/budget limits, agent-to-agent delegation, provider error
taxonomy + fallback chains, a real MCP client/server story, the tool DSL,
and server-side tool implementations. Also records the dead seams and
instrumentation bugs found during the audit (generation_provider /
around_generate / messages in telemetry instrumentation, the vestigial
Observers seam, unimplemented telemetry redaction, orphaned dashboard
models) and what deliberately stays platform-side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Instrumentation called self.class.generation_provider — a method that
doesn't exist (the accessor is prompt_provider_klass) — so agent.provider
and llm.provider were always rescued to "unknown", and the
respond_to?(:provider_name) guards were false anyway because the helpers
are private, so the attributes were never set at all. Same for
messages.count, which read a nonexistent Base#messages instead of
prompt_options[:messages]. Also removes the permanently-dead
around_generate registration (the macro is around_generation, and no
trace_generation method exists).

Root and llm spans now carry provider tag_name + model; the prompt span
carries the message count.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Thinking-first models reject sampling parameters with an API 400 —
Anthropic's Opus 4.7+/Opus 5/Sonnet 5/Fable 5/Mythos 5 and OpenAI's
o-series/GPT-5 family. ActiveAgent::ModelCapabilities strips the
parameters a model rejects from the prepared prompt parameters before
they reach the provider, so an agent configured with a shared
temperature keeps working across model switches instead of erroring.

Registry is extensible (ModelCapabilities.register(pattern,
unsupported:)) with app rules consulted ahead of the built-ins, and can
be disabled outright (ModelCapabilities.enabled = false) to defer to
vendor enforcement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
process_prompt_finished re-entered resolve_prompt with no cap — a model
that keeps emitting tool calls recursed until the provider stopped it.
Each generation now counts tool round-trips against max_tool_turns
(default 25, configurable per agent/prompt via generate_with/prompt
options). Hitting the cap emits a tool_turns_exceeded.active_agent
notification and finishes cleanly with the messages gathered so far
instead of raising or recursing on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Vendor SDKs raise their own classes for the same failures, so
retry/backoff/fallback policy couldn't be written portably. API-shaped
exceptions are now normalized in with_exception_handling into a small
taxonomy — RateLimited, ContextLengthExceeded, AuthenticationFailed,
ContentFiltered, ServiceUnavailable, InvalidRequest — classified by
vendor class name, HTTP status, and message heuristics (context
overflow and content-filter 400s classify specifically). The original
exception is preserved as #cause and the typed error carries status +
provider tag; ordinary Ruby errors pass through untouched. rescue_from
in agents now works identically across providers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
redact_attributes was documented configuration the tracer never
consumed. Span and span-event attribute keys matching any configured
pattern (case-insensitive substring; defaults cover password/secret/
token/key/credential/api_key) now have their values replaced with
[REDACTED] in build_trace_payload — the shared choke point for both
transmission and local storage — so secrets can't leave the process in
traces. Matching is deliberately over-broad: better a redacted
max_tokens than a shipped api_key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
Capability gating, tool-loop turn cap, error taxonomy, telemetry
redaction, and the instrumentation fixes are implemented; agent_runs
landed in solid_agent. Remaining v2 scope updated accordingly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
The gpt-5 / claude-sonnet-5 preview fixtures configure temperature: 0.7
— a combination those models reject with a 400 at the real API.
ModelCapabilities now strips it before the request is built, so the
preview (which renders the request that will actually be sent) no
longer shows it. Assert the strip instead of the stale passthrough.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP
@TonsOfFun
TonsOfFun marked this pull request as ready for review August 1, 2026 22:24
@TonsOfFun
TonsOfFun merged commit 36ced48 into main Aug 1, 2026
6 checks passed
@TonsOfFun
TonsOfFun deleted the claude/analytics-api-key-encryption-9kwpsc branch August 1, 2026 22:24
TonsOfFun added a commit that referenced this pull request Aug 1, 2026
…ent capture and tool spans

Resolves the overlap with #350 in favor of the richer branch
instrumentation (content attributes, timed tool spans via
tools_function, served-model stamping) while keeping #350's
messages.count fix, provider tag_name attribution, dead-hook removal,
error taxonomy, model capabilities gating, and max_tool_turns. #350's
payload-time redaction is ported to the shared-core report path
(Tracer#redact_trace!) since the in-repo payload builder no longer
exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants