From bf7613d808914b990c33e5c00f8aeadcd61fc88a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 15:47:47 +0000 Subject: [PATCH 1/8] docs: v2 extraction roadmap from the platform audit Audit of the ActiveAgents platform's app-level code that turned out to be framework-shaped: per-model capability gating, provider model catalogs, tool-loop turn/budget limits, agent-to-agent delegation, provider error taxonomy + fallback chains, a real MCP client/server story, the tool DSL, and server-side tool implementations. Also records the dead seams and instrumentation bugs found during the audit (generation_provider / around_generate / messages in telemetry instrumentation, the vestigial Observers seam, unimplemented telemetry redaction, orphaned dashboard models) and what deliberately stays platform-side. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- docs/framework/v2-extraction-roadmap.md | 116 ++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 docs/framework/v2-extraction-roadmap.md diff --git a/docs/framework/v2-extraction-roadmap.md b/docs/framework/v2-extraction-roadmap.md new file mode 100644 index 00000000..edc75f42 --- /dev/null +++ b/docs/framework/v2-extraction-roadmap.md @@ -0,0 +1,116 @@ +# v2 extraction roadmap — what the platform taught us belongs in the framework + +The ActiveAgents platform (activeagents/activeagents) has been the field lab +for the gems: every gap in `activeagent` or `solid_agent` shows up there as +app-level code. This document is the audit of that code, drawing the line for +the three-layer architecture: + +- **activeagent** — execution: agents, providers, tools, telemetry +- **solid_agent** — persistence: contexts, generations, tool streams, memory, + pricing +- **the platform** — accounts, billing, quotas, hosted UI, multi-tenancy + +The solid_agent side of this audit already landed (enriched tool persistence, +`ModelPricing`, memory/tool-cache contracts). What follows is the +framework-shaped remainder, proposed for v2 — each item exists today as +platform code that any serious consumer of the gem would have to rebuild. + +## Absorb from the platform + +### 1. Per-model capability gating +The platform strips `temperature`/`top_p` for models that reject them +(Claude Opus 4.7+, Sonnet 5, Fable/Mythos 5) via a regex before +`generate_with`. The framework has no capability layer at all — the request +objects apply `DEFAULTS` (`temperature: 1, top_p: 1`) unconditionally and let +the vendor 400. v2: a model-capability table consulted by the Request layer +(sampling params, max_tokens vs max_completion_tokens, reasoning-effort +support), with a config escape hatch for unknown models. + +### 2. Provider model catalogs +The platform's `/api/provider_models` queries Ollama's live model list, the +Anthropic Models API, and OpenRouter's catalog, with curated fallbacks. +v2: `Provider#models` on the provider contract (vendor SDKs all expose a +listing endpoint), so model pickers and validation stop being app problems. + +### 3. Tool-loop safety +`process_prompt_finished` re-enters `resolve_prompt` with **no max-turn cap +and no token/cost budget** — a looping model recurses until the provider +stops emitting tool calls. v2: `max_tool_turns` and a token budget on the +generation, with a clean partial-result return when hit. + +### 4. Agent-to-agent delegation +`tools_function` only routes back to `self`. The platform built `call_agent` +(sub-agent invocation with a `Thread.current` depth cap) as an app tool. +v2: a first-class delegation primitive — invoke another agent class/instance +as a tool, with depth limits and shared trace/context correlation. + +### 5. Provider error taxonomy + fallback +Only `ProvidersError` exists and nothing raises it; rate limits, context +overflows, and content filters surface as vendor-specific exceptions, so no +retry/fallback policy can be written against them. v2: typed errors +(`RateLimited`, `ContextLengthExceeded`, `ContentFiltered`, …) normalized +across providers, then a `generate_with ... fallback: [:anthropic, :ollama]` +chain becomes expressible. + +### 6. A real MCP story +Today MCP is pass-through only: `mcps:` options are normalized into each +vendor's *remote* MCP format (the LLM vendor's servers do the connecting). +There is no MCP client (stdio/HTTP, `tools/list` discovery → routable +actions) and no server facade. The platform built an MCP server over its +agents (`run_` tools, `agent://` resources, Bearer auth) as a +controller. v2: both halves — a client that turns any MCP server's tools +into agent actions, and a mountable engine that presents agents as an MCP +server. + +### 7. Tool DSL / schema derivation +`lib/active_agent.rb`'s docstring advertises a `tool def get_weather(...)` +macro that does not exist; tools are hand-written JSON Schema hashes. +solid_agent's `HasTools` (DSL + JSON view templates) already fills this — +v2 should either absorb it or bless it as the canonical declaration path, +not leave two half-standards. + +### 8. Server-side tool implementations +The platform's `AgentToolbox` (safe `fetch_url` with SSRF guard + redirect +caps, `web_search`, a no-eval `Calculator`, allowlisted `browse_page`) is +generic execution code with zero app coupling. It belongs beside the +framework's tool routing, not in a dashboard app — persistence/caching of +results stays solid_agent's. + +## Fix in place (bugs and dead seams found during the audit) + +- `telemetry/instrumentation.rb` calls `self.class.generation_provider` + (method doesn't exist → `llm.provider` attribute always "unknown"/absent), + registers `around_generate` (macro is `around_generation` → permanently + dead line), and guards on `respond_to?(:messages)` (Base has no + `#messages` → message counts never recorded). +- `Observers`/`Interceptors` call `Prompt.register_observer` on an + `ActiveAgent::Prompt` class that doesn't exist; nothing in the generation + path notifies them. Either implement the ActionMailer-style seam + (persistence layers want it) or delete it. +- Telemetry `capture_bodies`/`redact_attributes` are documented config that + is never consumed, while notification payloads attach full raw responses. + Implement redaction before v2 ships. +- The dashboard engine ships orphaned platform-shaped models + (`Dashboard::Agent`, `AgentRun`, `SandboxSession`, jobs, migrations) with + no controllers or routes. Decide: wire them (framework-level run + orchestration — run records, status, cancellation would pair with the + tool-loop limits above) or drop them from the gem. + +## Stays in the platform + +Accounts/users, billing and plan quotas, encrypted API/provider key storage, +trace retention by plan, the hosted React dashboard, sandboxes/session +recordings, and the account-scoped `TelemetryTrace` subclass. These touch +tenancy and money; the gems should expose seams (auth hooks, quota +callbacks), never implementations. + +## solid_agent follow-ups (tracked there, listed for completeness) + +- `agent_runs` + persisted run progress events (the platform's + `AgentRun#append_event` and instruction-cohort fingerprints are the + proven shape). +- Evaluation datasets — `docs/agent-md-spec.md` already specifies + `*.test.yml` cases; the platform's rule-criteria scorer and LLM-judge + plumbing are the reference implementation. +- Fold the platform's drifted model copies back onto the generator + templates once the app's Gemfile.lock reaches solid_agent 0.2. From 21bac48ece0fa28561312b7fb610853922843506 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 15:55:00 +0000 Subject: [PATCH 2/8] fix(telemetry): record provider/model attributes, drop dead callback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Instrumentation called self.class.generation_provider — a method that doesn't exist (the accessor is prompt_provider_klass) — so agent.provider and llm.provider were always rescued to "unknown", and the respond_to?(:provider_name) guards were false anyway because the helpers are private, so the attributes were never set at all. Same for messages.count, which read a nonexistent Base#messages instead of prompt_options[:messages]. Also removes the permanently-dead around_generate registration (the macro is around_generation, and no trace_generation method exists). Root and llm spans now carry provider tag_name + model; the prompt span carries the message count. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- lib/active_agent/telemetry/instrumentation.rb | 22 ++++++------- test/dashboard/telemetry_correlation_test.rb | 31 +++++++++++++++++++ 2 files changed, 41 insertions(+), 12 deletions(-) diff --git a/lib/active_agent/telemetry/instrumentation.rb b/lib/active_agent/telemetry/instrumentation.rb index c1643102..68900c8b 100644 --- a/lib/active_agent/telemetry/instrumentation.rb +++ b/lib/active_agent/telemetry/instrumentation.rb @@ -23,11 +23,6 @@ module Telemetry module Instrumentation extend ActiveSupport::Concern - included do - # Hook into generation lifecycle - around_generate :trace_generation if respond_to?(:around_generate) - end - class_methods do # Installs instrumentation on the agent class. # @@ -60,18 +55,20 @@ def process_prompt Telemetry.trace("#{self.class.name}.#{action_name}", span_type: :root, **{ trace_id: trace_id }.compact) do |span| span.set_attribute("agent.class", self.class.name) span.set_attribute("agent.action", action_name.to_s) - span.set_attribute("agent.provider", provider_name) if respond_to?(:provider_name) - span.set_attribute("agent.model", model_name) if respond_to?(:model_name) + span.set_attribute("agent.provider", provider_name) + span.set_attribute("agent.model", model_name) # Add prompt span prompt_span = span.add_span("agent.prompt", span_type: :prompt) - prompt_span.set_attribute("messages.count", messages.size) if respond_to?(:messages) + if (message_stack = prompt_options[:messages]).respond_to?(:size) + prompt_span.set_attribute("messages.count", message_stack.size) + end prompt_span.finish # Execute generation with LLM span llm_span = span.add_span("llm.generate", span_type: :llm) - llm_span.set_attribute("llm.provider", provider_name) if respond_to?(:provider_name) - llm_span.set_attribute("llm.model", model_name) if respond_to?(:model_name) + llm_span.set_attribute("llm.provider", provider_name) + llm_span.set_attribute("llm.model", model_name) begin result = super @@ -127,7 +124,7 @@ def process_embed Telemetry.trace("#{self.class.name}.embed", span_type: :embedding) do |span| span.set_attribute("agent.class", self.class.name) span.set_attribute("agent.action", "embed") - span.set_attribute("agent.provider", provider_name) if respond_to?(:provider_name) + span.set_attribute("agent.provider", provider_name) begin result = super @@ -150,7 +147,8 @@ def process_embed private def provider_name - self.class.generation_provider&.to_s || "unknown" + klass = prompt_provider_klass + klass.respond_to?(:tag_name) ? klass.tag_name : "unknown" rescue StandardError "unknown" end diff --git a/test/dashboard/telemetry_correlation_test.rb b/test/dashboard/telemetry_correlation_test.rb index 3d3623f8..1d29304d 100644 --- a/test/dashboard/telemetry_correlation_test.rb +++ b/test/dashboard/telemetry_correlation_test.rb @@ -94,6 +94,37 @@ def ping swap_global_tracer(original_tracer) end + test "instrumented generations record provider and model attributes" do + original_tracer = swap_global_tracer(ActiveAgent::Telemetry::Tracer.new(@configuration)) + + agent_class = Class.new(ApplicationAgent) do + def self.name = "AttributeProbeAgent" + generate_with :mock, model: "mock-model" + + def ping + prompt(message: "hello") + end + end + agent_class.include(ActiveAgent::Telemetry::Instrumentation) + agent_class.instrument_telemetry! + + agent_class.with({}).ping.generate_now + ActiveAgent::Telemetry.flush + + trace = ActiveAgent::TelemetryTrace.order(:created_at).last + root = trace.spans.find { |span| span["type"] == "root" } + llm = trace.spans.find { |span| span["type"] == "llm" } + prompt_span = trace.spans.find { |span| span["type"] == "prompt" } + + assert_equal "Mock", root.dig("attributes", "agent.provider") + assert_equal "mock-model", root.dig("attributes", "agent.model") + assert_equal "Mock", llm.dig("attributes", "llm.provider") + assert_equal "mock-model", llm.dig("attributes", "llm.model") + assert_operator prompt_span.dig("attributes", "messages.count").to_i, :>=, 1 + ensure + swap_global_tracer(original_tracer) + end + private def stored_payload From e1125f213f190533605a706240d3195b6d60e4f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 15:56:35 +0000 Subject: [PATCH 3/8] feat: per-model capability gating (ModelCapabilities) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thinking-first models reject sampling parameters with an API 400 — Anthropic's Opus 4.7+/Opus 5/Sonnet 5/Fable 5/Mythos 5 and OpenAI's o-series/GPT-5 family. ActiveAgent::ModelCapabilities strips the parameters a model rejects from the prepared prompt parameters before they reach the provider, so an agent configured with a shared temperature keeps working across model switches instead of erroring. Registry is extensible (ModelCapabilities.register(pattern, unsupported:)) with app rules consulted ahead of the built-ins, and can be disabled outright (ModelCapabilities.enabled = false) to defer to vendor enforcement. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- lib/active_agent.rb | 1 + lib/active_agent/base.rb | 4 ++ lib/active_agent/model_capabilities.rb | 89 ++++++++++++++++++++++++ test/model_capabilities_test.rb | 94 ++++++++++++++++++++++++++ 4 files changed, 188 insertions(+) create mode 100644 lib/active_agent/model_capabilities.rb create mode 100644 test/model_capabilities_test.rb diff --git a/lib/active_agent.rb b/lib/active_agent.rb index 7aeb9a31..6c2dcc74 100644 --- a/lib/active_agent.rb +++ b/lib/active_agent.rb @@ -105,6 +105,7 @@ module ActiveAgent autoload :Preview, "active_agent/concerns/preview" autoload :Previews, "active_agent/concerns/preview" autoload :GenerationJob + autoload :ModelCapabilities autoload :Observers, "active_agent/concerns/observers" autoload :Provider, "active_agent/concerns/provider" autoload :Rescue, "active_agent/concerns/rescue" diff --git a/lib/active_agent/base.rb b/lib/active_agent/base.rb index a665b7f6..54a383bc 100644 --- a/lib/active_agent/base.rb +++ b/lib/active_agent/base.rb @@ -304,6 +304,10 @@ def prepare_prompt_parameters # Render out proc/lamda attributes before rendering templates parameters.deep_transform_values! { _1.respond_to?(:call) ? _1.call : _1 } + # Strip parameters the target model rejects (e.g. temperature/top_p + # on thinking-first models) before they reach the provider. + ModelCapabilities.sanitize!(parameters) + # Apply Callbacks parameters.merge!( trace_id: prompt_options[:trace_id] || SecureRandom.uuid, diff --git a/lib/active_agent/model_capabilities.rb b/lib/active_agent/model_capabilities.rb new file mode 100644 index 00000000..9e9c5f84 --- /dev/null +++ b/lib/active_agent/model_capabilities.rb @@ -0,0 +1,89 @@ +# frozen_string_literal: true + +module ActiveAgent + # Per-model capability quirks, applied before a request reaches the + # provider. Vendors ship models that reject otherwise-standard sampling + # parameters (thinking-first models steered by prompting/effort instead) + # with an API 400 — this registry strips those parameters up front so an + # agent configured with a shared temperature keeps working across model + # switches. + # + # The built-in rules cover the known families; apps can extend the + # registry for new or self-hosted models: + # + # @example Register a custom rule + # ActiveAgent::ModelCapabilities.register(/\Amy-reasoning-model/, unsupported: [:temperature, :top_p]) + # + # @example Disable sanitization entirely + # ActiveAgent::ModelCapabilities.enabled = false + module ModelCapabilities + SAMPLING_PARAMS = [ :temperature, :top_p ].freeze + + # Model families that reject sampling parameters with a 400: + # - Anthropic thinking-first models (Opus 4.7+, Opus 5, Sonnet 5, + # Fable 5 / Mythos 5) + # - OpenAI reasoning models (o-series, GPT-5 family) + BUILTIN_RULES = [ + { pattern: /\Aclaude-(opus-5|opus-4-[78]|sonnet-5|fable-5|mythos-5)/, unsupported: SAMPLING_PARAMS }, + { pattern: /\A(o1|o3|o4)(-|$)/, unsupported: SAMPLING_PARAMS }, + { pattern: /\Agpt-5/, unsupported: SAMPLING_PARAMS } + ].freeze + + class << self + # Master switch; on by default. Set false to send parameters through + # untouched (the vendor then enforces its own rules). + attr_writer :enabled + + def enabled + return @enabled unless @enabled.nil? + + true + end + + # Registers an app-defined capability rule ahead of the built-ins. + # + # @param pattern [Regexp] matched against the model name + # @param unsupported [Array] parameter keys the model rejects + def register(pattern, unsupported:) + custom_rules << { pattern: pattern, unsupported: unsupported.map(&:to_sym) } + end + + def custom_rules + @custom_rules ||= [] + end + + def reset! + @custom_rules = [] + @enabled = nil + end + + # @return [Array] parameter keys the model rejects + def unsupported_params(model) + return [] if model.nil? + + (custom_rules + BUILTIN_RULES).each do |rule| + return rule[:unsupported] if model.to_s.match?(rule[:pattern]) + end + [] + end + + def sampling_supported?(model) + (unsupported_params(model) & SAMPLING_PARAMS).empty? + end + + # Strips parameters the model rejects, in place. Returns the removed + # keys (empty when nothing applied). + # + # @param parameters [Hash] prepared prompt parameters (must carry :model) + # @return [Array] removed parameter keys + def sanitize!(parameters) + return [] unless enabled + return [] unless parameters.is_a?(Hash) + + removed = unsupported_params(parameters[:model]).select { |key| parameters.key?(key) } + removed.each { |key| parameters.delete(key) } + removed + end + end + end +end diff --git a/test/model_capabilities_test.rb b/test/model_capabilities_test.rb new file mode 100644 index 00000000..30c3c453 --- /dev/null +++ b/test/model_capabilities_test.rb @@ -0,0 +1,94 @@ +# frozen_string_literal: true + +require "test_helper" + +class ModelCapabilitiesTest < ActiveSupport::TestCase + teardown { ActiveAgent::ModelCapabilities.reset! } + + test "thinking-first Claude models reject sampling params" do + %w[claude-sonnet-5 claude-opus-5 claude-fable-5 claude-mythos-5 claude-opus-4-7 claude-opus-4-8].each do |model| + assert_not ActiveAgent::ModelCapabilities.sampling_supported?(model), "expected #{model} to reject sampling" + end + end + + test "OpenAI reasoning models reject sampling params" do + %w[o1 o1-mini o3-mini o4-mini gpt-5.1 gpt-5].each do |model| + assert_not ActiveAgent::ModelCapabilities.sampling_supported?(model), "expected #{model} to reject sampling" + end + end + + test "conventional models keep sampling params" do + %w[claude-haiku-4-5 claude-sonnet-4-5 gpt-4o-mini gpt-4.1 qwen3:8b llama3.1:8b].each do |model| + assert ActiveAgent::ModelCapabilities.sampling_supported?(model), "expected #{model} to support sampling" + end + end + + test "sanitize! strips only the rejected params and reports them" do + parameters = { model: "claude-sonnet-5", temperature: 0.7, top_p: 0.9, max_tokens: 512 } + + removed = ActiveAgent::ModelCapabilities.sanitize!(parameters) + + assert_equal [ :temperature, :top_p ], removed.sort_by(&:to_s) + assert_equal({ model: "claude-sonnet-5", max_tokens: 512 }, parameters) + end + + test "sanitize! leaves conventional models untouched" do + parameters = { model: "gpt-4o-mini", temperature: 0.7 } + + assert_empty ActiveAgent::ModelCapabilities.sanitize!(parameters) + assert_equal 0.7, parameters[:temperature] + end + + test "custom rules are consulted ahead of the built-ins" do + ActiveAgent::ModelCapabilities.register(/\Ahouse-model/, unsupported: [ :temperature ]) + + assert_equal [ :temperature ], ActiveAgent::ModelCapabilities.unsupported_params("house-model-2") + parameters = { model: "house-model-2", temperature: 1.0, top_p: 0.5 } + ActiveAgent::ModelCapabilities.sanitize!(parameters) + assert_nil parameters[:temperature] + assert_equal 0.5, parameters[:top_p] + end + + test "disabling the switch passes parameters through" do + ActiveAgent::ModelCapabilities.enabled = false + parameters = { model: "claude-sonnet-5", temperature: 0.7 } + + assert_empty ActiveAgent::ModelCapabilities.sanitize!(parameters) + assert_equal 0.7, parameters[:temperature] + end + + test "prepared prompt parameters are sanitized for the configured model" do + agent_class = Class.new(ApplicationAgent) do + def self.name = "SanitizeProbeAgent" + generate_with :mock, model: "claude-sonnet-5", temperature: 0.7, top_p: 0.9, max_tokens: 256 + + def ping + prompt(message: "hello") + end + end + + agent = agent_class.new + agent.params = {} + agent.process(:ping) + parameters = agent.send(:prepare_prompt_parameters) + + assert_nil parameters[:temperature] + assert_nil parameters[:top_p] + assert_equal 256, parameters[:max_tokens] + assert_equal "claude-sonnet-5", parameters[:model] + end + + test "generation still succeeds end-to-end with stripped params" do + agent_class = Class.new(ApplicationAgent) do + def self.name = "SanitizeRunProbeAgent" + generate_with :mock, model: "claude-fable-5", temperature: 0.2 + + def ping + prompt(message: "hello") + end + end + + response = agent_class.with({}).ping.generate_now + assert response.message.content.present? + end +end From ae6629522ec1673bb01e775864aab3bbdb9a912f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 16:00:01 +0000 Subject: [PATCH 4/8] feat: bound the tool-calling loop with max_tool_turns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit process_prompt_finished re-entered resolve_prompt with no cap — a model that keeps emitting tool calls recursed until the provider stopped it. Each generation now counts tool round-trips against max_tool_turns (default 25, configurable per agent/prompt via generate_with/prompt options). Hitting the cap emits a tool_turns_exceeded.active_agent notification and finishes cleanly with the messages gathered so far instead of raising or recursing on. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- lib/active_agent/providers/_base_provider.rb | 25 +++++- .../base_provider_tool_turns_test.rb | 76 +++++++++++++++++++ 2 files changed, 99 insertions(+), 2 deletions(-) create mode 100644 test/providers/base_provider_tool_turns_test.rb diff --git a/lib/active_agent/providers/_base_provider.rb b/lib/active_agent/providers/_base_provider.rb index f2323fbd..165fa8f7 100644 --- a/lib/active_agent/providers/_base_provider.rb +++ b/lib/active_agent/providers/_base_provider.rb @@ -55,7 +55,13 @@ class ProvidersError < StandardError; end :request, :message_stack, # Runtime :stream_broadcaster, :streaming, # Callback (Streams) :tools_function, # Callback (Tools) - :usage_stack # Usage Tracking + :usage_stack, # Usage Tracking + :max_tool_turns, :tool_turns # Tool-loop safety + + # Upper bound on tool-calling round-trips within one generation. A + # model that keeps emitting tool calls otherwise recurses until the + # provider stops it — override per agent/prompt with max_tool_turns:. + DEFAULT_MAX_TOOL_TURNS = 25 # @return [String] e.g., "Anthropic", "OpenAI" def self.service_name @@ -106,6 +112,8 @@ def initialize(kwargs = {}) self.stream_broadcaster = kwargs.delete(:stream_broadcaster) self.streaming = false self.tools_function = kwargs.delete(:tools_function) + self.max_tool_turns = kwargs.delete(:max_tool_turns) || DEFAULT_MAX_TOOL_TURNS + self.tool_turns = 0 self.options = options_klass.new(kwargs.extract!(*options_klass.keys)) self.context = kwargs self.message_stack = [] @@ -344,7 +352,7 @@ def process_prompt_finished(api_response = nil) message_stack.push(*api_messages) end - if (tool_calls = process_prompt_finished_extract_function_calls)&.any? + if (tool_calls = process_prompt_finished_extract_function_calls)&.any? && tool_turn_allowed? process_function_calls(tool_calls) resolve_prompt else @@ -373,6 +381,19 @@ def process_prompt_finished(api_response = nil) end end + # Counts a tool round-trip against the per-generation cap. When the + # cap is hit the loop finishes cleanly with the messages gathered so + # far (a partial result) instead of recursing indefinitely. + # + # @return [Boolean] whether another tool round-trip may run + def tool_turn_allowed? + self.tool_turns += 1 + return true if max_tool_turns.nil? || tool_turns <= max_tool_turns + + instrument("tool_turns_exceeded.active_agent", limit: max_tool_turns) + false + end + # @abstract # @param api_response [Object] # @return [Array, nil] diff --git a/test/providers/base_provider_tool_turns_test.rb b/test/providers/base_provider_tool_turns_test.rb new file mode 100644 index 00000000..286c0976 --- /dev/null +++ b/test/providers/base_provider_tool_turns_test.rb @@ -0,0 +1,76 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "../../lib/active_agent/providers/mock_provider" + +# Tool-loop safety: process_prompt_finished re-enters resolve_prompt while +# the model keeps emitting tool calls. The max_tool_turns cap bounds that +# recursion and finishes cleanly with the messages gathered so far. +class BaseProviderToolTurnsTest < ActiveSupport::TestCase + # A mock provider whose "model" emits a tool call on every response — + # unbounded, this would recurse forever. + class LoopingMockProvider < ActiveAgent::Providers::MockProvider + # Type resolution (service_name/namespace) derives from the class + # name; keep the Mock identity for this test-local subclass. + def self.name = "ActiveAgent::Providers::MockProvider" + + attr_reader :tool_rounds + + def process_prompt_finished_extract_function_calls + [ { name: "spin", input: {}, id: "call_#{object_id}_#{@tool_rounds}" } ] + end + + def process_function_calls(_calls) + @tool_rounds = (@tool_rounds || 0) + 1 + message_stack.push({ role: "user", content: "tool result #{@tool_rounds}" }) + end + end + + test "max_tool_turns bounds the tool-calling recursion" do + provider = LoopingMockProvider.new( + messages: [ { role: "user", content: "go" } ], + max_tool_turns: 3 + ) + + response = provider.prompt + + assert_equal 3, provider.tool_rounds + assert response.present?, "capped loop should still return a response" + assert response.messages.any? + end + + test "hitting the cap emits a tool_turns_exceeded notification" do + events = [] + subscription = ActiveSupport::Notifications.subscribe("tool_turns_exceeded.active_agent") do |*, payload| + events << payload + end + + LoopingMockProvider.new( + messages: [ { role: "user", content: "go" } ], + max_tool_turns: 2 + ).prompt + + assert_equal 1, events.length + assert_equal 2, events.first[:limit] + ensure + ActiveSupport::Notifications.unsubscribe(subscription) + end + + test "the default cap applies when none is configured" do + provider = LoopingMockProvider.new(messages: [ { role: "user", content: "go" } ]) + + provider.prompt + + assert_equal ActiveAgent::Providers::BaseProvider::DEFAULT_MAX_TOOL_TURNS, provider.tool_rounds + end + + test "generations without tool calls are unaffected" do + provider = ActiveAgent::Providers::MockProvider.new( + messages: [ { role: "user", content: "hello there" } ] + ) + + response = provider.prompt + + assert response.message.content.present? + end +end From 7120207f7ce851131bb1cac64bce294fded11efa Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 16:01:54 +0000 Subject: [PATCH 5/8] feat: provider error taxonomy (Errors::*) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vendor SDKs raise their own classes for the same failures, so retry/backoff/fallback policy couldn't be written portably. API-shaped exceptions are now normalized in with_exception_handling into a small taxonomy — RateLimited, ContextLengthExceeded, AuthenticationFailed, ContentFiltered, ServiceUnavailable, InvalidRequest — classified by vendor class name, HTTP status, and message heuristics (context overflow and content-filter 400s classify specifically). The original exception is preserved as #cause and the typed error carries status + provider tag; ordinary Ruby errors pass through untouched. rescue_from in agents now works identically across providers. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- .../providers/concerns/exception_handler.rb | 13 +- lib/active_agent/providers/errors.rb | 140 ++++++++++++++++++ test/providers/errors_taxonomy_test.rb | 83 +++++++++++ 3 files changed, 235 insertions(+), 1 deletion(-) create mode 100644 lib/active_agent/providers/errors.rb create mode 100644 test/providers/errors_taxonomy_test.rb diff --git a/lib/active_agent/providers/concerns/exception_handler.rb b/lib/active_agent/providers/concerns/exception_handler.rb index 9abb1d08..4431e4b3 100644 --- a/lib/active_agent/providers/concerns/exception_handler.rb +++ b/lib/active_agent/providers/concerns/exception_handler.rb @@ -1,5 +1,7 @@ # frozen_string_literal: true +require_relative "../errors" + module ActiveAgent module Providers # Provides exception handling for provider operations. @@ -54,7 +56,16 @@ def configure_exception_handler(exception_handler: nil) def with_exception_handling(&block) yield rescue => exception - rescue_with_handler(exception) || raise + # Vendor API failures are normalized into the framework taxonomy + # (Errors::RateLimited, Errors::ContextLengthExceeded, ...) so + # rescue_from policy is portable across providers; the original + # exception is preserved as #cause. Anything unrecognizable — + # including ordinary Ruby errors — passes through untouched. + exception = Errors::Taxonomy.normalize( + exception, + provider_tag: (tag_name if respond_to?(:tag_name)) + ) + rescue_with_handler(exception) || raise(exception) nil # Discard handler return value to prevent polluting raw_response end diff --git a/lib/active_agent/providers/errors.rb b/lib/active_agent/providers/errors.rb new file mode 100644 index 00000000..bcb7d725 --- /dev/null +++ b/lib/active_agent/providers/errors.rb @@ -0,0 +1,140 @@ +# frozen_string_literal: true + +module ActiveAgent + module Providers + # Typed provider failures, normalized across vendor SDKs. + # + # Every vendor raises its own exception classes for the same underlying + # conditions (rate limits, context overflows, content filters, outages), + # which makes retry/backoff/fallback policy impossible to express + # portably. The taxonomy classifies vendor errors into a small set of + # framework types — the original exception is preserved as +#cause+, so + # nothing is lost. + # + # @example Portable retry policy + # rescue_from ActiveAgent::Providers::Errors::RateLimited do |error| + # retry_job wait: 30.seconds + # end + # + # @example Fallback on outage + # rescue_from ActiveAgent::Providers::Errors::ServiceUnavailable do |error| + # FallbackAgent.with(params).ask.generate_later + # end + module Errors + # Base class for normalized provider failures. + class ProviderError < StandardError + # @return [Integer, nil] HTTP status from the vendor error, when known + attr_reader :status + + # @return [String, nil] provider tag (e.g. "Anthropic", "OpenAI::Chat") + attr_reader :provider_tag + + def initialize(message = nil, status: nil, provider_tag: nil) + super(message) + @status = status + @provider_tag = provider_tag + end + end + + # 429s / vendor rate & quota limits. Retryable with backoff. + class RateLimited < ProviderError; end + + # The prompt exceeded the model's context window. Not retryable + # without shrinking the input. + class ContextLengthExceeded < ProviderError; end + + # Invalid, expired, or unauthorized credentials (401/403). + class AuthenticationFailed < ProviderError; end + + # The vendor's safety layer refused the request or response. + class ContentFiltered < ProviderError; end + + # Vendor-side failure or overload (5xx, timeouts, connection drops). + # Retryable; a natural trigger for provider fallback. + class ServiceUnavailable < ProviderError; end + + # Malformed or unsupported request the vendor rejected (400/422) + # that doesn't classify more specifically. + class InvalidRequest < ProviderError; end + + # Classifies vendor SDK exceptions into the taxonomy. Unrecognizable + # exceptions (including ordinary Ruby errors) pass through untouched — + # only errors that look like vendor API failures are normalized. + module Taxonomy + # Vendor SDK class names (demodulized) → taxonomy class. Covers the + # official anthropic/openai gems and SDKs following their naming. + NAME_MAP = { + "RateLimitError" => RateLimited, + "AuthenticationError" => AuthenticationFailed, + "PermissionDeniedError" => AuthenticationFailed, + "ContentFilterError" => ContentFiltered, + "InternalServerError" => ServiceUnavailable, + "APIConnectionError" => ServiceUnavailable, + "APIConnectionTimeoutError" => ServiceUnavailable, + "APITimeoutError" => ServiceUnavailable, + "OverloadedError" => ServiceUnavailable, + "ServiceUnavailableError" => ServiceUnavailable, + "BadRequestError" => InvalidRequest, + "UnprocessableEntityError" => InvalidRequest + }.freeze + + STATUS_MAP = { + 400 => InvalidRequest, + 401 => AuthenticationFailed, + 403 => AuthenticationFailed, + 408 => ServiceUnavailable, + 422 => InvalidRequest, + 429 => RateLimited, + 529 => ServiceUnavailable # Anthropic "overloaded" + }.freeze + + CONTEXT_LENGTH_PATTERN = /context length|context_length|maximum context|context window|too many tokens|prompt is too long|input (?:is )?too long/i + CONTENT_FILTER_PATTERN = /content (?:filter|policy|management)|filtered due to|blocked by|safety (?:system|filter)/i + + class << self + # @param exception [Exception] + # @param provider_tag [String, nil] + # @return [Exception] a taxonomy error, or the original exception + # when it doesn't classify + def normalize(exception, provider_tag: nil) + return exception if exception.is_a?(ProviderError) + + klass = classify(exception) + return exception unless klass + + klass.new(exception.message, status: status_of(exception), provider_tag: provider_tag) + end + + # @return [Class, nil] + def classify(exception) + name = exception.class.name.to_s.demodulize + status = status_of(exception) + api_error = NAME_MAP.key?(name) || !status.nil? + return nil unless api_error + + message = exception.message.to_s + return ContextLengthExceeded if CONTEXT_LENGTH_PATTERN.match?(message) + return ContentFiltered if CONTENT_FILTER_PATTERN.match?(message) + + NAME_MAP[name] || STATUS_MAP[status] || (status && status >= 500 ? ServiceUnavailable : nil) + end + + # @return [Integer, nil] + def status_of(exception) + [ :status, :status_code, :http_status, :code ].each do |reader| + next unless exception.respond_to?(reader) + + value = begin + exception.public_send(reader) + rescue StandardError + nil + end + return value if value.is_a?(Integer) + end + nil + end + end + end + end + end +end diff --git a/test/providers/errors_taxonomy_test.rb b/test/providers/errors_taxonomy_test.rb new file mode 100644 index 00000000..d607dada --- /dev/null +++ b/test/providers/errors_taxonomy_test.rb @@ -0,0 +1,83 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "../../lib/active_agent/providers/mock_provider" + +class ErrorsTaxonomyTest < ActiveSupport::TestCase + Errors = ActiveAgent::Providers::Errors + + # Vendor-SDK-shaped exceptions (the official gems expose #status). + class FakeStatusError < StandardError + def initialize(message, status) + super(message) + @status = status + end + attr_reader :status + end + + class RateLimitError < StandardError; end + + test "classifies by vendor class name" do + error = Errors::Taxonomy.normalize(RateLimitError.new("slow down")) + assert_instance_of Errors::RateLimited, error + assert_equal "slow down", error.message + end + + test "classifies by HTTP status" do + assert_instance_of Errors::RateLimited, Errors::Taxonomy.normalize(FakeStatusError.new("429", 429)) + assert_instance_of Errors::AuthenticationFailed, Errors::Taxonomy.normalize(FakeStatusError.new("bad key", 401)) + assert_instance_of Errors::InvalidRequest, Errors::Taxonomy.normalize(FakeStatusError.new("bad params", 400)) + assert_instance_of Errors::ServiceUnavailable, Errors::Taxonomy.normalize(FakeStatusError.new("boom", 503)) + assert_instance_of Errors::ServiceUnavailable, Errors::Taxonomy.normalize(FakeStatusError.new("overloaded", 529)) + end + + test "context overflow and content filter win over the generic 400" do + context_error = Errors::Taxonomy.normalize(FakeStatusError.new("prompt is too long: 250000 tokens > maximum context", 400)) + assert_instance_of Errors::ContextLengthExceeded, context_error + + filter_error = Errors::Taxonomy.normalize(FakeStatusError.new("Response blocked by content filter", 400)) + assert_instance_of Errors::ContentFiltered, filter_error + end + + test "captures status and provider tag" do + error = Errors::Taxonomy.normalize(FakeStatusError.new("429", 429), provider_tag: "Anthropic") + assert_equal 429, error.status + assert_equal "Anthropic", error.provider_tag + end + + test "ordinary Ruby errors pass through untouched" do + original = NoMethodError.new("undefined method") + assert_same original, Errors::Taxonomy.normalize(original) + + plain = StandardError.new("something odd") + assert_same plain, Errors::Taxonomy.normalize(plain) + end + + test "already-normalized errors pass through" do + original = Errors::RateLimited.new("again") + assert_same original, Errors::Taxonomy.normalize(original) + end + + test "provider raises the typed error with the original as cause" do + provider = ActiveAgent::Providers::MockProvider.new(service: "Mock") + + raised = assert_raises(Errors::RateLimited) do + provider.send(:with_exception_handling) { raise FakeStatusError.new("too fast", 429) } + end + + assert_instance_of FakeStatusError, raised.cause + assert_equal "Mock", raised.provider_tag + end + + test "exception_handler receives the typed error" do + seen = nil + provider = ActiveAgent::Providers::MockProvider.new( + service: "Mock", + exception_handler: ->(exception) { seen = exception } + ) + + provider.send(:with_exception_handling) { raise FakeStatusError.new("too fast", 429) } + + assert_instance_of Errors::RateLimited, seen + end +end From e4b6cdd91f19a00ea02384e866570e70a0c94bfb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 16:04:00 +0000 Subject: [PATCH 6/8] feat(telemetry): implement redact_attributes scrubbing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit redact_attributes was documented configuration the tracer never consumed. Span and span-event attribute keys matching any configured pattern (case-insensitive substring; defaults cover password/secret/ token/key/credential/api_key) now have their values replaced with [REDACTED] in build_trace_payload — the shared choke point for both transmission and local storage — so secrets can't leave the process in traces. Matching is deliberately over-broad: better a redacted max_tokens than a shipped api_key. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- lib/active_agent/telemetry/configuration.rb | 6 +- lib/active_agent/telemetry/tracer.rb | 42 +++++++++++- test/dashboard/telemetry_redaction_test.rb | 71 +++++++++++++++++++++ 3 files changed, 116 insertions(+), 3 deletions(-) create mode 100644 test/dashboard/telemetry_redaction_test.rb diff --git a/lib/active_agent/telemetry/configuration.rb b/lib/active_agent/telemetry/configuration.rb index 6ba61474..fc4967c8 100644 --- a/lib/active_agent/telemetry/configuration.rb +++ b/lib/active_agent/telemetry/configuration.rb @@ -50,8 +50,10 @@ class Configuration # @note Reserved: not yet consumed by the tracer/instrumentation. attr_accessor :capture_bodies - # @return [Array] Attributes to redact from traces - # @note Reserved: not yet consumed by the tracer/instrumentation. + # @return [Array] Attribute-key patterns to redact from + # traces (case-insensitive substring match against span and + # span-event attribute keys; matching values become "[REDACTED]" + # before the payload leaves the process) attr_accessor :redact_attributes # @return [String] Service name for trace attribution diff --git a/lib/active_agent/telemetry/tracer.rb b/lib/active_agent/telemetry/tracer.rb index 0edfbad1..3796e868 100644 --- a/lib/active_agent/telemetry/tracer.rb +++ b/lib/active_agent/telemetry/tracer.rb @@ -141,10 +141,26 @@ def build_trace_payload(root_span) environment: configuration.environment, timestamp: Time.current.iso8601(6), resource_attributes: configuration.resource_attributes, - spans: flatten_spans(root_span) + spans: redact_spans(flatten_spans(root_span)) } end + # Redacts span (and span-event) attributes whose keys match any + # configured redact_attributes entry, before the payload leaves the + # process. Matching is case-insensitive substring — deliberately + # over-broad: better to redact a harmless "max_tokens" than to ship + # an "api_key". + # + # @param spans [Array] flattened span data + # @return [Array] + def redact_spans(spans) + patterns = Array(configuration.redact_attributes).map(&:to_s).reject(&:empty?) + return spans if patterns.empty? + + matcher = Regexp.union(patterns.map { |pattern| Regexp.new(Regexp.escape(pattern), Regexp::IGNORECASE) }) + spans.map { |span| redact_span(span, matcher) } + end + # Flattens span hierarchy into array. # # @param span [Span] Root span @@ -157,6 +173,30 @@ def flatten_spans(span) result end + # @param span [Hash] + # @param matcher [Regexp] + # @return [Hash] span with matching attribute values replaced + def redact_span(span, matcher) + span = span.dup + span[:attributes] = redact_hash(span[:attributes], matcher) if span[:attributes].is_a?(Hash) + if span[:events].is_a?(Array) + span[:events] = span[:events].map do |event| + next event unless event.is_a?(Hash) && event[:attributes].is_a?(Hash) + + event.merge(attributes: redact_hash(event[:attributes], matcher)) + end + end + span + end + + REDACTED = "[REDACTED]" + + def redact_hash(attributes, matcher) + attributes.to_h do |key, value| + [ key, key.to_s.match?(matcher) ? REDACTED : value ] + end + end + # Returns whether this trace should be sampled. # # @return [Boolean] diff --git a/test/dashboard/telemetry_redaction_test.rb b/test/dashboard/telemetry_redaction_test.rb new file mode 100644 index 00000000..120bf6b3 --- /dev/null +++ b/test/dashboard/telemetry_redaction_test.rb @@ -0,0 +1,71 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "telemetry_trace_test" + +# redact_attributes: configured key patterns are scrubbed from span and +# span-event attributes before the payload leaves the process (transmission +# and local storage share build_trace_payload). +class TelemetryRedactionTest < ActiveSupport::TestCase + TelemetryTraceTest.ensure_table! + + def setup + ActiveAgent::TelemetryTrace.delete_all + + @configuration = ActiveAgent::Telemetry::Configuration.new + @configuration.enabled = true + @configuration.local_storage = true + @configuration.service_name = "dummy" + end + + test "redacts matching span and event attributes before storage" do + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") do |span| + span.set_attribute("llm.api_key", "sk-live-123") + span.set_attribute("http.authorization_token", "Bearer abc") + span.set_attribute("llm.model", "claude-sonnet-5") + span.add_event("tool.call", { "password" => "hunter2", "tool.name" => "fetch_url" }) + end + tracer.flush + + trace = ActiveAgent::TelemetryTrace.first + root = trace.spans.first + + assert_equal "[REDACTED]", root.dig("attributes", "llm.api_key") + assert_equal "[REDACTED]", root.dig("attributes", "http.authorization_token") + assert_equal "claude-sonnet-5", root.dig("attributes", "llm.model") + + event = root["events"].first + assert_equal "[REDACTED]", event.dig("attributes", "password") + assert_equal "fetch_url", event.dig("attributes", "tool.name") + end + + test "custom redact_attributes replace the defaults" do + @configuration.redact_attributes = %w[ssn] + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") do |span| + span.set_attribute("user.ssn", "000-00-0000") + span.set_attribute("llm.api_key", "left-alone-by-custom-config") + end + tracer.flush + + root = ActiveAgent::TelemetryTrace.first.spans.first + assert_equal "[REDACTED]", root.dig("attributes", "user.ssn") + assert_equal "left-alone-by-custom-config", root.dig("attributes", "llm.api_key") + end + + test "empty redact_attributes disables scrubbing" do + @configuration.redact_attributes = [] + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") do |span| + span.set_attribute("llm.api_key", "sk-live-123") + end + tracer.flush + + root = ActiveAgent::TelemetryTrace.first.spans.first + assert_equal "sk-live-123", root.dig("attributes", "llm.api_key") + end +end From 6f68c599052308b3fcc7e022dafcceb9fb443c78 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 16:07:07 +0000 Subject: [PATCH 7/8] docs: mark shipped roadmap items Capability gating, tool-loop turn cap, error taxonomy, telemetry redaction, and the instrumentation fixes are implemented; agent_runs landed in solid_agent. Remaining v2 scope updated accordingly. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- docs/framework/v2-extraction-roadmap.md | 64 +++++++++++++------------ 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/docs/framework/v2-extraction-roadmap.md b/docs/framework/v2-extraction-roadmap.md index edc75f42..4df2977a 100644 --- a/docs/framework/v2-extraction-roadmap.md +++ b/docs/framework/v2-extraction-roadmap.md @@ -17,14 +17,13 @@ platform code that any serious consumer of the gem would have to rebuild. ## Absorb from the platform -### 1. Per-model capability gating -The platform strips `temperature`/`top_p` for models that reject them -(Claude Opus 4.7+, Sonnet 5, Fable/Mythos 5) via a regex before -`generate_with`. The framework has no capability layer at all — the request -objects apply `DEFAULTS` (`temperature: 1, top_p: 1`) unconditionally and let -the vendor 400. v2: a model-capability table consulted by the Request layer -(sampling params, max_tokens vs max_completion_tokens, reasoning-effort -support), with a config escape hatch for unknown models. +### 1. Per-model capability gating — ✅ shipped +`ActiveAgent::ModelCapabilities` strips parameters the target model rejects +(temperature/top_p on thinking-first Claude, OpenAI o-series/GPT-5) from +prepared prompt parameters before they reach the provider. Extensible via +`ModelCapabilities.register(pattern, unsupported:)`; disable with +`ModelCapabilities.enabled = false`. Remaining for v2: max_tokens vs +max_completion_tokens switching and reasoning-effort awareness. ### 2. Provider model catalogs The platform's `/api/provider_models` queries Ollama's live model list, the @@ -32,11 +31,11 @@ Anthropic Models API, and OpenRouter's catalog, with curated fallbacks. v2: `Provider#models` on the provider contract (vendor SDKs all expose a listing endpoint), so model pickers and validation stop being app problems. -### 3. Tool-loop safety -`process_prompt_finished` re-enters `resolve_prompt` with **no max-turn cap -and no token/cost budget** — a looping model recurses until the provider -stops emitting tool calls. v2: `max_tool_turns` and a token budget on the -generation, with a clean partial-result return when hit. +### 3. Tool-loop safety — ✅ shipped (turns) +`max_tool_turns` (default 25, per agent/prompt override) now bounds the +tool-calling recursion; hitting the cap emits +`tool_turns_exceeded.active_agent` and returns the messages gathered so far. +Remaining for v2: a token/cost budget alongside the turn cap. ### 4. Agent-to-agent delegation `tools_function` only routes back to `self`. The platform built `call_agent` @@ -44,13 +43,14 @@ generation, with a clean partial-result return when hit. v2: a first-class delegation primitive — invoke another agent class/instance as a tool, with depth limits and shared trace/context correlation. -### 5. Provider error taxonomy + fallback -Only `ProvidersError` exists and nothing raises it; rate limits, context -overflows, and content filters surface as vendor-specific exceptions, so no -retry/fallback policy can be written against them. v2: typed errors -(`RateLimited`, `ContextLengthExceeded`, `ContentFiltered`, …) normalized -across providers, then a `generate_with ... fallback: [:anthropic, :ollama]` -chain becomes expressible. +### 5. Provider error taxonomy + fallback — ✅ taxonomy shipped +`ActiveAgent::Providers::Errors` (RateLimited, ContextLengthExceeded, +AuthenticationFailed, ContentFiltered, ServiceUnavailable, InvalidRequest) +now normalizes vendor exceptions in `with_exception_handling` — classified +by SDK class name, HTTP status, and message heuristics, original preserved +as `#cause` — so `rescue_from` policy is portable across providers. +Remaining for v2: the `generate_with ... fallback: [:anthropic, :ollama]` +chain the taxonomy makes expressible. ### 6. A real MCP story Today MCP is pass-through only: `mcps:` options are normalized into each @@ -78,18 +78,20 @@ results stays solid_agent's. ## Fix in place (bugs and dead seams found during the audit) -- `telemetry/instrumentation.rb` calls `self.class.generation_provider` - (method doesn't exist → `llm.provider` attribute always "unknown"/absent), - registers `around_generate` (macro is `around_generation` → permanently - dead line), and guards on `respond_to?(:messages)` (Base has no - `#messages` → message counts never recorded). +- ✅ **Fixed**: `telemetry/instrumentation.rb` — provider/model/message-count + attributes now record (was: nonexistent `generation_provider`, private + helpers hidden from `respond_to?`, nonexistent `Base#messages`); the dead + `around_generate` registration is removed. +- ✅ **Fixed**: `redact_attributes` is now consumed — span and span-event + attribute values matching the configured patterns become `[REDACTED]` in + `build_trace_payload`, covering both transmission and local storage. + `capture_bodies` remains reserved (telemetry spans capture no bodies yet; + the raw response on the ActiveSupport::Notifications payload is + in-process only). - `Observers`/`Interceptors` call `Prompt.register_observer` on an `ActiveAgent::Prompt` class that doesn't exist; nothing in the generation path notifies them. Either implement the ActionMailer-style seam (persistence layers want it) or delete it. -- Telemetry `capture_bodies`/`redact_attributes` are documented config that - is never consumed, while notification payloads attach full raw responses. - Implement redaction before v2 ships. - The dashboard engine ships orphaned platform-shaped models (`Dashboard::Agent`, `AgentRun`, `SandboxSession`, jobs, migrations) with no controllers or routes. Decide: wire them (framework-level run @@ -106,9 +108,9 @@ callbacks), never implementations. ## solid_agent follow-ups (tracked there, listed for completeness) -- `agent_runs` + persisted run progress events (the platform's - `AgentRun#append_event` and instruction-cohort fingerprints are the - proven shape). +- ✅ **Shipped**: `agent_runs` + persisted run progress events — the install + generator now ships an `AgentRun` model with lifecycle, `append_event`, + trace correlation, and `SolidAgent::RunFingerprint` instruction cohorts. - Evaluation datasets — `docs/agent-md-spec.md` already specifies `*.test.yml` cases; the platform's rule-criteria scorer and LLM-judge plumbing are the reference implementation. From a56b0c54e50c6b971c0214fe0cd120b779cc5137 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 16:49:12 +0000 Subject: [PATCH 8/8] test: previews reflect ModelCapabilities sanitization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gpt-5 / claude-sonnet-5 preview fixtures configure temperature: 0.7 — a combination those models reject with a 400 at the real API. ModelCapabilities now strips it before the request is built, so the preview (which renders the request that will actually be sent) no longer shows it. Assert the strip instead of the stale passthrough. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01B5PaDq3uaUWuA8UW4K7siP --- test/integration/anthropic/common_format/preview_test.rb | 7 +++++-- .../integration/open_ai/chat/common_format/preview_test.rb | 7 +++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/test/integration/anthropic/common_format/preview_test.rb b/test/integration/anthropic/common_format/preview_test.rb index fa6303ad..1c893b65 100644 --- a/test/integration/anthropic/common_format/preview_test.rb +++ b/test/integration/anthropic/common_format/preview_test.rb @@ -81,8 +81,11 @@ def comprehensive_test assert_includes preview, "Get detailed weather forecast" assert_includes preview, "Find popular attractions" - # Check parameters in YAML section - assert_includes preview, "temperature: 0.7" + # Check parameters in YAML section. The configured temperature is + # stripped by ModelCapabilities — claude-sonnet-5 is a + # thinking-first model that rejects sampling params — so the + # preview reflects the request that will actually be sent. + refute_includes preview, "temperature" assert_includes preview, "max_tokens: 2000" end end diff --git a/test/integration/open_ai/chat/common_format/preview_test.rb b/test/integration/open_ai/chat/common_format/preview_test.rb index 1ab4e91e..e9b0283d 100644 --- a/test/integration/open_ai/chat/common_format/preview_test.rb +++ b/test/integration/open_ai/chat/common_format/preview_test.rb @@ -82,8 +82,11 @@ def comprehensive_test assert_includes preview, "Get detailed weather forecast" assert_includes preview, "Find popular attractions" - # Check parameters in YAML section - assert_includes preview, "temperature: 0.7" + # Check parameters in YAML section. The configured temperature + # is stripped by ModelCapabilities — gpt-5 is a reasoning model + # that rejects sampling params — so the preview reflects the + # request that will actually be sent. + refute_includes preview, "temperature" assert_includes preview, "max_tokens: 2000" end end