fix(claude_agent_sdk): add provider metadata, drop deep-copy serialization, allowlist hook fields#585
Merged
Abhijeet Prasad (AbhiPrasad) merged 2 commits intoJul 17, 2026
Conversation
…ation, allowlist hook fields - LLM spans now include `metadata.provider = "anthropic"` per the spec-required `metadata.model` + `metadata.provider` rule. - `_serialize_content_blocks` replaces `dataclasses.asdict()` (which deep-copies leaves) with a shallow field copy. Nested content and ToolUseBlock inputs pass through to Braintrust's send-time serializer. - `_serialize_hook_value` drops the recursive walk and `dataclasses.asdict` path; hook payloads are filtered at the top level against an allowlist of documented Claude Code hook input/output fields. `cwd`, `transcript_path`, and `agent_transcript_path` remain excluded by default. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad)
approved these changes
Jul 17, 2026
Abhijeet Prasad (AbhiPrasad)
deleted the
fix-claude-agent-sdk-spec-and-serialization
branch
July 17, 2026 16:01
7 tasks
Abhijeet Prasad (AbhiPrasad)
pushed a commit
that referenced
this pull request
Jul 17, 2026
…er serialization (#587) ## Summary Align the CrewAI integration with `.agents/skills/sdk-integrations/SKILL.md`. Audit turned up five issues: - **Missing `metadata.provider` on `crewai.llm` spans** — spec requires every `llm` span to carry both `metadata.model` and `metadata.provider`. Added `_provider_from_source()`, which reads the `provider` string off the emitting CrewAI `LLM` object (`"openai"` / `"anthropic"` / `"bedrock"` / ...). - **Tools in `input` instead of `metadata.tools`** — spec says tool definitions live in `metadata.tools`, not `input`. Moved both the LLM-call `tools` and the agent `tools` onto `metadata`. - **Token double-counting on `crewai.llm`** — the previous rule only skipped tokens when the LiteLLM integration was patched, but CrewAI 1.x routes `gpt-*` to a native `openai` client, `claude-*` to a native `anthropic` client, etc. When those integrations are patched (as `auto_instrument()` does by default), they emit a leaf span with tokens and `crewai.llm` double-counted at every ancestor in the trace-tree rollup. The SKILL is explicit here: "Do not add 'if OpenAI is patched, skip metrics' checks — define clear ownership instead." CrewAI is always an orchestration layer that delegates to a provider SDK, so the clear rule is: **`crewai.llm` never owns tokens**. Same pattern `pydantic_ai` already uses for its wrapper spans. Dropped `_litellm_owns_leaf_span` + the token-name/prefix maps + the `extra_metrics` plumbing on `_end_span` accordingly. - **Excess serialization** — dropped eager `_try_to_dict` / `_normalize_output` / `_normalize_tools` calls across kickoff, task, agent, LLM, and tool output paths. `bt_json._to_bt_safe` at log time already handles Pydantic v2 (`model_dump`), Pydantic v1 (`dict`), and dataclasses; the eager pre-pass was wasted work (same pattern as #585 for claude_agent_sdk and #583 for anthropic). All other metadata extractors (`_causal_metadata`, `_crew_metadata`, `_task_metadata`, `_LLM_CONFIG_FIELDS`) were already allowlist-based — no changes needed there. ## Test plan - [x] Extended `test_kickoff_llm_event_tree_parents_and_shape` to emit with a real `LLM` source and assert `metadata.provider == "openai"`. - [x] Added `test_llm_tools_route_to_metadata_not_input` (positive-in-metadata + negative-not-in-input). - [x] Replaced the two conditional token tests with a single `test_llm_never_emits_token_metrics` that asserts no token key (`tokens`, `prompt_tokens`, `completion_tokens`, `prompt_cached_tokens`, `completion_reasoning_tokens`) leaks onto `crewai.llm` regardless of which provider integrations are patched. - [x] **No new cassettes.** The test file docstring documents why VCR is impractical for CrewAI (pytest-vcr + httpcore + CrewAI's native `OpenAICompletion` `model_post_init` interaction bug). Direct-event tests remain the source of truth here; the LiteLLM `mock_response` smoke test still exercises the full `crew.kickoff()` path. - [x] `cd py && BRAINTRUST_TEST_PACKAGE_VERSION=latest pytest src/braintrust/integrations/crewai/test_crewai.py -v` — 17 passed - [x] `cd py && nox -s pylint` — success - [x] `pre-commit run` on the changed files — passed (ruff format, ruff check, codespell, EOF/trailing-whitespace) 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- sfk:created-approved-by --> Created by abhijeet <!-- sfk:slack-thread --> [Slack thread](https://starfolkai.slack.com/archives/C0AQDETAVT3/p1784304223300019?thread_ts=1784304223.300019&cid=C0AQDETAVT3) --------- Co-authored-by: Starfolk <noreply@starfolk.ai> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Merged
5 tasks
Abhijeet Prasad (AbhiPrasad)
pushed a commit
that referenced
this pull request
Jul 17, 2026
…op eager serialization (#590) ## Summary Align the Google GenAI integration with `.agents/skills/sdk-integrations/SKILL.md`. Audit turned up four issues: - **Missing `metadata.provider` on every span** — spec requires every `llm` span to carry both `metadata.model` and `metadata.provider`. Added `metadata.provider = \"google\"` on `generate_content`, `generate_content_stream`, `embed_content`, `generate_images`, and every `interactions.*` span. - **Tools in `input.config` instead of `metadata.tools`** — spec says tool definitions live in `metadata.tools`. Route the serialized Google-native tool declarations onto `metadata.tools` for the models path, and route the interactions `tools` kwarg onto `metadata.tools` for the interactions path. New `_config_for_input()` strips `tools` from the config that flows into `input` via a shallow field iteration (no full model_dump). - **Excess serialization** — dropped `bt_safe_deep_copy(config)` in `_serialize_input` and `_prepare_generate_images_traced_call` (span logging already runs `bt_safe_deep_copy` at send time, so the eager copy was duplicate work — same pattern as #583 for anthropic and #585 for claude_agent_sdk). Also dropped the eager `grounding_metadata.model_dump(exclude_none=True)` in `_aggregate_generate_content_chunks`, and the eager `_materialize_interaction_value(...)` on interaction tools. - **Denylist → allowlist** in `_tool_span_input` / `_tool_span_output`. Replaced `key not in {\"id\", \"name\", \"type\", \"signature\", \"server_name\", ...}` with explicit `_TOOL_CALL_INPUT_FIELDS = (\"arguments\", \"code\", \"language\", \"url\", \"query\")` and `_TOOL_RESULT_OUTPUT_FIELDS = (\"result\", \"output\", \"outcome\", \"content\")`. Any unrecognized field on a call/result payload is dropped rather than passed through. Also trimmed narrative docstrings/comments that just described what the code does; kept the section dividers and the two \"why\" comments about interaction tool span lifetime. ## Test plan - [x] Extended existing VCR-backed tests to assert `metadata.provider == \"google\"`, `metadata.tools` presence for both `models.generate_content` and `interactions.create` tool cases, and negative assertions that tools do not leak into `input.config` / `input`. - [x] `cd py && CI=1 BRAINTRUST_TEST_PACKAGE_VERSION=latest pytest src/braintrust/integrations/google_genai/test_google_genai.py` — 45 passed - [x] `cd py && CI=1 BRAINTRUST_TEST_PACKAGE_VERSION=1.75.0 pytest src/braintrust/integrations/google_genai/test_google_genai.py` — 45 passed - [x] `cd py && CI=1 BRAINTRUST_TEST_PACKAGE_VERSION=1.30.0 pytest src/braintrust/integrations/google_genai/test_google_genai.py` — 38 passed, 7 pre-existing skips (interactions API unavailable on that version) - [x] **No cassettes re-recorded, no new mocks/fakes** — used the existing VCR-backed suite. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- sfk:created-approved-by --> Created by abhijeet <!-- sfk:slack-thread --> [Slack thread](https://starfolkai.slack.com/archives/C0AQDETAVT3/p1784308772517939?thread_ts=1784308772.517939&cid=C0AQDETAVT3) Co-authored-by: Starfolk <noreply@starfolk.ai> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
5 tasks
Abhijeet Prasad (AbhiPrasad)
pushed a commit
that referenced
this pull request
Jul 21, 2026
…ation (#608) ## Summary Aligns the OpenRouter integration with `.agents/skills/sdk-integrations/SKILL.md`, following #604 (mistral), #603 (livekit_agents), #594 (litellm), #593 (langchain), and #585 (claude_agent_sdk). Audit found two issues: - **Excess serialization.** `sanitize_openrouter_logged_value` called `bt_safe_deep_copy` on every span input, every kwargs dict, every response (for both output and metadata extraction), and every usage dict. Braintrust already runs `bt_safe_deep_copy` at log time in `Span.log_internal` (`logger.py:4699`), so every one of those was a duplicate walk. Dropped the wrapper; reads now go through a small `_get_field(obj, key)` helper (dict-or-attribute) and pydantic responses flow straight to `span.log(...)`. - **Denylist metadata → per-surface allowlist.** `_OMITTED_KEYS` was a small denylist (`execute`, `render`, `nextTurnParams`, `requireApproval`); the old `_build_openrouter_metadata` then dumped every *other* kwarg — including `messages`, `api_key`, `extra_headers`, and unknown kwargs — into `metadata`. Replaced with three request allowlists (`_CHAT_REQUEST_KEYS`, `_EMBEDDINGS_REQUEST_KEYS`, `_RESPONSES_REQUEST_KEYS`) and three response allowlists (`_CHAT_RESPONSE_KEYS`, `_EMBEDDINGS_RESPONSE_KEYS`, `_RESPONSES_RESPONSE_KEYS`), mirroring `litellm/_SAFE_METADATA_KEYS` and mistral's `_*_METADATA_KEYS` family. `is_byok` is still extracted from `usage` (it's still nested there). Also collapsed the three near-identical `_finalize_*_response` functions into one parameterized `_finalize_response(..., *, output, response_keys)`, and factored the shared model/provider stamping (`_stamp_model_and_provider`) used by both request and response metadata builders. Net: 1 file changed, +262 / −178 (net −0 after adding the allowlist tuples, but the tracing surface itself shrank meaningfully). ## Test plan - [x] **No new mocks/fakes.** All coverage stays on the existing `@pytest.mark.vcr` cassette-backed tests in `integrations/openrouter/test_openrouter.py` (chat sync/stream/async, embeddings, responses sync + async-stream, `setup_*_creates_spans`, `setup_*_is_idempotent`, auto_instrument subprocess). - [x] **No cassette re-recording.** HTTP behavior is unchanged; only in-process span shaping was touched. - [x] `cd py && nox -s "test_openrouter(latest)"` — 9/9 passed. - [x] `cd py && nox -s "test_openrouter(0.6.0)"` — 9/9 passed. - [x] `ruff check` + `ruff format --check` — clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- sfk:created-approved-by --> Created by abhijeet <!-- sfk:slack-thread --> [Slack thread](https://starfolkai.slack.com/archives/C0AQDETAVT3/p1784653510318699?thread_ts=1784653510.318699&cid=C0AQDETAVT3) Co-authored-by: Starfolk <noreply@starfolk.ai> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
metadata.provider = \"anthropic\"on theanthropic.messages.createLLM span so it satisfies the SKILL.md rule that everyllmspan carries bothmetadata.modelandmetadata.provider._serialize_content_blocksto do a shallow field copy instead ofdataclasses.asdict().asdictrecursivelycopy.deepcopys leaves (see the note inbt_json.py:61-65where Braintrust avoids it for the same reason); nestedToolUseBlock.inputdicts and any inner content dataclasses now pass through tobt_jsonat send time. The blocktypediscriminator and the ToolResult flatten/prune rules are preserved._serialize_hook_valuearound an allowlist of documented Claude Code hook input/output fields (session_id,hook_event_name,tool_name,tool_input,tool_response,prompt,message,stop_hook_active,trigger,custom_instructions,source,reason,agent_id,agent_type,continue,stopReason,suppressOutput,decision,hookSpecificOutput,systemMessage), applied only at the top level.cwd,transcript_path,agent_transcript_path, and anything else the SDK might add stay dropped by default; nested values insidetool_input/hookSpecificOutputpass through untouched.test_calculator_with_multiple_operationsto assertmetadata.provider == \"anthropic\"andmetadata.modelon every LLM span. No new cassettes.Test plan
cd py && BRAINTRUST_TEST_PACKAGE_VERSION=latest uv run pytest src/braintrust/integrations/claude_agent_sdk/test_claude_agent_sdk.py— 48 passedtest_serialize_content_blocks_keeps_malformed_text_block_payload,test_serialize_tool_result_output_flattens_text_blocks_and_errors) still cover the shallow-copy refactor's flatten/type/is_error branchestest_user_prompt_submit_hook_creates_function_span,test_tool_hooks_create_function_spans,test_hook_spans_parent_to_matching_tool_and_final_llm) exercise the allowlist end-to-end and still pass🤖 Generated with Claude Code
Created by abhijeet
Slack thread