Skip to content

feat(mllm): add ThunderPhone speech-to-speech extension - #2316

Open
kolchinski wants to merge 1 commit into
TEN-framework:mainfrom
kolchinski:add-thunderphone-mllm
Open

kolchinski wants to merge 1 commit into
TEN-framework:mainfrom
kolchinski:add-thunderphone-mllm

Conversation

@kolchinski

Copy link
Copy Markdown

Summary

Adds thunderphone_mllm_python: ThunderPhone voice agents as a speech-to-speech (MLLM) extension, alongside openai_mllm_python, gemini_mllm_python, azure_mllm_python, glm_mllm_python and stepfun_mllm_python.

ThunderPhone's Realtime WebSocket API speaks the OpenAI Realtime protocol, so the extension mirrors openai_mllm_python (same mllm-interface.json, same event mapping) and adds the ThunderPhone-specific parts:

  • the endpoint, its query parameters (saved agent, engine, language, caller/callee numbers, wire audio, call_events) and sk_live_ keys;
  • saved agents (agent_id): prompt, voice, engine, languages, tools and greeting live on ThunderPhone, the graph only moves audio;
  • inline sessions (prompt, product, voice, language): graph tools are collected and sent in the single session.update that starts the call, because ThunderPhone freezes instructions, tools and voice at that point (the extension waits up to 500 ms for late tool_registers unless audio arrives first);
  • ThunderPhone's call.* events (hang-up reason, transfer, keypad) are parsed and logged, and call.ended stops the extension from reconnecting, since a reconnect would start a new, separately billed call.

Also registers the extension in examples/voice-assistant-realtime (manifest dependency + README provider list) and adds THUNDERPHONE_API_KEY / THUNDERPHONE_AGENT_ID to .env.example.

Validation

  • tests/test_realtime.py (no credentials): 6 passed, covering the connect query, config validation, the inline session.update, and parsing of ThunderPhone's session.created, GA conversation.item.added and call.* events.
  • tests/test_extension.py: a credential-free load test through AsyncExtensionTester, plus a live session-ready test when THUNDERPHONE_API_KEY is set.
  • Verified live against the ThunderPhone API by driving ThunderPhoneRealtimeExtension directly (base class stubbed to record emits): saved-agent greeting → caller transcript → reply → hang-up (call.ended, no reconnect); inline session with a graph tool → mllm_server_function_callfunction_call_output → reply using the tool result → hang-up.

Disclosure: I'm the founder of ThunderPhone.

ThunderPhone's Realtime WebSocket API speaks the OpenAI Realtime protocol, so
thunderphone_mllm_python mirrors openai_mllm_python and adds the ThunderPhone
parts: endpoint and query parameters (saved agent, engine, language, phone
numbers, wire audio, call events), sk_live_ keys, saved agents that bring
their own prompt/voice/tools, one complete session.update for inline sessions
(ThunderPhone freezes instructions and tools when the call starts), call.*
events, and no reconnect after the platform ends the call.
@kolchinski

Copy link
Copy Markdown
Author

Checking in on this. The branch is still mergeable and the only red check is the repo's own claude-review job, which needs a maintainer-side secret rather than a change here. Happy to rebase or rework anything if there's feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant