feat(voice): add respondToGather to collapse the conversational-turn round trip - #14
Open
michaela-band wants to merge 1 commit into
Open
michaela-band wants to merge 1 commit into
michaela-band wants to merge 1 commit into
Conversation
…round trip Every turn of a live agent-driven conversation is a race against Bandwidth's answer/gather callback retry window: the callback lands, and the agent has a bounded window to queue a BXML response via respondToCallback before the call gives up and hangs up. The current pattern for a conversational turn is two separate tool calls -- generateBXML, then respondToCallback -- with a full model reasoning step in between. Each of those is wall-clock time that narrows the window before the race is lost. Ran an internal PoC (Claude Code + this server, live Bandwidth Build account) attempting exactly this pattern for a scripted greeting (not even a full conversation, just turn zero). 8 of 9 live call attempts connected and hung up with no audio delivered -- the agent's tool-call round trip lost the race against the callback retry window every time but one. respondToGather(call_id, verbs, auto_gather) merges generateBXML + respondToCallback into a single tool call: generate the BXML from verb descriptions and queue it as the callback response in one round trip. Same first-write-wins semantics as respondToCallback. Does not by itself guarantee winning the race -- it removes one full model-reasoning gap from the loop, which is the highest-leverage, lowest-risk fix available without changing the callback timing model itself (a separate, bigger conversation -- see the accompanying spike proposal for making the server's callback loop respond fast enough for genuine multi-turn conversation without depending on the outer agent's per-turn latency at all). - src/tools/voice.py: new respond_to_gather_flow() (mirrors the generate_bxml_flow / respond_to_callback_flow pattern already in this file) plus the respondToGather tool registration. - src/profiles.py: added to the voice profile. - src/instructions.py: voice section now tells agents to prefer respondToGather for every turn after the first. - src/specs/AGENTS.md, README.md: tool inventory updated. - test/test_bxml.py: two new tests -- generates+queues in one call, and first-write-wins semantics match respondToCallback. Full suite: 119 passed (117 existing + 2 new), no regressions.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
respondToGather(call_id, verbs, auto_gather)— a single tool that generates BXML from verbs and queues it as the callback response for an active call, in one round trip. It's the merge of the existinggenerateBXML+respondToCallbacktwo-step, kept as-is for callers who still want them separate (e.g. inspecting BXML before sending).Why
Every turn of a live, agent-driven voice conversation races Bandwidth's answer/gather callback retry window: the callback lands, and there's a bounded window to queue a BXML response via
respondToCallbackbefore the call gives up. The current two-tool-call pattern (generateBXMLthenrespondToCallback, with a full model-reasoning step in between) burns real wall-clock time inside that window.This isn't theoretical — ran an internal PoC (Claude Code driving this server against a live Bandwidth Build account) attempting exactly this pattern for a one-line greeting on call answer (turn zero, not even a full back-and-forth). 8 of 9 live call attempts connected and hung up with no audio delivered — the agent's tool-call round trip lost the race against the retry window every time but one.
Collapsing the two calls into one doesn't guarantee winning the race (the remaining latency is model inference time, which this doesn't touch), but it removes one full reasoning gap from the loop — the cheapest, lowest-risk fix available without changing the callback timing model itself. Michaela's asked me to separately scope a bigger fix (fast, non-agent-driven turn generation inside the callback handler, for genuine multi-turn conversation) as a spike in DX operations — this PR is the narrower, immediately-actionable half of that conversation.
What changed
src/tools/voice.py—respond_to_gather_flow()(mirrors the existinggenerate_bxml_flow/respond_to_callback_flowpattern in this file) plus therespondToGathertool registration. Same first-write-wins semantics asrespondToCallback.src/profiles.py— added to thevoiceprofile.src/instructions.py— voice section now tells agents to preferrespondToGatherfor every turn after the first, and explains why (the race).src/specs/AGENTS.md,README.md— tool inventory updated.test/test_bxml.py— two new tests: generates + queues BXML in one call, and first-write-wins matchesrespondToCallback's behavior.Testing
Full suite: 119 passed (117 existing + 2 new), no regressions. Also smoke-tested the flow function directly against a live
EventStoreinstance to confirm the queued BXML round-trips correctly.Flagging for review rather than assuming this is the final shape — happy to discuss whether the tool should also read the pending gather event itself (folding in
getCallbackEvents) rather than leaving that as a separate call, if that's a more natural round-trip reduction.