VAPI-3990: return a bot's mark only after its queued audio has played out

[mramasubramanian-bw]

Summary

In Twilio's Media Streams protocol a bot sends a mark after queuing audio and is told, via the mark echoed back, when that audio has finished playing on the call. Bots gate their next turn on it. The bridge echoed the mark the instant it arrived, so a mark-gated bot was told its utterance had finished while the audio was still queued, and started its next turn over its own speech.

Bandwidth's StartStream protocol sends no playback-complete signal and BwStreamSource has nothing to wait on, so the bridge now tracks playout by duration.

Changes

src/streams/bridge.ts

• Each outbound media frame extends a playout clock by its mulaw duration (8 kHz mono, 8 bytes per ms).
• A mark becomes due at the clock's value when it arrived, so audio queued after a mark does not delay it. Marks return in arrival order through a single timer. Nothing queued means the mark returns at once.
• clear empties the queue and returns every outstanding mark immediately.
• stop and close() drop pending marks, since nothing queued will play.
• Marks sent back now carry a sequenceNumber, as Twilio's do.
• New pendingPlayoutMs() for tests and diagnostics, and exported mulawPayloadDurationMs(), which derives the decoded length from the base64 string so no Buffer is allocated per frame.
• New BridgeOpts.playoutLatencyPadMs, added to the due time of any mark that has audio ahead of it. Default 0 until real measurements exist (see Not verified).

AGENTS.md: note the duration-based playout model under the Stream entry.

Deviation from the ticket

VAPI-3990 says a clear should discard pending marks. Twilio's docs say the opposite:

This empties all buffered audio and causes any mark messages to be sent back to your WebSocket server.

The documented behavior is implemented. A mark-gated bot that barges in and never receives its mark would otherwise wait forever.

The ticket's Expected behavior section has been corrected to match, with a dated note, so QA tests against the right wording.

Before / after

Bot queues 1 s of audio then a mark:

	mark returned
Before	~1 ms after it was sent
After	~1000 ms after it was sent

Bot queues 5 s of audio, a mark, then sends clear 500 ms later:

	mark returned
Before	~1 ms after it was sent
After	~1 ms after the clear

Verification

• npm run typecheck && npx vitest run: 44 files, 330 passed, 6 skipped. The bridge test files were run three times in a row with no flakes.
• test/streams-wire.test.ts: the test that pinned the synchronous echo is replaced by cases for the clock math, immediate echo with nothing queued, holding a 300 ms mark, ordering, non-delay by later audio, stop, close(), clear, the clock resetting after clear, the latency pad, and the base64 length math against a real decode for every padding shape.

Not verified

The clock starts when a frame reaches the bridge, but the caller hears it one-way network latency plus Bandwidth's jitter buffer later, so marks will still return slightly early on the live path. The bias is in the same direction as the original bug but likely tens of milliseconds rather than the full utterance. playoutLatencyPadMs exists to absorb it once VAPI-3991 provides a live source to measure against; until then it defaults to 0.

Reference: Twilio Media Streams WebSocket messages, mark and clear.

[bwappsec]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.