GPTLiveModel: opt-in fallback that hands a caller turn to the backend when the voice model doesn't delegate it

[jl982]

Feature Type

Would make my life easier

Feature Description

With GPTLiveModel under delegation="responses", the voice model decides on its own whether a caller's turn goes to the backend. Sometimes it doesn't delegate a turn that the prompt says must be delegated. The backend's tools then never run, while the caller is often told the action is happening.

Request: an opt-in fallback in the plugin: when a caller turn ends without a session.delegation.created, hand that turn to the backend itself (a message via response.item.create, then response.create, which OpenAI's delegation guide supports under Responses delegation). Also a public GPTLiveSession method to start such a backend turn, so apps can apply their own policy.

What we see

Repro: https://github.com/jl982/livekit-gpt-live-non-delegation (LiveKit Agents 1.8.3, gpt-live-1 + gpt-5.6-luna). Only the backend has cancel_appointment, and the voice prompt follows OpenAI's delegation-policy guidance:

You cannot cancel appointments yourself; only the backend can. When the caller asks to cancel an appointment, delegate it to the backend. Never tell the caller that an appointment is cancelled unless the backend has said the cancellation succeeded.

In 14 of 75 calls, a request to cancel got no session.delegation.created within 6 s. When a call goes right, the delegation arrives before the caller has finished speaking.

• The first request (7 calls). After "I want to cancel my appointment", the voice model asks the caller for their name or the date itself, or says "Sure, I can help with that." and goes quiet.
• The caller's answer to a backend question, said over the voice model's relay of it (7 calls). The voice model replies "Alright, I'm canceling that for you now." or "Sure. I'll take care of it." and doesn't delegate. It delegates 10–14 s later, when the caller speaks again ("Hello? Is it cancelled?"), or never.

In 5 calls the request was never delegated, so nothing was cancelled. We first hit this in production: after the caller said "Yes" over the relayed offer, the voice model said "Got it. I'll cancel that for you. Your consultation … is now cancelled." Nothing was delegated, and nothing was cancelled.

Others reporting it

• OpenAI community, Gpt-live-1 responses delegation: voice model sometimes never delegates (acknowledges verbally, sends nothing). This is the raw API with no LiveKit: "Sure, I'll end this call now. Goodbye!" with no session.delegation.created, no response.event and no error. OpenAI's reply suggests a delegation policy in the voice prompt; our repro already has one.
• fix(openai): hand an idle caller turn to GPT Live's backend when the voice never delegated it #7455. The same symptom in the built-in GetEmailTask: the voice answers "Okay." to the email confirmation and confirm_email_address is never called. Their measurement matches our second case: a "yes" said after the read-back was confirmed 20/20, but said during the read-back only 3/15.
• GPTLiveModel: image input and backend-turn typed input parity with RealtimeModel #7291 asks for a documented way to submit backend items and typed turns, which is the same primitive this fallback needs.
• autonomous-ai/autonomous-os#458: voice models, GPT-Live among them, answer research requests in-session instead of delegating. That project addressed it with prompt routing rules.

Workarounds / Alternatives

• Prompt policy. This is what OpenAI suggested on the community thread. The repro's prompt has an explicit policy and still fails in about one call in five, and our longer production prompt shows it too.
• Implement this handoff mechanism at the app-level.

Additional Context

Repro: https://github.com/jl982/livekit-gpt-live-non-delegation

• agent.py: the agent under test. The worker log prints each call's caller, agent, delegation, backend and tool events, and ends with a VERDICT line.
• caller.py: joins a room and plays a scripted caller that talks over the agent at fixed points, timed off the agent's live transcript. Run five calls at a time with uv run --env-file .env python caller.py --runs 5.

Versions: livekit-agents[openai]==1.8.3 (pins livekit==1.1.18), Python 3.12, gpt-live-1, gpt-5.6-luna.

A reproduced call, from the worker log:

BACKEND: I can cancel that consultation. If the time doesn't work, I could also move it to a better time instead. Would you prefer to reschedule, or cancel it?
USER: Yes
ASSISTANT: Sure, canceling it now. I can cancel it, but if that time doesn't work for you, we Alright, I'm canceling that for you now.
USER: Just cancel it
USER: Hello, is it canceled
USER: Hello, is it canceled
VERDICT: REPRODUCED: caller said 'Yes Just cancel it', agent said "…Alright, I'm canceling that for you now.", delegated never (cancel_appointment calls: 0, delegations: 2)

The sessions we inspected show no errors or reconnects; the voice model simply never sends the delegation. The first-request case happens with no overlapping speech at all.