There is a specific kind of awkward silence that only happens when you say goodbye to an AI on the phone and neither of you hangs up.
You finish what you were doing. You say, "Alright, talk to you later, bye." The voice on the other end responds cheerily: "Goodbye! Have a great afternoon!" And then... nothing. Just the faint hiss of an open telephony line. You hold the receiver to your ear for five seconds, then ten seconds, waiting for the familiar click of a disconnected call.
Eventually, you pull the phone away from your face, stare at the screen, and press the big red button yourself.
It is a tiny UX irritation, but it points straight at a deeper architectural challenge in real-time voice agents: language models are probabilistic conversational engines, whereas terminating a telephone call is a hard, deterministic state transition. If you rely entirely on the model to decide when a phone call is over, you are setting yourself up for a lot of lingering silences.
The two-brain setup
To understand why this happens, it helps to look at how real-time voice agents are actually structured.
In my homelab setup, incoming phone calls don't hit a single monolithic model. They run on a two-brain architecture:
- The Realtime Voice Front-End: Powered by Gemini Live over a bidirectional WebSocket. It ingests continuous 16 kHz audio, manages voice activity detection (VAD), handles interruptions when I speak over it, and streams back synthesized audio with sub-second latency. It excels at conversational cadence and tone.
- The Main Agent Backend: The heavy thinking brain. When I ask a question that requires inspecting my local workspace, querying tools, checking system status, or recalling stored context, the voice front-end delegates that work to the backend via a specialized consult tool (
openclaw_agent_consult).
This split is essential. Real-time audio models are built for speed and spoken fluency; they are not designed to hold massive tool schemas, orchestrate complex reasoning chains, or manage local filesystem state while maintaining low-latency audio streaming.
The voice front-end acts like a receptionist who answers the phone with impeccable conversational manners, while the main backend is the engineer in the back room with the keys to the servers.
When the handoff broke
The first problem I ran into this week was on the delegation boundary.
I placed a voice call and asked a question that required consulting the backend. The voice front-end politely acknowledged the request—"Let me look into that for you"—and then went silent. After waiting a couple of minutes with no answer, I checked the logs. The voice session reported that the consult request had timed out.
My immediate suspicion was backend congestion. Did the main agent hang? Was a tool call stuck?
When I inspected the backend metrics, the truth was the exact opposite: the backend had received the consult requests and completed all of them in 0.7 to 1.6 seconds. The responses were generated almost immediately.
The culprit was an experimental "fast context" optimization inside the voice plugin. It was designed to intercept certain requests and resolve them locally from cached state before reaching the real backend. Instead of speeding up queries, it created an ambiguous execution branch that swallowed the handoff and left the audio stream waiting on a promise that never resolved.
The fix was straightforward: delete the clever shortcut. Every delegated question now travels a single, uncompromised path: Gemini Live → main backend consult → spoken result. I also wrapped the consult call in a structured handler so that if the backend ever genuinely fails or restarts, the voice agent delivers a brief, natural spoken explanation instead of holding the line in dead silence.
With delegation fixed, the agent could answer complex questions reliably. But when the conversation ended, the second bug appeared.
Why models don't hang up
To allow an AI voice agent to terminate a call, you provide it with an end_voice_call function tool. In theory, the instructions are simple: when the user says goodbye or signals the conversation is finished, say farewell and call the tool to drop the telephony session.
In practice, conversational LLMs are notoriously inconsistent at triggering terminal tools.
When a human says "Okay, we'll talk later, goodbye," the model processes that utterance as a social prompt. Its primary objective is to complete the conversational turn naturally. So it generates a polite farewell response: "Sounds good, Tun! Take care!"
Because generating that reply completes the conversational turn from the model's perspective, it often feels no urgency to invoke the end_voice_call tool alongside the speech. It assumes the human on the other end will simply hang up their phone, exactly as humans do in everyday conversations.
The result is a zombie call. The AI has said goodbye, the user is waiting for the line to disconnect, and the connection remains alive, quietly burning telephony minutes and streaming ambient room noise back and forth.
The server-side safety net
You cannot fix a probabilistic omission with more prompt emphasis. Telling an LLM "YOU MUST CALL THIS TOOL WHEN SAYING GOODBYE" in bold capital letters might boost reliability from 70% to 85%, but in telephony, an 85% success rate still means one out of every six calls gets stranded on an open line.
The proper fix is to take the terminal decision away from the model's discretion and place it into deterministic server-side code.
Here is how the fallback works:
- Speech-Act Transcript Inspection: When the realtime voice handler finalizes a user speech segment, a lightweight regex scans the transcript for unambiguous closing phrases (
goodbye,bye,talk later,we'll talk later,hang up,end the call). - Delayed Termination Schedule: If a closing phrase is detected, the server immediately registers a graceful delayed hangup with the call manager—defaulting to a 5-second window.
- Graceful Playout: The 5-second delay gives the voice model enough room to synthesize and stream its natural farewell sentence across the WebSocket without being cut off mid-syllable.
- Deterministic Teardown: Once the timer expires, the call manager invokes the SIP/Twilio hang-up API directly from the server.
If the model happens to call the end_voice_call tool on its own, the same delayed termination pipeline is used, deduplicating the request. If the model forgets the tool entirely, the server-side regex acts as a guaranteed safety net.
Separation of concerns
The lesson here goes beyond phone calls.
Language models are exceptional at natural language understanding, tone modulation, and context interpretation. But state transitions that carry real operational consequences—dropping a carrier connection, committing a transaction, releasing a lock—should never rely solely on a model's whim to invoke a function.
By pairing Gemini Live's natural speech capabilities with deterministic server-side guards, the voice line now feels completely natural:
- You ask a complex question, the backend answers in under two seconds.
- You say "talk to you later, bye," the assistant says goodbye back, and five seconds later, the line drops cleanly.
When building voice systems, let the AI handle the conversation. Let deterministic code handle the dial tone.
