This weekend I called a phone number and reached a system running in my own homelab.
Not a prerecorded menu. Not “press one for support.” A real, full-duplex conversation that could hear me, answer naturally, load familiar context, and consult the same knowledge I use from text chat.
The call worked in both directions by the end of the night. Getting there taught me something slightly embarrassing: the clever voice model was the easy part. The hard part was making five ordinary pieces of infrastructure agree that the call was allowed to exist.
The stack is simpler than it sounds
I used the OpenClaw Voice Call plugin with Twilio for the phone network and Gemini Live for realtime speech.
The path looks roughly like this:
Phone call → Twilio → public webhook → my home gateway → Gemini Live → audio back through Twilio.
Twilio’s bidirectional Media Streams carry call audio over a secure WebSocket. Gemini Live receives audio and produces audio directly, so the conversation does not have to crawl through a separate speech-to-text, text-generation, and text-to-speech chain for every turn. That matters because even a good answer feels broken when every pause lasts three seconds.
OpenClaw sits between those services and my existing agent. It handles the call session, caller policy, context, and the bridge back to the system I already use. In theory, it is a clean little diagram.
In practice, diagrams never include the part where you stare at a 502 error.
Localhost means local, unfortunately
My first real blocker was wonderfully boring. The voice service was listening on 127.0.0.1 by default. That is sensible when everything lives on one machine, but my public Cloudflare tunnel reaches the gateway across my home network.
The tunnel could reach the box. The box was healthy. The port existed. Yet every request returned 502 because the service was only accepting connections from itself.
Changing the bind address to 0.0.0.0 fixed the route immediately.
That does not mean “open the service to the universe.” It means bind it where the reverse proxy can reach it, then enforce access at the firewall, tunnel, and application layers. The distinction is basic networking, but defaults are good at making basic networking look mysterious.
The next failure came from the opposite direction: too much protection. Cloudflare saw Twilio’s webhook requests, decided they looked suspicious, and blocked them. The fix was a narrowly scoped rule for the voice hostname, while keeping Twilio signature verification enabled. Twilio explicitly recommends validating the X-Twilio-Signature header so a random POST cannot pretend to be a call.
I also kept an inbound caller allowlist. A phone number connected to a conversational system is not something I want strangers stress-testing at 3 AM.
Telephone audio is still telephone audio
There is also an audio mismatch hiding under the friendly APIs.
Twilio sends phone audio as 8 kHz μ-law data. Gemini Live works with raw PCM audio and produces higher-rate output. Something has to translate between those worlds, preserve timing, buffer chunks, and clear queued audio when I interrupt.
The plugin handles that plumbing, but understanding it explains several behaviors that otherwise feel supernatural. A delayed response may be model latency, network latency, audio buffering, or all three taking turns. An interruption only feels natural if the system can stop audio that was already queued for playback.
Google’s Live API documentation describes native audio, turn detection, multiple voices, and session resumption. Those features make the conversation sound good. Twilio’s marks, media messages, and clear events make it behave like a phone call instead of two voice notes colliding.
The glamorous demo is the voice. The product is the buffering.
One agent, not two strangers
The most important decision had nothing to do with codecs.
I did not want a separate phone bot that knew nothing about my text conversations. That creates the “two strangers” problem: one system receives a trigger to place a call, another system talks, and neither really knows what happened on the other side.
My setup uses Gemini Live for the fast conversational layer, but it can consult the main agent for substantive facts and shared memory. The voice session also receives the relevant identity and user context when it starts. With per-phone session scope, a later call can continue from an earlier one instead of treating every ring as a first meeting.
That separation feels right. Realtime speech needs speed, interruption handling, and a natural voice. Deeper questions need the same memory and reasoning I already trust elsewhere. One layer does not have to pretend it is excellent at both.
I also set the call tools to safe, read-only access. A voice interface is wonderful while driving or walking, but it is a terrible place to confirm a destructive action. My rule is simple: voice for talking, text for doing. If a call surfaces something actionable, I can handle it later through the channel with a screen, an audit trail, and fewer opportunities for “yes” to be misheard.
What actually made it feel finished
A successful call proves that the audio path works. Preserving context is what turns the phone from a demo into another doorway into the same system. The number, the realtime voice, and the WebSocket bridge are replaceable parts. Continuity is the feature.
If I rebuilt this tomorrow, I would start with four questions before touching the voice model:
- Can the carrier reach the webhook?
- Can I verify every incoming request?
- Where does conversation memory live?
- What is the caller allowed to do?
Get those right and the voice layer is almost pleasant. Skip them and you have built an impressive stranger with a phone number.
