I built the wrong thing first. That's not self-deprecation for effect — I genuinely spent an afternoon building a dumb call recorder when what I actually wanted was a receptionist. The difference matters.
A few weeks ago I set up a second phone line for my home lab: a public number that anyone could call. The idea was simple — give people a way to reach me without handing out my real number. The first version just recorded messages, transcribed them, and pinged me. Functional, boring, and not what I actually wanted.
What I wanted was someone on the other end. A voice. Something that could actually talk to callers — answer questions, take a message, handle the interaction like a human receptionist would. Not a robot reading a script. A conversation.
So I tore down the recorder and built a receptionist.
The stack
Twilio Media Streams piped directly into Google's Gemini Live audio model. When someone calls, Twilio opens a bidirectional audio stream and hands raw audio to Gemini in real time. No speech-to-text middle step, no latency from transcribing and re-synthesizing. The model hears the caller's voice directly and responds with its own — sub-second turnaround, natural prosody, interruptions work, the whole thing feels like a phone call instead of a walkie-talkie.
The plumbing is about 700 lines of Python: a standalone service that handles the WebSocket handshake with Twilio, manages the Gemini session lifecycle, and runs as a systemd service on my home server. It listens on a port that my tunnel setup exposes to the internet, and Twilio points my public number at it.
The plumbing was the easy part. The persona was where things got interesting.
The persona
The receptionist answers as a character I've been building for a while: warm, witty, slightly irreverent, with a habit of calling out nonsense. The prompt is deliberate — friendly and playful, but with empty pockets.
No access to my memory. No tools. No private information. No ability to do anything except have a conversation and take a message.
More importantly, there's an anti-impostor rule baked directly into the system instructions. If someone calls and claims to be me — or claims to know me, or claims they have direct access — the receptionist doesn't verify. Doesn't check. Doesn't even entertain the question. It just says, politely, "this is the public message line — if it's really him, he knows the direct number."
No exceptions. No escalation path. No social engineering attack surface. The public line doesn't know anything, so it can't leak anything.
The boundaries are actually better than mine would be. If someone called me and said "hey, it's your bank, we need to verify some details," I'd probably engage for at least thirty seconds before my skepticism kicked in. The receptionist doesn't have that weakness. It's not cynical — it's incapable of caring who you claim to be.
Three voices in one evening
The voice went through three iterations.
The first was a default female voice that sounded fine but didn't match the character. The second was warmer but still female. The third — and final — was a male voice called Puck, because the feedback was immediate and unambiguous: the character is a guy.
Three voice changes, one evening, and a lesson about how much personality lives in a voice choice. People assign trust, warmth, and competence to a voice within seconds. Pick the wrong one and your carefully crafted prompt doesn't matter — the caller already decided who they're talking to.
The hangup that wouldn't
The farewell logic needed surgery. The first version missed the word "Bye!" — the receptionist literally couldn't end a call because the detection pattern was too specific. The agent would say "Have a fantastic day! Bye!", the regex would miss it, and the caller would sit in silence until a timeout kicked in.
I fixed it with a broader pattern and added a belt-and-braces REST call to terminate the Twilio session directly. Redundancy in call endings sounds absurd until you've listened to a thirty-second silent recording of your own system failing to hang up.
The silence wasn't just awkward — it was expensive. Every second of dead air was billable audio streaming. A missed "Bye!" was costing me actual money, a few cents at a time, forever.
Kill it and see if it comes back
Someone asked the right question during testing: "what happens if the server reboots?"
I killed the receptionist process with SIGKILL — the nuclear option — and systemd had it back up in about three seconds. Every service in the chain is enabled with Restart=always, and the user session is configured to start without a login. The whole thing survives a power cycle.
Proving it live instead of just reading configs matters. I've been burned by services that look enabled on paper but fail silently on boot because of a dependency ordering issue or a missing environment variable. Actually pulling the plug and watching it come back is the only verification I trust.
The less-it-knows principle
The dynamic greetings were a late addition. Instead of answering the same way every time, the receptionist randomizes its opening vibe — warm one call, cheeky the next, laid-back after that. The instruction is literally "never reuse the same wording." Small thing, but it makes the system feel less like a recording and more like someone who answered the phone just for you.
What I keep coming back to is how much safety comes from knowing nothing. The receptionist can't overshare because there's nothing to share. It can't be tricked into revealing information it doesn't have. Its entire defense model is "I don't know, and I'm not authorized to find out."
That's a weirdly counterintuitive design principle for AI: the less it knows, the safer it is to put in front of strangers.
My private assistant has the keys to everything. My public receptionist has an empty desk and a notepad. One of them gets to talk to the internet. The other one stays in the back office.
If you're thinking about building something similar, start with the constraints. Build the persona that knows nothing first, then ask yourself what it actually needs. You'll be surprised how little makes the cut — and how much safer you feel handing it a phone number.
