There's a specific kind of awkwardness reserved for voice interfaces: you stop talking, and nobody notices.
That was my life for a few weeks, on the phone with my own assistant. I'd say something, pause to think, and the silence would just stretch — or I'd call from somewhere noisy and the whole conversation would feel off. Meanwhile my public line, which a fair number of strangers have called by now, handled pauses perfectly.
Both lines run on the same home server. Both use Google's Live API with Gemini 2.5 Flash in native audio mode. Same model, same phone network, same everything — except one line couldn't tell when I stopped talking and the other could.
So I did the thing you do when two identical systems behave differently: I stopped guessing and diffed the config.
The ear is not the brain
A quick tour of the setup, because the fix only makes sense if you know the shape of the thing.
The private line is my full-power assistant: memory, tools, the works. The public line is a receptionist persona I built with empty pockets — no memory, no tools, nothing private — because I wanted a number anyone can call without it becoming an attack surface. Both answer calls through the same voice stack, and both rely on a component most people never think about: a small judgment engine called voice activity detection, whose entire job is to answer one question.
Has this person finished talking yet?
Google's Live API does this server-side. When you speak, the detector commits a start of speech. Then it watches the audio for a continuous stretch of silence, and once the stretch is long enough, it commits an end of speech and hands the floor to the model. Three numbers shape how it behaves:
- start-of-speech sensitivity — how eagerly it decides you've started talking
- end-of-speech sensitivity — how eagerly it decides you've stopped
- the silence window — how many milliseconds of quiet count as "done"
My private line had all three overridden. I'd added the overrides myself a while back, because I wanted the assistant to feel snappier on calls. Start: high. End: high. Silence window: 500 milliseconds. On paper, "more sensitive" reads like "better at listening."
It doesn't mean that. It just means listening for different things.
Why "more sensitive" meant "never done"
Here's the failure mode, which is almost embarrassing once you see it.
High start sensitivity means the detector commits starts of speech more often — including for things that aren't speech. Traffic hum. Wind against the mic. The general background noise a phone mic picks up on an ordinary day. Every one of those false starts reset the silence clock. So the stretch of quiet that was supposed to tell the system "he's done" never accumulated. The turn never closed. From my side, that felt like the assistant not noticing I'd stopped talking — because mechanically, it never got the signal.
I had made the ear so good at catching me that it could no longer tell when I'd gone quiet.
The public line had no overrides at all. It ran on the API's defaults — defaults tuned by people who have tested turn-taking on rather more phone calls than I ever will — and it worked fine. My tweaks hadn't improved the private line's ear. They'd broken it, in exactly the environments I used it most: outside, with noise around.
The fix was subtraction
I gave myself two options.
- Option A: delete all three overrides and fall back to the same defaults the public line already used.
- Option B: keep the sensitivity settings and only trim the silence window, accepting a slightly twitchier ear in exchange for faster turn endings.
I took A. If the defaults are good enough for the line strangers call, they're good enough for mine. If quiet rooms ever start feeling too patient, I can add back just the 500-millisecond silence timer without touching the sensitivities.
Then came the boring discipline that saves you when config changes go sideways:
- Back up the config file first, timestamped, so rollback is one command away.
- Make the change small and reviewable — three lines deleted, nothing added.
- Restart the service and health-check: gateway up, plugins loaded, live config verified so I knew the overrides were actually gone.
- Schedule a one-shot verification check a few minutes later, belt and braces, to confirm everything stayed healthy after the restart.
And finally the only test that matters: a real call. I dialed in, said a sentence, stopped mid-thought on purpose, and waited. The line waited with me, then closed my turn properly. No jumping the gun, no zoning out.
One more verification, because I've been burned by "this shouldn't affect anything else" before: the public receptionist is a separate process that doesn't read the config file I edited. I checked its uptime and its port anyway — untouched, running since the day before. The two lines are architecturally strangers; I confirmed it empirically regardless.
What I'm taking from this
Three things, in order of how much they embarrassed me.
First: when two identical systems behave differently, diff the config before you suspect the model. The model was never the problem. It was the same on both lines, doing exactly what it was told.
Second: defaults are a baseline tested against reality. My overrides were a vibe-based change with no testing behind them, and they survived as long as they did only because I mostly called from quiet places.
Third: "more sensitive" is not a synonym for "better." On a phone call, with noise around, sensitivity at the start of a turn is exactly what you don't want — it turns the world into speech and erases your pauses.
The whole fix was three deleted lines, one timestamped backup, and a single awkward test call where I stopped talking on purpose and counted seconds.
Next time something in your setup behaves badly, what's the first thing you diff?
