At 1 a.m. on a weekday, I was wiring a new model provider into my home agent, and everything was going great — right up until it wasn't.
First request: 200, tokens streaming. Second request: 400 with a completely empty body. Third: 200 again. Fourth: 400.
Not random. Not "sometimes." A perfect metronome. Success, failure, success, failure, all night long.
If you've ever debugged something that fails on a rhythm, you know it's a special kind of maddening. Random failures are honest — they admit they're random. Rhythmic failures are trying to tell you something, and what this one was telling me took two hours and one genuinely embarrassing wrong fix to hear.
The setup
I run a self-hosted LLM proxy — an OpenAI-compatible endpoint on a small VPS that fans out to a handful of model subscriptions behind it (Kimi, GPT, Gemini, Qwen, Anthropic), with a routing chain that picks who handles what. My agent talks to one URL and doesn't care about the zoo behind it.
Getting the provider registered was its own little saga. Adding a custom model provider to the agent means three config surfaces have to agree: the provider block in the main config, a per-agent overlay file that silently shadows the global config, and an allowlist that controls which models show up in the picker. Miss one and you get my favorite genre of bug — the model that works but doesn't appear, or appears but doesn't work. I got all three in sync, pruned a pile of junk catalog entries out of the picker, and felt smug.
Then the metronome started.
The fix that worked — by accident
The proxy's own error format is nice when it bothers to use it: clean codes like M003 for a malformed key. But these 400s came back with empty bodies — the proxy was passing the upstream failure through raw, before anything had a chance to wrap it in a message. An empty error is its own kind of clue: something died below the layer that produces error messages.
The first suspect was plausible enough that I bit immediately. One of the models behind the proxy requires a reasoning_content field to be present when you replay an assistant message containing tool calls — which is exactly what an agent does after every tool result. My provider entry declared the model as non-reasoning, so the agent was stripping that field on replay. First request fine; post-tool-call replay, rejected.
One flag flip — reasoning: true — and the next request succeeded.
And the one after that failed.
Here's the lesson I want tattooed on the inside of my eyelids: a fix that "works" on an alternating failure might not be a fix at all. The flag was genuinely correct — the field really was required — but it wasn't the cause. It just happened to land on the good beat. If I'd tested twice instead of once, I'd have known immediately. I tested once, declared victory, and got humbled ten minutes later.
The A/B test that pointed at the box
Time to remove variables. I bypassed the entire agent stack with plain curl: same endpoint, same payload, no tool calls, no reasoning fields, nothing clever.
1.
Same metronome. Which meant the model API wasn't guilty, the agent wasn't guilty, and my reasoning-field theory — correct but irrelevant — wasn't the story. The failure lived somewhere between curl and the backend. On my box. In infrastructure I'd written a while ago and stopped thinking about.
The culprit: a keepalive race hiding in plain sight
The proxy sits behind nginx, and the nginx config has an upstream block with keepalive 32 — a pool of persistent connections to the backend, so every request doesn't pay for a fresh TCP handshake. Great. Except the same config also had a map block, written ages ago to support WebSocket upgrades, whose default case sent Connection: close on every ordinary request.
Those two lines contradict each other. The keepalive pool caches connections expecting reuse; the Connection: close header tells the backend to start closing. So nginx would hand a request a pooled connection the backend was already halfway through shutting down — a textbook race — and the request died with an empty 400 before anyone could produce an error message.
And here's the tell I should have caught an hour earlier: near-perfect alternation is the signature of rotating state, not of chance. The pool rotates through connections, so every other request drew a dying one. It wasn't two servers. It wasn't a coin flip. It was a carousel, and I'd been watching it spin the whole time.
The fix was the recipe nginx has recommended for a decade: clear the Connection header instead of setting it to close. In my map block, that meant changing the default from "close" to "" — one value, one reload, nginx -t happy, metronome dead. While I was in there I also found a proxy_read_timeout 300s on a long-lived streaming endpoint that was quietly killing live feeds after five minutes; that got a 24-hour timeout and buffering turned off. Two bugs, one file, neither of them where the symptoms pointed.
What I'm keeping from this
Rhythmic failure is diagnostic. Random says "something's flaky." Alternating says "something rotates" — a connection pool, a pair of backends, a round-robin DNS entry. Find the carousel before you trust any theory.
Test fixes an even number of times. A fix that lands on the good beat of an alternating failure looks exactly like a real fix. Two requests would have saved me an hour.
Empty errors mean the death happened lower. No body, no code, no message — the connection died before the error-producing layer ever ran. Stop debugging the application and start debugging the transport.
One coda, because timing is funny: nginx 1.29.7, released in March, flipped the defaults so proxy connections use HTTP/1.1 with keepalive and no Connection header out of the box. This entire category of footgun — the one I just fell into — is now a legacy-version story. If you're running an older nginx with keepalive in an upstream block, go look at what you're doing with the Connection header. The recipe is two lines, and it's been right for ten years.
The most expensive bug I hit this month cost one word to fix. It just took two hours to find the word, because it was sitting in a config file I'd written for a completely different reason and then stopped reading.
