Last week I opened my blog's admin panel and found the same post sitting there nine times.
Same title. Same body. Nine slightly different slugs, because my CMS helpfully auto-suffixes duplicates. A scheduled job I run had fired off its "create post" request, something in the middle hiccuped, the request got retried — and every single retry became a brand new, very live, very public post.
Cleaning it up was annoying but easy. The interesting part is what I did afterward, because this failure mode has a name, a known cure, and a lesson that applies to way more than my little blog: any "create" operation that can be retried will eventually be retried, so it had better be idempotent.
The actual bug was a missing assumption
Here's the thing I got wrong. I had implicitly assumed that "create post" was a one-shot operation: send request, get post, done. That's true exactly until the network eats the response, the server times out at the wrong moment, or a client library decides to be helpful and retry.
From the client's perspective, a timeout is ambiguous. Did the request fail before it arrived? Or did it succeed and only the response got lost? You cannot tell. So the safe move is to retry — and if the server treats every attempt as a fresh "create," one hiccup becomes nine posts.
The server wasn't broken, by the way. It did exactly what I told it to do, nine times in a row. The bug was in the contract: I never gave it a way to recognize "this is the same request you already handled."
The fix: idempotency keys, the Stripe pattern
The payments world solved this years ago, because "charge the customer nine times" is a slightly worse bug than "nine blog posts." Stripe's API is the canonical example: every mutating request can carry an Idempotency-Key header — a unique string the client generates per logical operation. The server stores the result of the first request against that key. If the same key shows up again, the server doesn't redo the work; it just replays the original response.
That's the whole trick, and it's beautiful because it moves the retry-safety problem from "hope the network behaves" to "the protocol itself is safe to retry."
So I rebuilt the blog's create path around that pattern, in three layers:
1. Server-side idempotency. The CMS now accepts an Idempotency-Key header on post creation. First request with a key: create the post, store the response against the key, return it. Any repeat with the same key: return the stored response with a flag saying "this is a replay." No new post, no matter how many times the request arrives.
2. A database seatbelt. Idempotency logic lives in application code, and application code has bad days. So there's also a uniqueness constraint at the database level on the collision surface. Even if every layer above it fails, the database itself refuses to mint a true duplicate. Defense in depth isn't paranoia when the failure mode is public.
3. A stable client-side key. This last part is subtle and easy to get wrong. An idempotency key only works if the same logical operation generates the same key across retries. If the client generates a fresh random key per attempt, you've built a very elaborate system that deduplicates nothing. So the client derives its key deterministically from the post's title and content. Same post, retried forever, always the same key. The whole stack collapses to exactly one create.
The underrated detail: collisions that aren't dupes
One more wrinkle worth mentioning, because it's the kind of thing you only think of after it bites you: slug conflicts aren't always duplicates.
Two genuinely different posts can want the same slug — I've written more than one post that could plausibly be called "fixing-the-thing," believe me. A naive system sees a slug collision and either errors or overwrites. So the collision handling has to distinguish "same content, same key — replay it" from "different content — suffix the slug and carry on." Same title does not mean same post. Only the idempotency key gets to make that call.
What I took from it
Honestly, the embarrassing part is that I knew all of this already. I've read the Stripe docs. I've nodded along to "design for failure" talks. And I still shipped a create endpoint with no idempotency and a client that could retry, because it worked fine in testing — and in testing, the network never hiccups at the wrong moment.
That's the real lesson. Retry storms and ambiguous timeouts are not edge cases; they're the normal behavior of every distributed system on a long enough timeline. If an operation matters — if it publishes, charges, deletes, sends — the question "what happens if this request arrives twice?" has to have a designed answer, not a discovered one.
Mine now does. And the next time I build anything that creates, charges, or sends, the idempotency key goes in on day one — not on day "nine identical posts."
Look at any mutating endpoint you've built recently. If the same request arrived five times, what would happen? If the honest answer is "I don't know," well — now you know what your weekend project is.
