Backend

The Idempotency Bug That Only Showed Up Under Retries

Our payments API had an idempotency key on every write. It still double-charged a customer, because the bug wasn't in the idea — it was in the four-line race condition between checking the key and writing the result.

June 29, 20268 min read

idempotencyrace conditionsdistributed systemspayments

We had done everything the blog posts tell you to do: every write endpoint required an Idempotency-Key header, we stored keys in Redis, and we checked for an existing key before processing a request. And a customer still got double-charged, from a single client-side retry, because the naive "check then write" pattern has a race condition sitting right in the middle of it that only shows up under real concurrent retries — which is, ironically, the exact scenario idempotency keys exist to handle.

The naive version (this is what most idempotency guides show you)

Gocheck-then-write-—-looks-correct,-isn't.go

14 linesLn 1, Col 1Go

This reads correctly top to bottom. The bug is invisible unless you think specifically about what happens when two requests with the same key arrive close together.

The race: two requests, one key, one gap

A mobile client on a flaky connection sent a charge request, didn't receive the response in time (the response was actually fine — it was just slow to arrive over a bad connection), and retried with the same idempotency key, as it's supposed to. Both requests hit our service roughly 40ms apart — well within the same request-processing window.

Request A checks the cache for key-123. Not found.
Request B (the retry) checks the cache for key-123, 40ms later. Also not found — request A hasn't written its result yet, because it's still waiting on the payment provider's API call.
Request A calls the payment provider, gets a successful charge, writes the result to cache under key-123.
Request B, having already decided "not found" 40ms earlier, proceeds to also call the payment provider — with the same amount, same card — and gets a second successful charge.
Request B writes its own result to cache under key-123, overwriting request A's (harmless at this point — the damage is already done).

The race, as a sequence diagram

Rendering diagram…

The gap that matters

Between "check the cache" and "write the cache," there's a window — however small — where a second request with the same key sees no record and assumes it's safe to proceed. The check and the write are two separate operations with a real (if often brief) gap between them. Under load or with retries close together, that gap gets hit.

The fix: make the check-and-claim atomic

The fix isn't a smarter cache — it's collapsing "check" and "claim" into a single atomic operation, so there's no gap for a second request to slip through. Redis's SET key value NX (set-if-not-exists) does exactly this in one round trip: it either claims the key and proceeds, or discovers someone already claimed it and backs off, atomically, with no window in between.

Gofixed:-atomic-claim,-then-process.go

32 linesLn 1, Col 1Go

SETNX is atomic at the Redis level — two concurrent requests can never both succeed at claiming the same key. The second request immediately knows it lost the race and waits for the real result instead of racing ahead to also call the payment provider.

The second gotcha: what if the process crashes mid-charge?

Claiming the key with a TTL (30*time.Second above) matters for a reason beyond the obvious: if the process crashes *after* claiming the key but *before* calling the payment provider, the key would otherwise stay claimed forever, permanently blocking any real retry. The TTL ensures a crashed, stuck claim eventually expires and a genuine retry can proceed — trading a small window of possible duplicate processing (only if a crash happens at exactly the wrong moment and a retry arrives after the TTL expires) for guaranteed forward progress. We chose 30 seconds because it's comfortably longer than our payment provider's worst-case response time, but short enough that a real crash doesn't block a legitimate retry for long.

The general pattern

Any "check if X exists, then do Y" logic that's meant to prevent duplicate work has this exact race unless the check-and-claim step is a single atomic operation. This applies far beyond idempotency keys: rate limiters, distributed locks, "only one worker processes this job" logic — all the same shape, all the same fix (atomic claim with a TTL as a safety valve, not a plain read-then-write).

Read-then-write is never safe for anything meant to prevent duplicates under concurrency, no matter how small the code looks.
Use an atomic primitive for the claim step: SETNX in Redis, a unique constraint + INSERT ... ON CONFLICT in Postgres, or a compare-and-swap in whatever store you're using.
Always attach a TTL or expiry to the claim, so a crash mid-processing doesn't permanently block legitimate retries.
Test this specifically with real concurrent requests in your test suite — not sequential ones. A sequential test will never catch this bug, because the race only exists when the gap between check and write is actually contended.