Idempotency Is a Design Constraint, Not a Retry Handler

Every distributed system eventually produces the same incident report: a request timed out, a client retried, and the work happened twice. The usual fix is to wrap the call in a retry helper with jitter and call it a day. That helps with availability. It does nothing for correctness. Retrying is only safe when the operation on the other end was designed to be repeated, and that design happens long before you write the retry loop.

I think of idempotency the same way I think of a database constraint: something the system enforces structurally, not something each caller promises to be careful about.

Where "just retry" quietly breaks

The failure is almost never the happy path. It is the ambiguous middle: you sent the request, the work committed, and the response never came back. From the client's point of view a timeout and a hard failure look identical. It has no way to tell whether retrying is free or catastrophic.

A few shapes I have actually hit:

• Chunked fan-out. FCM multicast caps at 500 tokens per call, so a push to a large audience is a loop over chunks. If chunk 3 of 9 times out after the send succeeded, a naive retry of the whole batch re-notifies everyone in chunks 1 and 2. Users see the same notification twice and trust the product a little less.
• OAuth authorization codes. An authorization code is single-use by spec. A mobile client that retries the token exchange — or a deep link that fires twice — races itself. One exchange wins, one gets an opaque invalid_grant, and the user sees a login failure on a login that actually worked.
• Background workers. A worker picks up a job, starts processing, then its process pauses long enough for the scheduler to decide it is dead and hand the job to someone else. Now two workers believe they own the same row.

The thing that makes these hard is that none of them are bugs in the retry logic. The retry logic is correct. The operation underneath it was never repeatable.

Make the database the arbiter

The only durable place to decide "has this already happened?" is the same place the effect lands. For me that is almost always Postgres, because the check and the write can share a transaction.

Claiming work exactly once

Lease-based ownership solves the double-worker problem without a separate coordination service. Workers do not get assigned jobs; they claim them, and the claim expires.

UPDATE jobs
SET    status      = 'running',
       locked_by   = $1,
       lease_until = now() + interval '60 seconds',
       attempts    = attempts + 1
WHERE  id = (
  SELECT id
  FROM   jobs
  WHERE  status = 'queued'
     OR (status = 'running' AND lease_until < now())
  ORDER  BY created_at
  FOR    UPDATE SKIP LOCKED
  LIMIT  1
)
RETURNING id, payload;

FOR UPDATE SKIP LOCKED is the important part: concurrent workers skip rows another transaction already holds instead of queueing behind them. The lease does the rest — a worker that dies simply stops renewing, and the row becomes claimable again. The worker's job is now to heartbeat lease_until while it works, and to treat a lost lease as a signal to stop, not to push through.

Recording the request, not just the result

For client-facing writes, the durable record is the idempotency key. The client generates it, sends it on every attempt including retries, and the server inserts it before doing anything expensive.

INSERT INTO idempotency_keys (key, endpoint, request_hash, status)
VALUES ($1, $2, $3, 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING id;

If that returns a row, you won the race and you own the work. If it returns nothing, someone else already has it — you either wait for their stored response or return 409 Conflict and let the client retry the read. The request_hash matters more than it looks: it catches the client that reuses a key with a different body, which is a bug you want to surface loudly rather than silently serve a stale response.

The handler ends up boring, which is the goal:

async def create_payout(req: Request, key: str = Header(alias="Idempotency-Key")):
    body = await req.json()
    digest = sha256(canonical_json(body)).hexdigest()
 
    async with db.transaction():
        claimed = await db.fetchrow(CLAIM_KEY_SQL, key, "create_payout", digest)
        if claimed is None:
            existing = await db.fetchrow(LOAD_KEY_SQL, key)
            if existing["request_hash"] != digest:
                raise HTTPException(422, "Idempotency-Key reused with a different body")
            if existing["status"] == "in_progress":
                raise HTTPException(409, "Request already in flight")
            return JSONResponse(existing["response_body"], status_code=existing["status_code"])
 
        result = await do_the_actual_work(body)
        await db.execute(FINISH_KEY_SQL, key, 201, result)
 
    return JSONResponse(result, status_code=201)

Note what is inside the transaction: the claim, the work, and the stored response. If any of it fails, the key rolls back and the next retry gets a clean shot. Splitting those across transactions gives you a key that is claimed forever with no work behind it — a poison pill that looks like a duplicate to every future attempt.

The auth-code race, specifically

The OAuth case needed two changes. PKCE was the first: binding the authorization code to a code_verifier the client holds means an intercepted or replayed code is useless on its own. The second was a short-lived server-side cache of exchanged codes, so a second exchange of the same code within its lifetime returns the same token response instead of invalid_grant. That is a deliberate narrowing of the spec's single-use rule, scoped to a few seconds and a matching verifier, and it turned a confusing login failure into a no-op.

What actually saves you

The pattern underneath all of these is the same: move the "have I done this?" decision next to the effect, and make it a write rather than a read. A read-then-write check across two round trips is just a race with extra steps. Postgres's documentation on explicit locking is worth reading end to end if you are building this — the semantics are subtler than the syntax suggests.

None of this is exotic. It is a unique index, a lease column, and the discipline to decide up front which operations are allowed to be repeated. Do that work early and retries become the boring safety net they are supposed to be. Skip it, and every retry you add is a slightly faster way to corrupt your data.