Idempotency Keys: How to Make Retries Safe in Distributed Systems

Retries are how distributed systems survive flaky networks. They’re also how a customer gets charged twice.

Updated 13 October 2026 · Tested with Python 3.12, FastAPI 0.142, PostgreSQL 18.6, Redis 8.10.2

Part 1 of the Reliable Python services series

Your client sends a payment, the connection drops, and you can’t tell if the card was charged. Retry, and you might charge it twice. An idempotency key is a unique value the client sends with a state-changing request, usually in an Idempotency-Key header, so the server runs the operation once and replays the stored result to every retry.

The idea is simple. Where implementations break is the claim: the moment the server decides which request gets to run. Make that step atomic, keep it in the same database as the side effect, and prove it under concurrency. In this post I’ll take you through a Python build with FastAPI and Postgres (plus Redis as the alternative), a race test with real numbers, and when not to bother at all.

Jargon Alert: the claim is the atomic step where one request wins the right to run. A fingerprint is a hash of the request, so a reused key with a new body gets caught. A lease is how long a claim is honored before another request may take it over. The IETF (Internet Engineering Task Force) is the group that writes the internet’s technical standards, HTTP included. A finished standard is published as an RFC; a proposal still being worked on is a draft, numbered by version (-01, -02 and so on).

Why does a retry charge the card twice?

Because a timeout tells the client nothing about the charge. Your client sends POST /payments, waits, and gives up. Was the card charged? It can’t know, so it retries, and if the first attempt went through, that’s two charges. A timeout can hide three outcomes:

1. The request never reached the server.
2. The server did the work, and the response got lost.
3. The server is still working on it.

From the client’s side, all three look identical. And your code isn’t the only thing retrying:

• SDKs. Stripe’s official libraries retry network errors with an idempotency key attached. Node, Go and .NET do it by default.
• Users. Double-clicks, refreshes, and mobile apps resubmitting after a reconnect.
• Proxies and load balancers. RFC 9110 says a proxy must not automatically retry non-idempotent requests, and nginx has followed that since 1.9.13. Add non_idempotent to proxy_next_upstream and it will retry your POSTs too.
• Queues. Amazon SQS standard queues deliver at least once, and AWS tells you to design your applications to be idempotent.
• Your own backoff code, including the loop that waits out a 429 and Retry-After.

What one duplicated batch costs: Santander’s Christmas Day

On Christmas Day 2021, Santander UK processed payments from about 2,000 business accounts twice: 75,000 transactions worth £130 million. Wages and supplier bills landed a second time, many of them in accounts at other banks (CNBC). The bank blamed a “scheduling issue”, paid for the duplicates itself and had to claw the money back through the industry’s bank error recovery process, bank by bank.

Santander didn’t publish the technical details, so I won’t pretend to know the root cause. But look at the shape of the failure: the same payment instruction executed twice. That’s exactly what an idempotency key is for.

Give every instruction a key derived from the instruction itself (the batch ID plus the line number, say), claim it atomically before any money moves, and a job that runs twice turns into one run plus 75,000 replayed results.

And notice where the pain was: days of recovery after the money had moved. A duplicate check that runs after the side effect, like the unique key in the race test below, can only report the damage.

Is POST idempotent? What HTTP actually promises

No. RFC 9110 defines PUT, DELETE and the safe methods as idempotent, and leaves POST out. Safe means essentially read-only; idempotent means sending the same request many times has the same intended effect as sending it once. PATCH isn’t idempotent either (RFC 5789).

Idempotency is about the effect, not the response: deleting an order twice might return 204 and then 404, and the order is gone either way. It’s also a promise you have to keep; a PUT that increments a counter isn’t idempotent just because it’s called PUT. POST is the problem because it’s what we use for payments, orders and emails, and the spec says clients shouldn’t retry it automatically unless they know it’s idempotent. An idempotency key is how they know.

Before: POST /payments times out after the card was already charged, so the client retries. Two charges.
After: The retry carries the same Idempotency-Key, so the server finds the completed record and returns the original 201 with the original payment ID. One charge.

How do idempotency keys work, end to end?

The client makes one key per operation and sends it on every attempt. The server claims the key atomically before doing any work, saves the response next to the side effect, and answers every repeat from that record: the stored response if the body matches, a 409 while the first attempt is still running, a 422 if the body changed.

In a REST API, the key rides in the Idempotency-Key header of a POST, not in the body, where proxies and middleware can’t see it. The client has three jobs:

• Generate the key once per operation, not per attempt.
• Persist it before the first attempt, so a retry after a crash reuses it.
• Send the same key and body on every retry.

The server keeps one record per key: the key (scoped to the client), a fingerprint of the request, a status, the stored response, and when the current owner claimed it. Every request then takes one of five paths:

1. No key on an endpoint that needs one: reject with 400.
2. New key: claim it atomically as in progress, do the work, store the response, mark it completed.
3. Completed, same fingerprint: return the stored response. Execute nothing.
4. Completed, different fingerprint: reject with 422.
5. Still in progress: reject with 409; the client backs off and retries.

How do you generate an idempotency key?

Use a random UUID, made once when the operation starts, and store it with the operation before the first attempt. UUIDv4 is the usual choice; on a busy key table, a time-ordered UUIDv7 or ULID inserts faster (Shopify’s numbers are below). If the operation already has a stable business identity, like “charge order 42”, you can hash that instead, so even a client that lost its state produces the same key.

Either way, never make the key inside the retry loop, and never on the server when the client leaves it out: a key the server makes is new on every retry, so it protects nothing.

Here’s the client side with httpx (client.py):

"""The client's half: make the key once per operation, then reuse it on every retry."""
import hashlib
import random
import time
import uuid

import httpx

RETRY_STATUSES = {409, 429, 502, 503, 504}


def new_key() -> str:
    return str(uuid.uuid4())  # random: store it with the operation before the first attempt


def key_for(action: str, *ids: object) -> str:
    """Deterministic: the same business action always gets the same key."""
    return hashlib.sha256(":".join([action, *map(str, ids)]).encode()).hexdigest()


def post_with_retries(http: httpx.Client, url: str, body: dict, key: str,
                      attempts: int = 5, base_delay: float = 0.2) -> httpx.Response:
    # The key is a parameter, so this loop can't make a new one per attempt.
    headers = {"Idempotency-Key": key}
    attempt = 1
    while True:
        try:
            resp = http.post(url, json=body, headers=headers)
            if resp.status_code not in RETRY_STATUSES or attempt == attempts:
                return resp
        except httpx.TransportError:  # timed out or reset: it may or may not have run
            if attempt == attempts:
                raise
        time.sleep(base_delay * 2 ** (attempt - 1) * random.uniform(0.5, 1.0))
        attempt += 1

The retry function takes the key as a parameter, so it can’t mint a new one per attempt. The tests drive it with httpx’s MockTransport: a timeout, a 409, then a 201, all with the same key and body. Another drops the first response after the server charged the card; the retry gets that charge back with Idempotent-Replayed: true. One trap with hashed keys: two legitimate operations with the same inputs collapse into one, so hash something unique to each, like the order ID.

Two requests with the same key at the same time

This isn’t an edge case: a client times out while the server is still busy, and its retry lands mid-flight. A “SELECT, then INSERT if missing” check races, and both requests do the work (the race test below measures how badly).

The claim has to be atomic: a unique constraint on (client_id, idem_key) plus INSERT … ON CONFLICT, or SET … NX in Redis. The winner does the work. The loser gets a 409, backs off, and probably finds a completed result to replay next time.

Same key, different payload

Almost always a client bug: a key reused for the next operation. Without a fingerprint, the server replays the first response, and the client believes its second payment went through. Compare fingerprints on every repeat and fail loudly with 422.

What if the first attempt failed?

Then you pick a rule, and the two Stripe APIs picked opposite ones. API v1 saves the first result, success or failure, 500 errors included, and replays it, because a 500 may already have had side effects; the IETF draft says the same. API v2 re-executes a request whose first attempt failed.

The code below departs from the draft in one case. If all the work ran in one transaction that rolled back, nothing happened, so it deletes the key and lets the retry run. A business error you return on purpose, like a declined card, is stored and replayed. Only release a key when you can prove there were no side effects.

Idempotency key vs request ID vs correlation ID: aren’t they the same thing?

No, and mixing them up breaks deduplication. An idempotency key names one operation, so it stays the same on every retry. A request ID names one attempt and changes each time. A correlation ID ties together every message in a flow, so many different operations share it. Only the first tells the server “you’ve seen this before”.

Two of those trip people up. MCP’s 2026-07-28 spec says a request ID must not match any request the sender has issued and not yet got a response for, so retrying an unanswered tool call needs a new ID, and the JSON-RPC ID can’t be your idempotency key. Microsoft’s Idempotent Consumer pattern says not to deduplicate on a correlation ID, because it refers to groups of related messages. And a nonce is the mirror image of a key: a repeat is rejected as a replay attack, never answered with the stored result.

A minimal idempotency key implementation in Python (FastAPI + Postgres)

You need two tables, one function and a thin endpoint. The function, run_once, claims the key with INSERT … ON CONFLICT, runs your business logic and writes the response in one transaction, and replays the stored result to any repeat. The endpoint only parses the header and turns the result into an HTTP response.

The schema (schema.sql); charges stands in for whatever your endpoint really does:

-- One row per (client, key). The primary key is the claim: only one INSERT can win.
CREATE TABLE IF NOT EXISTS idempotency_keys (
    client_id     text        NOT NULL,
    idem_key      text        NOT NULL,
    fingerprint   text        NOT NULL,
    status        text        NOT NULL DEFAULT 'in_progress'
                              CHECK (status IN ('in_progress', 'completed')),
    response_code int,
    response_body jsonb,
    locked_at     timestamptz NOT NULL DEFAULT now(),  -- lease: when the current owner claimed it
    created_at    timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (client_id, idem_key)
);
CREATE INDEX IF NOT EXISTS idempotency_keys_created_at ON idempotency_keys (created_at);

-- The side effect we protect: one row per card charge.
CREATE TABLE IF NOT EXISTS charges (
    id         bigint      GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    client_id  text        NOT NULL,
    amount     int         NOT NULL,
    currency   text        NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now()
);

Now the core, in psycopg 3. This is idempotency.py minus the cleanup job, which comes later:

"""Idempotency keys on PostgreSQL with psycopg 3: claim the key atomically, run the
work once, store the response beside the side effect, and replay it to every retry."""
import hashlib
import json
from collections.abc import Callable
from datetime import timedelta
from typing import Any, NamedTuple

import psycopg
from psycopg.types.json import Jsonb

LEASE = timedelta(seconds=30)  # longer than your slowest do_work


class Result(NamedTuple):
    code: int
    body: dict[str, Any]
    replayed: bool = False


class LeaseLost(Exception):
    """Our claim went stale and another request took it over, so we rolled back."""


def fingerprint(method: str, path: str, body: dict) -> str:
    canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(f"{method} {path} {canonical}".encode()).hexdigest()


def replay_or_reject(fp: str, stored_fp: str, status: str,
                     code: int | None, body: dict | None) -> Result:
    """The answer for a key we have seen before."""
    if stored_fp != fp:
        return Result(422, {"error": "Idempotency-Key was used with a different request"})
    if status == "in_progress":
        return Result(409, {"error": "A request with this Idempotency-Key is in progress"})
    return Result(code, body, replayed=True)


# Insert a new claim, or take over a stale one. Anything else returns no row.
CLAIM = """
INSERT INTO idempotency_keys (client_id, idem_key, fingerprint)
VALUES (%(client)s, %(key)s, %(fp)s)
ON CONFLICT (client_id, idem_key) DO UPDATE SET locked_at = now()
    WHERE idempotency_keys.status = 'in_progress'
      AND idempotency_keys.fingerprint = excluded.fingerprint
      AND idempotency_keys.locked_at < now() - %(lease)s
RETURNING locked_at
"""


def run_once(conn: psycopg.Connection, client_id: str, key: str, fp: str,
             do_work: Callable[[psycopg.Connection], tuple[int, dict]],
             lease: timedelta = LEASE) -> Result:
    # The claim must commit on its own, so a duplicate gets its 409 straight away
    # instead of waiting on the unique index until our transaction ends.
    if not conn.autocommit:
        raise ValueError("run_once needs a connection opened with autocommit=True")

    args = {"client": client_id, "key": key, "fp": fp, "lease": lease}
    claimed = conn.execute(CLAIM, args).fetchone()

    if claimed is None:  # someone else holds or finished this key
        row = conn.execute(
            "SELECT fingerprint, status, response_code, response_body "
            "FROM idempotency_keys WHERE client_id = %(client)s AND idem_key = %(key)s",
            args,
        ).fetchone()
        if row is None:  # released a moment ago
            return Result(409, {"error": "Retry this request"})
        return replay_or_reject(fp, *row)

    args["token"] = claimed[0]  # our lease; completion checks that we still hold it
    try:
        with conn.transaction():  # business writes and stored response commit together
            code, body = do_work(conn)
            done = conn.execute(
                "UPDATE idempotency_keys SET status = 'completed', "
                "response_code = %(code)s, response_body = %(body)s "
                "WHERE client_id = %(client)s AND idem_key = %(key)s "
                "AND locked_at = %(token)s",
                {**args, "code": code, "body": Jsonb(body)},
            )
            if done.rowcount != 1:
                raise LeaseLost(key)  # rolls back do_work's writes as well
        return Result(code, body)
    except Exception:
        # Rolled back, so nothing happened: free the key and let a retry run.
        conn.execute(
            "DELETE FROM idempotency_keys WHERE client_id = %(client)s "
            "AND idem_key = %(key)s AND locked_at = %(token)s",
            args,
        )
        raise

The claim is one statement. A new key inserts a row and gets locked_at back; a key someone else holds returns no row, and replay_or_reject answers. The DO UPDATE … WHERE branch only fires for a same-body claim that’s still in progress and older than the lease, which is how a retry takes over from a crashed worker.

Then the endpoint (app.py):

"""POST /charges, protected by an Idempotency-Key (FastAPI + psycopg 3 + PostgreSQL).

Run: uvicorn app:app --port 8000
"""
import os
import time
from collections.abc import Iterator

import psycopg
from fastapi import Depends, FastAPI, Header, HTTPException
from fastapi.responses import JSONResponse
from pydantic import BaseModel

from idempotency import Result, fingerprint, run_once

DSN = os.environ.get("IDEM_DSN", "postgresql://patterns:patterns@localhost:55432/idem")
CHARGE_DELAY = float(os.environ.get("CHARGE_DELAY", "0.05"))  # seconds

app = FastAPI()


class ChargeIn(BaseModel):
    amount: int
    currency: str = "usd"


def get_conn() -> Iterator[psycopg.Connection]:
    # One connection per request keeps the example short; use psycopg_pool in production.
    with psycopg.connect(DSN, autocommit=True) as conn:
        yield conn


def parse_key(raw: str | None) -> str:
    if raw is None:
        raise HTTPException(400, "Idempotency-Key header is required")
    key = raw.strip()
    if len(key) >= 2 and key[0] == key[-1] == '"':  # the IETF draft sends a quoted string
        key = key[1:-1]
    if not 1 <= len(key) <= 255:
        raise HTTPException(400, "Idempotency-Key must be 1-255 characters")
    return key


def charge(conn: psycopg.Connection, client_id: str, amount: int,
           currency: str) -> tuple[int, dict]:
    time.sleep(CHARGE_DELAY)  # stand-in for the call to the card network
    (charge_id,) = conn.execute(
        "INSERT INTO charges (client_id, amount, currency) VALUES (%s, %s, %s) RETURNING id",
        (client_id, amount, currency),
    ).fetchone()
    return 201, {"id": charge_id, "amount": amount, "currency": currency}


def to_response(result: Result) -> JSONResponse:
    headers = {"Idempotent-Replayed": "true"} if result.replayed else None
    return JSONResponse(result.body, status_code=result.code, headers=headers)


@app.post("/charges")
def create_charge(
    body: ChargeIn,
    client_id: str = Header(alias="X-Client-Id"),  # stand-in for your auth layer
    # Optional on purpose: a missing required header would be FastAPI's own 422.
    idempotency_key: str | None = Header(default=None),
    conn: psycopg.Connection = Depends(get_conn),
) -> JSONResponse:
    key = parse_key(idempotency_key)
    fp = fingerprint("POST", "/charges", body.model_dump())
    result = run_once(conn, client_id, key, fp,
                      lambda c: charge(c, client_id, body.amount, body.currency))
    return to_response(result)

Three details are easy to get wrong. The header is optional in the signature, because a required Header() makes FastAPI answer a missing key with its own 422, which reads like “reused with a different payload”. parse_key strips the quotes the IETF draft puts around the value. And replays carry Stripe’s Idempotent-Replayed: true header. Four design choices are worth calling out too:

• The claim commits on its own. Inside a longer transaction, a duplicate would wait on the unique index until that transaction ended; a repo test shows it hanging until lock_timeout. With autocommit, the duplicate gets its 409 at once.
• The charge and the stored response commit together, so “charged, response not recorded” can’t happen.
• Returned business errors are stored and replayed. A 402 for a declined card replays like a 201; only an exception frees the key.
• The claim is a 30-second lease, so a crashed worker’s key gets taken over. If the old owner was only slow, its final UPDATE finds locked_at changed and raises LeaseLost, rolling its charge back.

That rollback only works because the charge is a row in the same database. If your side effect is a call to a payment provider, pass your idempotency key on to it as well, and set the lease well above your slowest call.

The code for this post has 40 pytest tests against Postgres and Redis in Docker, one per path above, including the lost lease, the lost response and the races. With the containers from the series folder’s docker-compose.yml running, this reruns the tests and the measurements:

cd reliable-python-services/01-idempotency-keys
pytest -q
python measure_race.py
python measure_claim_cost.py
python measure_claim_pileup.py

Postgres or Redis? Where to store idempotency keys, and for how long

In Postgres, if that’s where the side effect lives: it’s the only place where the key and the side effect commit together. Redis is faster and expires keys for free, but it’s a second system, so a key lost there means a duplicate charge. Keep each key for longer than anyone might retry, and publish that number. The table below has both options, with numbers from the test.

My recommendation: start with a table in the same database as the data it protects. Reach for Redis when volume makes that painful or the stakes are low, knowing its replication docs say even WAIT can’t stop acknowledged writes from being lost in a failover.

If you do use Redis, treat SET NX as a claim, not a lock (redis_claim.py):

"""Idempotency keys in Redis: SET NX claims the key, a Lua script stores the result
only if the claim is still ours. Faster than Postgres, but not atomic with the work."""
import json
import logging
import uuid
from collections.abc import Callable

import redis

from idempotency import Result, replay_or_reject

LEASE_SECONDS = 30        # how long an in-progress claim lives if we crash
KEEP_SECONDS = 24 * 3600  # how long a finished result is replayed

log = logging.getLogger(__name__)

# Replace our claim with the final record ('' = delete it), unless someone else
# has claimed the key since. Plain Lua, so it runs on Redis 6.2+, 7.x and Valkey.
SWAP_IF_OURS = """
local current = redis.call('GET', KEYS[1])
if current and current ~= ARGV[1] then return 0 end
if ARGV[2] == '' then
  redis.call('DEL', KEYS[1])
else
  redis.call('SET', KEYS[1], ARGV[2], 'EX', ARGV[3])
end
return 1
"""


def run_once_redis(r: redis.Redis, client_id: str, key: str, fp: str,
                   do_work: Callable[[], tuple[int, dict]]) -> Result:
    rkey = f"idem:{client_id}:{key}"
    claim = json.dumps({"status": "in_progress", "fp": fp, "owner": uuid.uuid4().hex})

    # SET NX is the claim; GET hands back the existing record in the same round trip.
    existing = r.set(rkey, claim, nx=True, ex=LEASE_SECONDS, get=True)
    if existing is not None:
        rec = json.loads(existing)
        return replay_or_reject(fp, rec["fp"], rec["status"], rec.get("code"), rec.get("body"))

    swap = r.register_script(SWAP_IF_OURS)
    try:
        code, body = do_work()
    except Exception:
        swap(keys=[rkey], args=[claim, "", 0])  # the work failed cleanly: release the key
        raise

    record = json.dumps({"status": "completed", "fp": fp, "code": code, "body": body})
    if not swap(keys=[rkey], args=[claim, record, KEEP_SECONDS]):
        # The work outlived the lease and a retry claimed the key: it may run twice.
        log.warning("claim on %s expired mid-work; a retry may have repeated it", rkey)
    return Result(code, body)

SET … NX GET (allowed together since Redis 7.0) claims the key or hands back the stored record in one round trip, and the Lua script stores the result only if the claim is still ours. On Redis 8.4 and later, SET … IFEQ does that compare-and-set without Lua; the repo tests both. What Redis can’t do is roll the charge back. If the work outlives the lease, another request can claim the key and charge again, and that gap is why a Redis lock isn’t enough when correctness matters.

How Airbnb got a double payment from a correct retry

Airbnb’s payments team hit a double payment with the keys stored in the right place, and wrote it up in 2019. A guest’s payment went through, but the response got lost on the network. The client did everything right and retried with the same idempotency key. The service looked the key up on a read replica, the replica was a few seconds behind, and the stored response wasn’t there yet. So it charged again.

Their fix was an idempotency library called Orpheus, and it lands on the same design as this post:

They also sharded the idempotency tables by key, since random keys spread evenly, and reported five nines of consistency for payments while annual payment volume doubled. The lesson I take from the replica story: storing keys isn’t enough. You have to read them from somewhere that already has the last write. A Redis replica, or a read replica you point at by accident, quietly rebuilds Airbnb’s bug.

Testing it: what happens when 100 retries land at once?

With check-then-insert, every one of them charges the card. In 20 runs of 100 identical requests, the naive version made 100 charges out of 100 every single time, and 50 out of 50 at the smaller size. INSERT … ON CONFLICT and Redis SET NX made exactly one charge in all 80 of their runs.

The test app (race_app.py) serves the post’s own POST /charges plus the same endpoint on the other two claims. All three do the same work in charge(): a 50 ms pause for the card network, then one row in charges. Each run sends 50 or 100 identical POSTs with one fresh key at once (asyncio and httpx) and counts the rows, 20 runs per strategy and size.

Here’s the version that races (naive.py). It passes a test that sends one request and then one retry:

"""The version that races: look for the key, do the work, then record the key.
Kept here so the race test can show what it costs. Don't ship it."""
from collections.abc import Callable

import psycopg
from psycopg.types.json import Jsonb

from idempotency import Result, replay_or_reject


def check_then_insert(conn: psycopg.Connection, client_id: str, key: str, fp: str,
                      do_work: Callable[[psycopg.Connection], tuple[int, dict]]) -> Result:
    row = conn.execute(
        "SELECT fingerprint, status, response_code, response_body "
        "FROM idempotency_keys WHERE client_id = %s AND idem_key = %s",
        (client_id, key),
    ).fetchone()
    if row is not None:
        return replay_or_reject(fp, *row)

    code, body = do_work(conn)  # every request that got past the SELECT charges the card
    conn.execute(  # the primary key rejects the late ones here, after the charge
        "INSERT INTO idempotency_keys "
        "(client_id, idem_key, fingerprint, status, response_code, response_body) "
        "VALUES (%s, %s, %s, 'completed', %s, %s)",
        (client_id, key, fp, code, Jsonb(body)),
    )
    return Result(code, body)

Guess the naive number first. Out of 100 identical requests, how many charges?

The primary key was there all along, and it fired: 99 of the 100 naive requests hit UniqueViolation on the final INSERT and returned a 500, after the card was charged. A constraint that runs after the side effect can only report the duplicate.

The atomic claims answered the other 99 differently. Redis gave all of them a 409. Postgres gave mostly 409s in 12 of its 20 runs at 100, but in the other 8, most got the replayed 201.

The INSERT docs explain it: DO UPDATE, which the lease takeover needs, locks the conflicting row even when its WHERE is false, so duplicates queue. In a separate test (measure_claim_pileup.py), 99 duplicate claims took a median 137.5 ms to all come back with this claim, against 16.5 ms with plain DO NOTHING, so the back of the queue arrives after the first request commits and gets the replay. Both answers are correct.

One more thing: your server’s concurrency sets the size of the damage. The test app raises AnyIO’s worker-thread limit to 200 so all 100 sync requests really run at once. At FastAPI’s default of 40 threads, check-then-insert charged exactly 40 of 100 in every run; more threads or workers would let more through.

Note: AMD Ryzen 9 7950X3D (16 cores/32 threads), 64 GB RAM, Windows 11, Docker Desktop 28.5 (WSL2); Python 3.12, PostgreSQL 18.6 and Redis 8.10.2 in Docker; one uvicorn process. Each request gets its own httpx.AsyncClient, because one shared client queued them. Your counts will differ; the atomic claims’ single charge shouldn’t.

How long should you keep idempotency keys?

Longer than anyone might retry. Stripe’s API v1 removes keys once they’re at least 24 hours old; API v2 replays retries within 30 days. If a queue can redeliver for three days, 24 hours is too short. Whatever you pick, publish it, as the IETF draft asks. In Postgres, expiry is a scheduled job (the rest of idempotency.py):

def purge_expired(conn: psycopg.Connection, keep: timedelta = timedelta(hours=24)) -> int:
    """Delete keys older than your published retention. Run it from a scheduled job."""
    cur = conn.execute("DELETE FROM idempotency_keys WHERE created_at < now() - %s", (keep,))
    return cur.rowcount

Storage is cheap: 100,000 completed keys took 351.3 bytes each in Postgres, both indexes included, and a median 275 bytes each in Redis (MEMORY USAGE). A million keys a day kept for 30 days is about 10.54 GB on disk in Postgres, or 8.25 GB of RAM in Redis.

Redis is faster, though both are small next to a 50 ms card call. One call at a time over localhost, a new-key claim took 1.19 / 2.01 ms (p50 / p99) in Postgres and 0.33 / 0.47 ms in Redis. A repeat, which is what a retry costs, took 1.53 / 2.28 ms in Postgres (the claim plus the SELECT) and 0.33 / 0.47 ms in Redis, where SET … NX GET answers in one round trip.

What about message queues and webhooks?

Same idea, different key. “Exactly-once delivery” is really at-least-once delivery plus idempotent processing: the consumer deduplicates on the message ID (never on a correlation ID) in the same transaction as its side effect. That’s the idempotent consumer pattern, and it’s also why a transactional outbox in Python is safe to run: its relay sometimes publishes twice, and the consumer shrugs it off.

Webhooks are the same problem with the roles swapped: the provider is now the client that retries, and you’re the server. Stripe’s webhook docs say an endpoint might occasionally receive the same event more than once, and in live mode Stripe keeps retrying a failed delivery for up to three days. So treat the event ID as the idempotency key: claim it with the same INSERT … ON CONFLICT before you fulfill the order, keep it at least as long as the provider retries, and answer a repeat with a 2xx so the provider stops resending.

The Idempotency-Key header: is it a standard yet?

Not yet. As of October 2026, the IETF HTTPAPI working group’s Idempotency-Key HTTP Header Field draft is at revision -07, its seventh version (15 October 2025). It expired on 18 April 2026 without becoming an RFC, and no -08 has appeared. So it’s a well-documented convention, and still the best reference for the status codes.

The draft suggests 400 for a missing key, 409 while the original is still being processed, and 422 for a key reused with a different payload. It asks servers to publish their idempotency rules, expiry included, and to combine the client’s key with client-specific attributes, since low-entropy keys let attackers guess other clients’ keys.

One gotcha: the draft makes the value a Structured Fields string, so its examples quote the key (Idempotency-Key: "8e03978e-…"), while Stripe’s send it bare. Normalize before storing, or a client that switches styles between attempts gets two charges. parse_key above does it, and a test sends both forms of one key.

Stripe’s implementation is the one most people copy (the draft lists it under Implementation Status):

• Keys go in the Idempotency-Key header, up to 255 characters. V4 UUIDs are suggested.
• Nothing is saved if validation fails or the request conflicts with one that’s still running, so those can be retried.
• Replays carry an Idempotent-Replayed: true header, according to Stripe’s low-level error handling guide. Worth copying.

API v2 changed three rules: keys also work on DELETE, replays last 30 days, and a failed first attempt is re-executed. Side by side:

Idempotency key examples: how Amazon, Shopify, PayPal and Stripe do it

They all hit the same problem, a request that timed out after the work was done, and they all landed on a key the client sends and reuses. Where they differ is what a retry gets back and how long the key lives:

Amazon: a timed-out launch that could start a second server

Amazon’s Builders’ Library tells the problem through EC2. A customer runs a workload that must have at most one instance, calls RunInstances, and the call times out. Retry blindly and there may be two instances. The fix is the ClientToken: the AWS SDK and CLI generate one when you don’t, and reuse it on every retry, so the retry gets the same instance back.

Two details line up with this post. Amazon insists that recording the token and creating the resource happen as one ACID operation, which is why run_once claims the key in the same database as the charge. And it stores the original parameters with the token, so the same token with different parameters gets a validation error, the same job the fingerprint and the 422 do here. Amazon also explains why it doesn’t just hash the parameters: a customer might really want two identical instances. That’s the trap with hashed keys mentioned above.

Shopify: once in a million is many times a day

Shopify’s payments engineering team puts the scale problem plainly: a once-in-a-million network glitch happens many times a day at their volume, and a double charge doesn’t just annoy the buyer, it can land the merchant a chargeback. Their payment service tracks every attempt by its idempotency key, records which steps finished, and makes sure only one request reaches the financial partner. If a retry arrives after a partial failure, recovery steps rebuild the state before carrying on.

One number from them changes the key-generation advice above. They use ULIDs, which start with a timestamp, instead of random UUIDv4 keys, and in one high-throughput system INSERT time dropped by 50 percent after the switch. Random keys scatter across the B-tree index; time-ordered ones land at the end of it. If your key table is busy, generate time-ordered keys: a ULID library, uuid.uuid7() on Python 3.14, or uuidv7() in PostgreSQL 18.

PayPal: the latest status instead of the old response

PayPal’s idempotency docs use the exact scenario from the top of this post: a request to capture a payment times out, but PayPal already captured it. Retry with the same PayPal-Request-Id and you get 201 Created with the capture details, and no second capture. Leave the header out, and PayPal says it duplicates the request.

The interesting difference is what comes back. Stripe replays the stored response; PayPal returns the request’s latest status. Both are valid, as long as you document which one you picked. PayPal also says that of two simultaneous requests with the same ID, it processes the first and may fail the second, which is the in-progress 409 in another form.

AI agents retry tool calls too

Agents make this worse, quietly. In a ReAct-style loop, the model calls a tool and reads the result. If the result is a timeout, the natural next move is to call it again. Add framework retries, or several agents sharing a task in a multi-agent setup, and send_invoice can easily run twice.

Same fix, one twist: derive the key in the tool runtime. Not in the model, since nothing guarantees it reuses one, and not from the MCP request ID, since a retry must send a new one:

• For harness retries of the same call, use the run ID plus the tool call ID.
• For the model trying again, hash the task ID, tool name and canonical arguments, so the second call replays the first. Decide per tool whether two identical calls could ever be legitimate.
• Pass it through to downstream APIs, so protection holds end to end. Gateways should forward it too, because LLM gateway retries are one more layer that resends.

MCP tools can declare an idempotentHint (default false), but the schema says annotations are hints, and clients should never make tool-use decisions based on annotations from untrusted servers. The hint says retrying is harmless. It doesn’t make it so.

Mistakes I’d look for in code review (and when not to bother)

These are the six problems I’d look for in a pull request, and the worst one passes every happy-path test. After them come the questions that tell you whether you need keys at all.

• Generating the key inside the retry loop. Every attempt gets a new key, and you’ve built an elaborate no-op.
• Check-then-insert. It races. Let a unique constraint do the claiming, before the work.
• Storing the key but not the response. The retry needs the original result, payment ID included, not a vague “already exists”.
• No fingerprint. A reused key silently returns the wrong operation’s response.
• Global keys. Scope them per client; the draft warns that low-entropy keys can expose other clients’ stored responses.
• Testing only the happy path. One request plus one retry passes with check-then-insert. Only a concurrent test shows it charging the card 100 times out of 100.

And before any of that, ask whether the endpoint needs keys at all:

1. Does it change state? If not, stop. A GET doesn’t need a key.
2. Is it naturally idempotent already? A full-replacement PUT, a client-supplied ID, a unique business constraint or an If-Match condition may be enough.
3. What does a duplicate cost? Money, stock, emails, third-party calls: use keys. A duplicated “last seen” timestamp: don’t bother.
4. Can the work and the key commit in one transaction? Then a table in the same database is simple. If not, pass keys downstream.
5. How long can retries keep arriving? Set the expiry well beyond that, and publish it.
6. What do you return for replays, conflicts, mismatches and missing keys? Decide now, and document it.

The rest of this series, and more system design patterns like this one, live in the System Design hub.

If you do one thing this week, put a unique constraint on your key table and write the race test. It’s a few dozen lines of pytest, and it tells you in seconds whether your claim is atomic.

Over to you: where have retries bitten you? A double charge, a duplicate email, an agent booking the same meeting twice? Tell me in the comments. For the next part of this series in your inbox, join the newsletter.