Idempotency keys, and the quiet cost of getting them wrong

Every system that takes money or sends a message eventually meets the same problem: the client sent a request, the network swallowed the response, and now nobody knows whether it happened. The client retries. If the server is naive, the customer gets charged twice.

The usual answer is an idempotency key: the client picks a unique token, sends it with the request, and the server promises that two requests with the same token produce one effect. Simple to say. The interesting part is where the promise actually lives.

The naive version, and why it bites

The first version everyone writes looks like this:

def charge(idempotency_key, amount):
    if store.exists(idempotency_key):
        return store.get(idempotency_key)
    result = payment_gateway.charge(amount)  # side effect
    store.put(idempotency_key, result)
    return result

Read, then act, then write. The window between the exists check and the put is wide open. Two concurrent retries both see "no key yet," both call the gateway, and you have charged twice anyway. The idempotency key bought you nothing because the check and the effect are not atomic.

Put the uniqueness where the durability is

The fix is to make the key claim and the effect share a single point of serialization. A unique constraint in the same transaction that records the work is the cheapest version:

-- The unique index does the coordination for you.
INSERT INTO idempotency (key, status, created_at)
VALUES ($1, 'in_progress', now())
ON CONFLICT (key) DO NOTHING
RETURNING id;

If the insert returns a row, this request owns the key and may proceed. If it returns nothing, someone else got there first, and you wait on or return their result. The database, not your application code, is now the thing deciding who wins.

Approach	Atomic?	Cost
Read-check-write	No	A race per retry
Unique constraint	Yes	One index write
Distributed lock	Yes	A second system to operate

The state that nobody talks about

The hard case is the request that claims the key, calls the gateway, and then crashes before recording the result. Now the key says in_progress forever, and every retry blocks on a ghost.

You need a third state and a reconciliation path:

• in_progress with a lease that expires
• succeeded with the cached response
• failed that is safe to retry under a new attempt

The lease is the part people skip, and it is the part that pages you at 3am.

An idempotency key is not a deduplication trick. It is a tiny state machine you are agreeing to operate, forever, for every write that matters.

What I tell people to check

1. The key claim and the effect serialize on the same thing.
2. There is an expiry on in_progress, not just succeeded.
3. The cached response is the whole response, including the status code.
4. You have decided, on purpose, what a key collision across two different requests should do.

None of this is exotic. It is just the difference between a retry that is safe and a retry that is a second charge, and the gap between them is exactly as wide as the code you didn't make atomic.