AlgoInsight

System Design

Idempotency: The One Word That Saved a Payment System

A customer got charged twice because of a network timeout. The bug wasn't in our payment logic. It was that I didn't yet understand one word.

By Admin2026-06-1810 min readidempotencydistributed-systemspaymentsretriesreliabilitysystem-design

At Razorpay we were moving eight hundred crore rupees a day through our systems. One afternoon a customer was charged twice for the same order. The payment code was correct. The amount was correct. We had not written a bug in the sense most people mean the word.

What happened was this: the customer's app sent the charge request, the network stalled for a few seconds, the app gave up waiting and retried - and our server happily processed both the original request (which had actually gone through, the response just hadn't made it back) and the retry. Two charges. One angry customer. A very long evening.

The fix was not a clever algorithm. It was one property I hadn't fully internalized yet, even though I'd been building distributed systems for years. The word was idempotency. This is the explanation I wish someone had handed me before that evening.

What "idempotent" means

An operation is idempotent if doing it many times has the same effect as doing it once.

That's it. Turning a light switch to "on" is idempotent - flip it on five times, the light is just on. Adding ₹500 to an account is not idempotent - do it five times and you've added ₹2,500.

You already rely on this with HTTP without thinking about it:

That last line is where payment systems, and careless candidates, get hurt.

Why this isn't optional: retries are guaranteed

Here's the uncomfortable truth about networks that took me too long to respect: you cannot tell the difference between "my request failed" and "my request succeeded but the response got lost." From the client's side, both look identical - silence.

So every robust client does the only sane thing: it retries. Mobile apps retry. Payment gateways retry. Message queues retry. Your load balancer retries. This is called at-least-once delivery, and it's the default behavior of basically every real distributed system, because the alternative - giving up after one try - loses data.

Which means retries aren't an edge case you might hit. They're a certainty. And if your write operation isn't idempotent, every retry is a landmine. My double-charge wasn't bad luck. It was the system working exactly as designed, on top of an operation that couldn't tolerate being repeated.

Warning: "We'll just make sure the client only sends the request once" is not a fix. You do not control the network, the user's flaky connection, the intermediate proxies, or the retry logic three layers down your stack. The only durable fix is to make the server safe to call twice.

The fix: idempotency keys

The pattern is simple and it's what every serious payments API uses. The client generates a unique idempotency key - a random string - for each logical operation, and sends it with the request. The server remembers which keys it has already processed.

Now a retry is harmless. The duplicate request carries the same key, the server recognizes it, and instead of charging again it just replays the original answer.

sequenceDiagram
    participant C as Client
    participant S as Server
    participant DB as Dedup Store
    C->>S: POST /charge (key=abc, ₹500)
    S->>DB: seen key "abc"?
    DB-->>S: no
    S->>S: charge ₹500 ✓
    S->>DB: store abc -> {txn: 9981}
    S--xC: response lost in network ✗
    Note over C: timeout - client retries
    C->>S: POST /charge (key=abc, ₹500)
    S->>DB: seen key "abc"?
    DB-->>S: yes -> {txn: 9981}
    S-->>C: returns SAME result (no second charge)

This sequence diagram shows a client sending a charge request with idempotency key abc for 500 rupees. The server checks a dedup store, finds the key is new, performs the charge, and stores the result. The response is then lost in the network. The client times out and retries with the same key abc. This time the server finds the key already exists in the dedup store and returns the same stored result, so no second charge happens.

Here's the whole idea in runnable form:

processed = {}   # idempotency_key -> stored result

def charge(key, amount):
    if key in processed:                          # already handled this key
        return f"[deduped] {processed[key]}"
    txn_id = abs(hash(key)) % 10000               # pretend: do the real charge
    result = f"charged Rs.{amount}, txn_id={txn_id}"
    processed[key] = result
    return result

key = "order-7f3a9"
print(charge(key, 500))   # first attempt - real charge happens
print(charge(key, 500))   # network retry - safely deduped
print(charge(key, 500))   # another retry - still deduped

Three calls, one charge. The second and third return the exact same transaction ID, because they replayed the stored result instead of creating new work.

Where it gets hard (and where interviews go)

The toy version above hides three real problems. Knowing them is what moves you from "I've heard of idempotency" to "I've shipped it."

1. Where do the keys live, and for how long? That processed dictionary is really a database table or a Redis store. It can't grow forever, so keys get a TTL - often 24–72 hours, long enough to cover any realistic retry window. After that, the key expires and could be reused, which is fine because no sane client retries a day-old request.

2. The concurrency race. What if two copies of the request arrive at the same time, before either has finished writing the key? Both check the store, both see "not seen," both charge. You've recreated the original bug. The fix is to make the key claim atomic - a unique constraint on the key column, or an atomic "insert if not exists." The database enforces that only one of the racing requests wins the right to do the work; the other gets rejected and retries into the deduped path.

Tip: A unique constraint on the idempotency key is the unglamorous hero here. Let the database be the referee. Application-level "check then act" logic will always have a race window between the check and the act.

3. Idempotency is not "exactly-once delivery." You'll hear people promise "exactly-once." In a distributed system with an unreliable network, exactly-once delivery is essentially impossible - you can't guarantee a message arrives precisely one time. What you can do is accept at-least-once delivery (messages may arrive multiple times) and make your processing idempotent (duplicates are harmless). The combination behaves like exactly-once from the outside. That distinction - "we don't prevent duplicates, we make them safe" - is exactly the kind of precision that gets you a strong-hire signal.

Make idempotency a habit, not a patch

The deepest shift this taught me: idempotency isn't a feature you bolt onto payments after something breaks. It's a design default for any operation that changes state in a distributed system - creating an order, sending a notification, deducting inventory, publishing an event. Ask of every write you design: "what happens if this runs twice?" If the answer is "something bad," you have an idempotency key to add.

I'd built distributed systems for years before a double-charge made me feel that lesson in my chest. You don't have to learn it the same way. We dig into idempotency, retries, and the rest of the reliability toolkit in System Design Fundamentals - the building blocks an interviewer expects you to reach for without being prompted.

Go deeper

More articles