# System Design Drill Day 9: Idempotency Keys for Payments and Retries A customer taps "Pay" on a shaky mobile connection. The charge goes through on the server, the response never reaches the phone, and the app retries. Whether the customer is now charged twice depends on what the server does with that retry. Networks drop, clients retry, queue jobs are redelivered and payment webhooks arrive twice. This drill designs idempotency for an online marketplace's checkout so that none of those can double-charge a customer or decrement stock twice. ## Core concepts Pressing a lift's call button five times brings one lift, not five. An operation that leaves the system in the same state whether it runs once or many times is idempotent. A `PUT` that says "set this to X" is a typical idempotent write. Safe operations such as `GET` go further and change nothing at all. Real systems deliver at least once. HTTP clients retry, jobs get redelivered and webhooks repeat, so any handler that creates an order, charges a card or decrements stock has to recognise a replay and return the same result without repeating the side effect. The common pattern is an idempotency key. The client sends a key with the request, and the server stores that key together with its result, in Redis or the DB. A request with a key that has been seen before gets the stored result back. Keys are scoped (per user or per payment intent) and given a TTL. Two identical requests arriving at the same instant are the tricky case, handled by claiming the key atomically before any work starts. > **Pause and think:** Is `DELETE /cart/items/42` idempotent? Is it safe? **Answer:** Idempotent, yes: deleting the same item twice leaves the cart in the same state as deleting it once. Safe, no: it changes state. Safe operations like `GET` don't change anything. ## The design problem The marketplace's checkout runs on 3 Laravel servers with shared Redis and a payment provider that sends webhooks. Three kinds of duplicate can occur. The mobile client may retry `POST /checkout` if the network drops after the charge started, the queue job that sends the order confirmation email may run twice, and the payment webhook may be delivered twice. Design a way to prevent double charges and double stock decrements. Cover where the key comes from, what you store and where (Redis or MySQL), the TTL and scope, the response on a replay, and how it fits with the Day 8 checkout lock. Then explain what breaks if the key is missing, or if it's only stored after the charge. > **Try the problem before reading the solution.** ## Questions to think about 1. What's the difference between idempotent and safe, and why is checkout neither by default? 2. Two identical checkouts with the same key hit two servers in the same millisecond. How do you avoid charging twice? 3. Should the webhook handler deduplicate on the provider's event id or on your own checkout key? ## Model solution ### Claim the key first The client sends the key with `POST /checkout`. Before doing anything else, the server claims it atomically, with Redis `SET ... NX` or an insert under a unique index, and marks it `processing`. A second request that finds the key still `processing` gets `409` and retries the same key shortly afterwards. Give the `processing` state a short lease, for example a `locked_until` timestamp a minute or two ahead (or a short TTL on the Redis key). If a worker crashes after claiming the key, the key stays `processing` and every retry gets `409` until that lease expires; after that, a retry can take the key over, and the provider's idempotency key still prevents a second charge. Store a hash of the request body with the key as well. If a client reuses a key with a different body (a different cart, a different amount), that's a client bug, and the server should refuse it with `422` (or `409`) instead of replaying a response that belongs to another request. ### Side effects once With the key claimed and the Day 8 stock lock held, charge once, then decrement stock and write the order together in one DB transaction, as in Day 3. Then mark the key `completed` and store the response body. Later retries, including the one on the second server, find the completed record and don't charge again. ![Sequence diagram: two requests with the same idempotency key reach two servers, one wins the atomic claim and charges once, the other gets 409, and a later retry receives the stored response](https://cdn.hashnode.com/uploads/covers/5fdde6221485f22e24b34a2c/f13844ee-dcf3-4af8-a3f3-1cfdc4822c7c.png align="center") *Figure 1: Claim first, then do the work. The loser of the race gets a 409, a later retry gets the stored result.* ### Each entry point gets its own dedup For the charge call, also pass your key to the provider's own idempotency option on the charge API, so even a crash between charging and saving can't produce a second charge. Providers only remember those keys for a limited window (Stripe keeps them for at least 24 hours), so a retry that comes much later can't rely on the provider's key alone. The webhook is keyed on the provider's event id in a `processed_webhooks` table with a unique index; a duplicate event id is acknowledged and ignored. The queue job receives the checkout or payment id and checks a flag such as `confirmation_email_sent` before sending the email, which makes a redelivered job a no-op. ![](https://cdn.hashnode.com/uploads/covers/5fdde6221485f22e24b34a2c/6f8816c8-3617-42a3-a164-64f6e868f582.png align="center") *Figure 2: Every entry point deduplicates on its own id, with DB unique constraints as the last line.* ### Getting it wrong If the key is stored only after the charge, the first request can charge, crash before saving, and leave the retry free to charge again. The claim has to come first. > **Pause and think:** A webhook arrives before your checkout request has finished saving the order. What should the webhook handler do? **Answer:** Record the event id (so duplicates are ignored) and link it to the order using your own checkout or payment id. If the order row isn't there yet, the handler shouldn't invent one. Either return a non-2xx so the provider retries later, or store the event and let a job apply it once the order exists. ### TTL Keep keys for at least the client retry window plus the webhook redelivery window, often 24 hours to 7 days. DB rows can be kept longer for auditing. That TTL is for your own keys; the provider's window is separate and may be shorter. ### Answers to the questions 1. Safe means no state change. Idempotent means repeats have no extra effect. Checkout creates a new order and charge each time it runs, so it's neither until you add a key. 2. The atomic claim (`NX` or the unique index) lets exactly one request win. The loser sees `processing` and gets a `409`. 3. Both, for different jobs. Use the provider's event id for deduplicating webhooks, since the provider generates and retries those. Use your checkout or payment id to link the webhook to the right order. ## Interview one-liner > "Claim the key before any side effect, deduplicate every entry point (client, queue, webhook) on its own id, and let DB unique constraints be the final guard." ## Conclusion An idempotency key only protects you if it's claimed before the first side effect. A key saved after the charge can't stop a retry that arrives in between, so claim first, give every entry point its own dedup id, and keep a unique constraint underneath as the last guard. ## Try it yourself Add an `idempotency_keys` table with a unique `key` column, a `status` and a `response` JSON column. Write a middleware that inserts the key as `processing` before the controller runs, returns `409` on a duplicate that's still processing, and replays the stored response on a completed one. Send the same `POST` twice with the same `Idempotency-Key` header and confirm the second call doesn't create a second order. --- This is part of my daily [System Design Drills](https://blog.anishansari.com.np/series/system-design-drills) series.