# System Design Drill Day 8: Distributed Locks with Redis and Fencing Tokens One unit of a hot SKU is left, and two customers press "Buy" at the same moment on two different app servers. Both see "1 in stock", both checkouts succeed, and one of them gets an apology email. A mutex inside one PHP process can't help, because the other request isn't on the same machine. This drill uses Redis locks, with leases and fencing tokens, to stop an online marketplace from overselling or charging twice across three Laravel servers, and shows why the database still has the final say. ## Core concepts What's needed is a way for one request to claim "I'm handling this" and for every server to respect the claim. That's a distributed lock, or more accurately a lease, because it expires. In Redis this is usually `SET key value NX EX ttl`. `NX` sets the key only if nobody holds it, and `EX` makes sure a holder that crashes can't keep the lock forever. Laravel wraps it as `Cache::lock('name', 10)`. Lock exactly what needs protecting, one SKU rather than the whole catalog, or the store ends up processing one checkout at a time. The TTL has to balance two risks. Too long, and a crash blocks everyone for a while. Too short, and the lock expires while the holder is still working. Store a random token as the lock value and release the lock only if it still holds your token; a plain `DEL` can remove another server's lock, as the quiz below shows. That bug only applies when you use raw `SET` and `DEL` yourself. `Cache::lock()` already stores a random owner value and checks it on release, so a Laravel lock can't free someone else's. A fencing token closes the last gap. It's a number that only increases, handed out on each successful acquire, and the database rejects any write carrying an older number, so a slow holder whose lease expired can't overwrite the new owner's work. Neither `SET NX` nor `Cache::lock()` gives you one. You make it yourself, typically with `INCR fence:sku:{id}` right after the lock is acquired, and the database has to store the last token it accepted so it can compare. Redlock, which locks across several Redis nodes, comes up often in interviews, along with its critique: clock drift and long process pauses can break it. A single Redis with a careful TTL, backed by a database check, is often enough. Whichever you use, keep the critical section small and never hold a lock across slow network calls. > **Pause and think:** Your lock value is just the string `"1"`, and you release with a plain `DEL`. What can go wrong? **Answer:** If your work runs longer than the TTL, the lock expires and another server takes it. When you finish and call `DEL`, you delete their lock, not yours. A third request can now get in too. Store a random token and only delete the key if it still holds your token. ## The design problem Three Laravel app servers share one Redis and one MySQL. Checkout can oversell a hot SKU, and retries risk charging twice. Make "reserve N units of SKU X" safe, so two concurrent checkouts can't both succeed when one unit is left. Choose the lock key, the TTL and the lock value. Handle the process dying mid-checkout, Redis being briefly unavailable, and the lock expiring before payment and inventory have both committed. Decide whether DB transactions and constraints also belong behind the lock. > **Try the problem before reading the solution.** ## Questions to think about 1. Why can two servers oversell without a lock, even inside a transaction? 2. What goes wrong if the lock's TTL is shorter than the work? 3. Do a lock and an idempotency key solve the same problem? ## Model solution ### The critical section Take a per-SKU lock, check stock (or write a reservation row) if there's enough, and release. The critical section is only the stock change. Payment and email go onto a queue with an idempotency key (`checkout_id` or `payment_intent_id`) so retries never double-charge. ![Checkout flow: acquire a per-SKU Redis lock, run a conditional stock update in MySQL, release the lock, then queue payment with an idempotency key](https://cdn.hashnode.com/uploads/covers/5fdde6221485f22e24b34a2c/9c73dda5-8676-456f-bb88-0761af75d29b.png align="center") *Figure 1: The lock wraps only the stock change. Payment happens after, outside the lock.* ### Database as backstop I'd still use a transaction with a conditional update, `UPDATE ... SET stock = stock - N WHERE sku_id = ? AND stock >= N`, and check the affected-row count afterwards. The lock reduces contention. The conditional update is what makes overselling impossible, even if two requests slip past the lock. If you could keep only one, keep the conditional update. ### Failure cases If the process dies while holding the lock, the TTL frees it. Reservations should be short-lived as well, for example a "held until T" row that expires, or rolled back on timeout. If the lock expires mid-work, don't commit; re-acquire or abort. A fencing token stops a late request from writing after a new owner has taken over. The SKU row keeps a `fence_token` column, and every guarded write passes the token it got from `INCR`: A stock write that only succeeds with a newer fencing token: ```sql -- :token comes from INCR fence:sku:{id} after acquiring the lock UPDATE skus SET stock = stock - :n, fence_token = :token WHERE sku_id = :id AND fence_token < :token -- reject writers holding an older token AND stock >= :n; -- still never oversell ``` Zero affected rows means either the stock ran out or a newer lock holder has already written, and in both cases the late request must give up. ![Sequence diagram: server A's lease expires during a pause, server B acquires the lock with a higher fencing token, and the database rejects A's late write with the older token](https://cdn.hashnode.com/uploads/covers/5fdde6221485f22e24b34a2c/ec96cbb7-eee1-4ab2-94fa-983d83eb3ddf.png align="center") *Figure 2: The late writer carries an older token, so the database rejects its write.* If Redis is down, I'd fail closed for contended SKUs with a 503 and "try again" instead of selling without protection. When MySQL can take the load, falling back to row locking with `SELECT ... FOR UPDATE` is also reasonable. > **Pause and think:** If the conditional `UPDATE ... WHERE stock >= N` already prevents overselling, why bother with the Redis lock at all? **Answer:** The conditional update is the correctness guarantee. The lock reduces contention, so on a hot SKU you don't have dozens of transactions fighting over the same row and failing. Use the lock to keep things calm and the database constraint as the final word. ### Answers to the questions 1. Two servers can both read "1 left" before either one writes. A plain `SELECT` doesn't lock the row, so both decrement. Fix it with `FOR UPDATE`, a conditional `UPDATE`, or a lock. 2. The lock expires while the first request is still working, a second request grabs it, and both commit. You're back to overselling. That's exactly why fencing and the conditional update matter. 3. No. A lock gives mutual exclusion, which stops overselling. An idempotency key makes retries safe, which stops double charges. They solve different problems, so you need both. ## Interview one-liner > "The lock gives mutual exclusion for a short critical section, idempotency keys make retries safe, and DB constraints are the final word. Together they cover overselling and double-charging on a multi-server Laravel and Redis stack." ## Conclusion A Redis lock is a lease that can expire underneath you, so it should never be the only thing standing between you and an oversold SKU. Put the guarantee in a conditional database update and make payments retry-safe with an idempotency key. ## Try it yourself Seed a product with `stock = 1` in a local Laravel app. Write a checkout route that reads stock, sleeps for a second, then decrements it with an unguarded `UPDATE ... SET stock = stock - 1`. Fire two requests at once and watch stock reach -1. (If you instead write back the value you read, `stock = $read - 1`, you'll get a lost update and stock ends at 0, which is a different bug.) `php artisan serve` handles one request at a time unless you start it with `PHP_CLI_SERVER_WORKERS` set, for example `PHP_CLI_SERVER_WORKERS=4 php artisan serve`, so without that the two requests simply run one after the other. Switch to the conditional `UPDATE ... WHERE stock >= 1` and check the affected rows. Then wrap it in `Cache::lock('sku:1', 5)->block(2, ...)` and see how behaviour changes under a burst of requests. --- This is part of my daily [System Design Drills](https://blog.anishansari.com.np/series/system-design-drills) series.