System Design Drill Day 8: Distributed Locks with Redis and Fencing Tokens

One unit of a hot SKU is left, and two customers press "Buy" at the same moment on two different app servers. Both see "1 in stock", both checkouts succeed, and one of them gets an apology email.

A mutex inside one PHP process can't help, because the other request isn't on the same machine. This drill uses Redis locks, with leases and fencing tokens, to stop an online marketplace from overselling or charging twice across three Laravel servers, and shows why the database still has the final say.

Core concepts

What's needed is a way for one request to claim "I'm handling this" and for every server to respect the claim. That's a distributed lock, or more accurately a lease, because it expires.

In Redis this is usually SET key value NX EX ttl. NX sets the key only if nobody holds it, and EX makes sure a holder that crashes can't keep the lock forever. Laravel wraps it as Cache::lock('name', 10).

Lock exactly what needs protecting, one SKU rather than the whole catalog, or the store ends up processing one checkout at a time.

The TTL has to balance two risks. Too long, and a crash blocks everyone for a while. Too short, and the lock expires while the holder is still working.

Store a random token as the lock value and release the lock only if it still holds your token; a plain DEL can remove another server's lock, as the quiz below shows. That bug only applies when you use raw SET and DEL yourself. Cache::lock() already stores a random owner value and checks it on release, so a Laravel lock can't free someone else's.

A fencing token closes the last gap. It's a number that only increases, handed out on each successful acquire, and the database rejects any write carrying an older number, so a slow holder whose lease expired can't overwrite the new owner's work. Neither SET NX nor Cache::lock() gives you one. You make it yourself, typically with INCR fence:sku:{id} right after the lock is acquired, and the database has to store the last token it accepted so it can compare.

Redlock, which locks across several Redis nodes, comes up often in interviews, along with its critique: clock drift and long process pauses can break it. A single Redis with a careful TTL, backed by a database check, is often enough. Whichever you use, keep the critical section small and never hold a lock across slow network calls.

Pause and think: Your lock value is just the string "1", and you release with a plain DEL. What can go wrong?

Answer: If your work runs longer than the TTL, the lock expires and another server takes it. When you finish and call DEL, you delete their lock, not yours. A third request can now get in too. Store a random token and only delete the key if it still holds your token.

The design problem

Three Laravel app servers share one Redis and one MySQL. Checkout can oversell a hot SKU, and retries risk charging twice.

Make "reserve N units of SKU X" safe, so two concurrent checkouts can't both succeed when one unit is left. Choose the lock key, the TTL and the lock value. Handle the process dying mid-checkout, Redis being briefly unavailable, and the lock expiring before payment and inventory have both committed. Decide whether DB transactions and constraints also belong behind the lock.

Try the problem before reading the solution.

Questions to think about

1. Why can two servers oversell without a lock, even inside a transaction?
2. What goes wrong if the lock's TTL is shorter than the work?
3. Do a lock and an idempotency key solve the same problem?

Model solution

The critical section

Take a per-SKU lock, check stock (or write a reservation row) if there's enough, and release. The critical section is only the stock change. Payment and email go onto a queue with an idempotency key (checkout_id or payment_intent_id) so retries never double-charge.

Figure 1: The lock wraps only the stock change. Payment happens after, outside the lock.

Database as backstop

I'd still use a transaction with a conditional update, UPDATE ... SET stock = stock - N WHERE sku_id = ? AND stock >= N, and check the affected-row count afterwards.

The lock reduces contention. The conditional update is what makes overselling impossible, even if two requests slip past the lock. If you could keep only one, keep the conditional update.

Failure cases

If the process dies while holding the lock, the TTL frees it. Reservations should be short-lived as well, for example a "held until T" row that expires, or rolled back on timeout.

If the lock expires mid-work, don't commit; re-acquire or abort. A fencing token stops a late request from writing after a new owner has taken over. The SKU row keeps a fence_token column, and every guarded write passes the token it got from INCR:

A stock write that only succeeds with a newer fencing token:

-- :token comes from INCR fence:sku:{id} after acquiring the lock
UPDATE skus
SET stock = stock - :n, fence_token = :token
WHERE sku_id = :id
  AND fence_token < :token   -- reject writers holding an older token
  AND stock >= :n;           -- still never oversell

Zero affected rows means either the stock ran out or a newer lock holder has already written, and in both cases the late request must give up.

Figure 2: The late writer carries an older token, so the database rejects its write.

If Redis is down, I'd fail closed for contended SKUs with a 503 and "try again" instead of selling without protection. When MySQL can take the load, falling back to row locking with SELECT ... FOR UPDATE is also reasonable.

Pause and think: If the conditional UPDATE ... WHERE stock >= N already prevents overselling, why bother with the Redis lock at all?

Answer: The conditional update is the correctness guarantee. The lock reduces contention, so on a hot SKU you don't have dozens of transactions fighting over the same row and failing. Use the lock to keep things calm and the database constraint as the final word.

Answers to the questions

1. Two servers can both read "1 left" before either one writes. A plain SELECT doesn't lock the row, so both decrement. Fix it with FOR UPDATE, a conditional UPDATE, or a lock.
2. The lock expires while the first request is still working, a second request grabs it, and both commit. You're back to overselling. That's exactly why fencing and the conditional update matter.
3. No. A lock gives mutual exclusion, which stops overselling. An idempotency key makes retries safe, which stops double charges. They solve different problems, so you need both.

Interview one-liner

"The lock gives mutual exclusion for a short critical section, idempotency keys make retries safe, and DB constraints are the final word. Together they cover overselling and double-charging on a multi-server Laravel and Redis stack."

Conclusion

A Redis lock is a lease that can expire underneath you, so it should never be the only thing standing between you and an oversold SKU. Put the guarantee in a conditional database update and make payments retry-safe with an idempotency key.

Try it yourself

Seed a product with stock = 1 in a local Laravel app. Write a checkout route that reads stock, sleeps for a second, then decrements it with an unguarded UPDATE ... SET stock = stock - 1. Fire two requests at once and watch stock reach -1. (If you instead write back the value you read, stock = $read - 1, you'll get a lost update and stock ends at 0, which is a different bug.) php artisan serve handles one request at a time unless you start it with PHP_CLI_SERVER_WORKERS set, for example PHP_CLI_SERVER_WORKERS=4 php artisan serve, so without that the two requests simply run one after the other.

Switch to the conditional UPDATE ... WHERE stock >= 1 and check the affected rows. Then wrap it in Cache::lock('sku:1', 5)->block(2, ...) and see how behaviour changes under a burst of requests.

---

This is part of my daily System Design Drills series.