# System Design Drill Day 10: Circuit Breakers, Timeouts and Degradation The payment gateway starts timing out. The HTTP client retries, the queue retries, and workers pile up waiting on calls that won't come back. Soon pages that have nothing to do with payments are slow as well, and one sick dependency has dragged down the whole app. This drill protects an online marketplace's checkout and login with timeouts, circuit breakers, bulkheads and fallbacks, so a flapping payment or SMS provider stays a contained problem. ## Core concepts When a dependency such as a payment gateway, an SMS provider or even Redis starts failing or slowing down, blind retries make things worse. Requests queue up, workers stay busy waiting, and the healthy parts of the app go down with the sick one. The first defence is a timeout, a cap on how long a single call may wait. A circuit breaker works like the trip switch in a fuse box. It watches recent failures, and past a threshold it opens: for a cool-down period every call fails immediately without touching the dependency. After the cool-down it goes half-open and lets one probe request through. If the probe succeeds, the breaker closes and normal traffic resumes. ![State diagram of a circuit breaker moving from closed to open after too many failures, to half-open after the cool-down, and back to closed or open depending on the probe](https://cdn.hashnode.com/uploads/covers/5fdde6221485f22e24b34a2c/64d195a4-e56e-406b-9fe1-0b365b83c12a.png align="center") *Figure 1: The three breaker states and what moves you between them.* Bulkheads give each dependency its own worker pool or queue, so SMS trouble can't starve checkout. Fallbacks return a reduced answer instead of an error, such as a cached value, "pay later", or a different OTP channel. These pieces depend on each other. Timeouts without a breaker still stampede a sick dependency, and a breaker without timeouts never sees the calls that hang. Retries into an open circuit are wasted work unless you check the circuit before retrying or enqueuing. > **Pause and think:** You've set a 3 second timeout on the payment call but no circuit breaker. The provider is down. What does your system still do wrong? **Answer:** Every request still calls the dead provider and waits the full timeout before failing. Workers and PHP processes stay tied up, and the provider gets hammered while it's trying to recover. A breaker would fail fast without making the call at all. ## The design problem The marketplace (Laravel, 3 app servers, Redis, queues) calls an eSewa/Khalti-style payment API when an order is placed, and an SMS OTP provider for login and high-value checkout. Lately both flap: about 30% of calls time out for 2 to 5 minutes, then recover. HTTP client retries and queue job retries pile up, checkout p95 jumps from about 400 ms to several seconds, and workers clog on SMS jobs. Decide where the timeouts live (HTTP client, job or gateway). Keep one circuit per dependency with its state in Redis so all 3 servers agree, and choose the thresholds, cool-down and half-open probe. Then define what checkout and login return while each circuit is open, and how all of this interacts with the Day 7 rate limits, Day 8 locks, Day 9 idempotency keys and queue retries. > **Try the problem before reading the solution.** ## Questions to think about 1. Why share the circuit state in Redis instead of keeping a breaker on each server? 2. The payment circuit is open. Should checkout fail now, queue the payment for later, or mark the order `payment_pending`? 3. The SMS circuit is open. What should login do, and what should an NPR 50k checkout do? ## Model solution ### Timeouts and circuits Every outbound HTTP call gets an explicit timeout; in Laravel's HTTP client that's a single `->timeout()` call. Without it, the client waits up to its default of 30 seconds, which is far too long for a checkout request. Keep one circuit per dependency (payment and SMS) with its state in Redis, so all 3 servers trip and recover together. While a circuit is open, calls fail fast without reaching the provider, and after the cool-down one half-open probe decides whether it closes. ![Architecture: three Laravel servers read shared circuit state from Redis, with separate queues and workers for payment and SMS so one failing provider cannot starve the other](https://cdn.hashnode.com/uploads/covers/5fdde6221485f22e24b34a2c/d0230a96-098c-43ea-bf38-bb3fb68aefc4.png align="center") *Figure 2: Shared circuit state in Redis, plus separate queues per dependency (bulkheads).* ### When the SMS circuit is open Fall back to another channel, such as an in-app OTP if the app has one. If SMS is the only channel, show "SMS delayed, try again in a minute", which is a product decision. For high-value checkout I'd fail closed or require another factor, since protecting money matters more than conversion. Day 7 rate limits still apply to whichever channel you fall back to. ### Retries Allow 0 to 1 HTTP retries on payment calls unless you're passing the provider an idempotency key. Queue jobs should use exponential backoff and release themselves while the circuit is open, so workers don't spend their retry budget on guaranteed failures. How many attempts a job gets depends on your queue config, the job's `$tries` property or the worker's `--tries` option; with the default of one attempt, a released job is never run again, so set it (or `retryUntil()`) deliberately. The Day 8 lock still wraps only the stock change, on the success path after payment confirms. ### If Redis (the circuit store) is down Fall back to a local in-memory breaker on each server, which is still better than calling blindly. For payment, lean toward failing closed; for non-critical SMS such as promotions, fail open. It's the same reasoning as the Day 7 Redis-down case. ### Observability Log every state change in both directions, graph it, and alert when a circuit stays open too long. Make half-open probes visible as well, so you never have to guess why one request suddenly succeeded. > **Pause and think:** Why should a queued payment job *release* itself back onto the queue while the circuit is open, instead of just throwing an exception? **Answer:** Throwing counts as a failure, so the job burns through its retries on failures you already know are coming and ends up in `failed_jobs`. Releasing it with a delay puts it back to run once the circuit has had time to close. In Laravel a release still counts as an attempt, so limit the job with `maxExceptions` or a time-based `retryUntil()` instead of a low `$tries`. ### Answers to the questions 1. With a breaker per server, each server has to discover the outage on its own and keeps hammering the provider until it does. A shared circuit trips once for everyone, and one probe speaks for all 3 servers. 2. Failing now is simple and honest, but you lose some sales. Queuing for later keeps the sale, but the user waits with no answer. `payment_pending` is the nicest experience, but you need reconciliation and stock holds that expire. For a wallet API that flaps for a few minutes, a clear "try again shortly" with the cart kept is usually the right call. 3. For login, switch channels or show a short delay, since locking everyone out is worse. For an NPR 50k checkout, fail closed or require another factor. ## Interview one-liner > "Timeouts on every call, one Redis-backed circuit per dependency, bulkheaded queues that release jobs while a circuit is open, and a deliberate fail-open or fail-closed choice per feature, with money always failing closed." ## Conclusion A timeout limits how long one call can hurt you, and a shared circuit stops every server from hammering a provider that's already down; you need both. Whatever else you decide, anything that moves money should fail closed. ## Try it yourself Point a Laravel HTTP call at a local endpoint that sleeps longer than your timeout. Build a small breaker from three Redis keys: a failure counter with a short TTL, an `open_until` timestamp, and a probe lock. Make the call fail fast while `open_until` is in the future, and let exactly one request through after it passes. Watch the logs while you start and stop the slow endpoint. --- This is part of my daily [System Design Drills](https://blog.anishansari.com.np/series/system-design-drills) series.