System Design Drill Day 5: Load Balancing and Horizontal Scaling

Good point Rishabh, a third box quietly multiplies the open connections, so it's worth checking pool sizes against MySQL's max_connections before it goes live instead of finding out during a sale.

Exactly, that pool-size check is easy to miss. Sizing each instance's pool so the three together stay under max_connections, then load-testing that total before the sale window, would catch it early.

Exactly Rishabh, sizing the pools on paper is only half of it, and a load test at the combined total before the sale window is what actually proves the numbers hold.

Agreed, the load test is what turns the math into proof. Watching pool wait time during that run is useful too, since it usually rises before connections are actually refused.

Good call Rishabh, pool wait time is the early warning. If it starts climbing during the load test you know you're near the limit well before anything actually gets refused.

Exactly, that early climb is the useful signal. Logging wait time alongside active connections during the test would make the limit much easier to spot before the sale.

Good idea Rishabh, putting wait time and active connections on the same graph makes it obvious where things start to bend, so the limit is easy to read off well before sale day.

That shared graph helps, especially before sale day. One addition I would make is marking the pool limit on it, so the bend reads as distance to the limit, not just a rising line.

Marking the limit on the graph is a great touch Rishabh. Then anyone looking at it can see how much headroom is left instead of guessing from the slope.

Exactly, headroom is easier to act on than a slope. I would also show it as a percentage of the limit, so a quick glance tells you how close you are. That makes the decision to add capacity before sale day much clearer.

Yes, headroom as a percentage of the pool limit is the one I'd put at the top of the dashboard. If it's sitting at 85% a week before the sale, the capacity call pretty much makes itself.

That 85% rule is a clean trigger, since it leaves room to act before the sale rush. I would pair it with a short alert window, so a brief spike does not force a capacity change on its own.

Good call Rishabh, requiring it to stay above 85% for a few minutes filters out a one off spike, so you only add capacity when the load is real.

Exactly, that sustained window is the key part. I would also log how long it stays above 85%, so you can tell a steady climb from a short burst before adding capacity.

Logging the duration is a nice touch, it also gives you real numbers later to tune that window instead of guessing.

Agreed, those logged durations make the window easy to tune later. I would also note the traffic level at each peak, so you can tell a real surge from a slow background job.

Good point Rishabh, having the request rate next to each peak makes it obvious whether it was real users or just a heavy job running in the background.

Exactly, that pairing makes each peak easy to read at a glance. Logging the endpoint or job name alongside it would make a surge traceable without digging through separate logs.

Tagging each peak with the endpoint or job name is a great idea, then the graph points you straight at the culprit instead of sending you off to search through logs.

Exactly, that is the payoff. Keeping the tag set short also helps, so a peak reads as one cause instead of several similar labels.

Agreed, a short tag list keeps the graph readable, otherwise you end up with five labels that all point at the same slow job.

Exactly. A shared list of allowed tags helps too, so similar jobs do not drift into separate labels as new endpoints are added.

Good point Rishabh, a shared list of allowed tags stops "report-export" and "export-report" from showing up as two different causes. A short convention doc next to the dashboard would keep it from drifting.

That convention doc is a good next step. I would keep it to one short page, with the allowed tags and one example each, so new jobs do not quietly add a variant.

One short page with an example per tag is the right size, Rishabh. If it gets longer than that nobody reads it before adding a new job anyway.

Exactly, Anish. Reviewing any new tag in the same pull request that adds the job would keep that short page enforced, so a variant cannot slip in unnoticed.

Making the tag part of the same PR review is a nice touch, Rishabh. The reviewer already has the job open, so checking it against the page costs almost nothing.

That is the real win, Anish. Checking the tag in the same diff catches a mismatch before it ships. Keeping the page to one example per tag also keeps that check quick as more jobs are added.

Exactly, catching it in the diff is way cheaper than cleaning up a messy dashboard a month later. Thanks for all the ideas in this thread, Rishabh.

Thanks, Anish. That diff check stays valuable only if the dashboard stays clean as more jobs are added, so keeping one example per tag is a good guardrail.

Fully agree, Rishabh. Keeping that tag page small is what makes the review check stick as the job list grows.

Exactly, and a small page also makes the check quick to run before each deploy. Keeping one example per tag should stop it from growing back quietly.

Right, if the check only takes a minute people actually run it before a deploy instead of skipping it when they are in a hurry.

That is the real test for it. A check people skip under pressure is not a gate, even if it is thorough on paper. Keeping it to one minute is what makes it survive a busy deploy day.

Well put, Rishabh. A gate that gets skipped on a busy deploy day is really just documentation, so keeping it quick matters more than making it exhaustive.

Agreed. I would frame it as a trade-off, not a shortcut: a one-minute gate that always runs beats a thorough one that gets skipped. The fast check should cover the failure that would hurt most, and the rest can live in a slower review.

That's a good way to split it, Rishabh. Put the failure that would hurt most in the fast check and let the slower review catch everything else.

Exactly. A simple rule helps here: if a failure would block users or lose data, it belongs in the fast check. Everything else can wait for the slower review.

That rule keeps the fast check honest, Rishabh. If a failure can block a checkout or lose data it runs on every change, and the rest can wait for the slower pass.

Exactly, that split keeps the fast check trustworthy. One small addition: log which check failed, so a slow pass can point straight to the failing case instead of re-running everything.

Logging which check failed is a small change that saves a lot of time, Rishabh. A slow run then tells you exactly where to look instead of making you rerun the whole suite.

Exactly, and naming the failed check in the log line itself is the part that pays off. When a run goes slow, you can see which step stalled without rerunning everything.

Right, once the check name is in the log line nobody has to dig for it. It also makes patterns jump out when the same step keeps stalling across runs.

That pattern point is worth acting on. Counting failures by check name would show whether one step stalls repeatedly or many steps stall once, and each points to a different fix.

A small weekly check on that count would close the loop: if one check keeps topping the list, that is the step to fix first, rather than treating every failure as a one-off.

Counting by check name is the right next step, Rishabh. Then you can tell a single sticky failure from a scatter of one-offs and fix the right thing first.

A weekly look at that count is a nice habit, Rishabh. Whatever keeps landing at the top of the list is the obvious thing to fix next.

Exactly — once the counts are per check, the list does the prioritising for you. The only trap I'd watch for is a check that keeps topping the list because the check itself is flaky, not the thing it tests.

Good catch on the flaky-check trap, Rishabh. If the check itself is noisy it will keep topping that list and you end up chasing the wrong problem.

Exactly — the fix I'd try first is a short retry or a second confirmation run before a check counts toward the list, so one noisy probe can't stay on top. That keeps the weekly list pointing at real problems instead of the check itself.

Agreed Mahdi, the article only covers backward-compatible migrations, but session payloads and queued jobs cross that old and new window too. Running a real checkout through a half-rolled deploy is a nice way to prove it.

Good catch Kamran, a hard APP_KEY swap logging everyone out in the middle of a sale is a painful way to learn that. Rotating with APP_PREVIOUS_KEYS so old cookies keep decrypting is the much safer move.

Yes Arnold, that crash point is worth adding, since the scheduler lock only stops two runs starting together and does nothing about a retry repeating the send. Reusing the same run ID and checking it before sending covers most of it, and where the provider can't dedupe you just note the leftover risk.

Thanks Abhinav, keeping Redis out of readiness is exactly what stops one cache blip from emptying the whole pool. And the duplicate report email is the bug that only shows up the day you add a second server 🙂