System Design Drill Day 13: Designing a Multi-Channel Notification System

Marketing starts a big campaign, and a few minutes later support starts hearing "I'm not receiving my OTP". The OTPs were queued fine. They're sitting in the same queue as the campaign, behind the whole blast, and they expire before they go out. Keeping those two kinds of message apart is the first thing this design has to get right.

It's a good first "classic" design, because it reuses nearly everything from the first twelve days: queues, retries, idempotency, rate limits, circuit breakers and the outbox. The service takes an online marketplace's business events and turns them into OTPs, emails, pushes, merchant webhooks and marketing blasts.

Core concepts

One business event, say "order placed", should become a receipt email for the buyer, a push notification, and a webhook for the merchant, each on the right channel and in the right language. That translation is the notification system's job.

The app only emits the event; a notification service decides who gets what. Checkout code should never call the SMS API directly.

Fan-out can happen at write time or at read time. Fan-out on write creates one delivery record per recipient when the event happens. It's simple and right for transactional messages, and for marketing blasts sent in batches. Fan-out on read stores the event once and shows it when users open their inbox, which is cheaper for huge audiences such as an in-app banner for everyone.

Each channel (email, SMS, push, webhooks) gets its own queue and workers, so a slow email provider can't hold up SMS. Each channel is also split by priority, and critical messages like OTPs and receipts never wait behind a 200k marketing blast.

Figure 1: Events in, deduplicated delivery rows out, and a separate lane per channel and priority.

Before sending, the service checks preferences such as opt-outs, quiet hours and language, and renders a versioned template at send time. Delivery is at-least-once, so duplicates will happen; a dedup key like event_id + user_id + channel stops a duplicate SMS, and the second charge that comes with it.

Each message also moves through states (queued, sent, delivered, failed), updated by provider callbacks. That record drives retries, support lookups and metrics.

Webhooks are notifications sent to machines. Sign the payload with an HMAC, retry with exponential backoff and jitter, move the message to a dead-letter table after N attempts, and let merchants replay it.

Pause and think: You want to show a "Dashain sale starts tomorrow" banner to every user in the app. Fan-out on write or on read?

Answer: Fan-out on read. Store the banner once and show it to whoever opens the app. Writing one row per user for the same message is wasted work and storage.

The design problem

Design the marketplace's notification service on Laravel and Redis queues. It sends OTP SMS, order emails, push notifications, merchant webhooks and marketing blasts of around 200k recipients.

Cover the queues and priorities, and dedup. Explain what happens when the SMS provider is down and when a merchant's webhook endpoint is down for 6 hours, and how support can check whether a customer's OTP was delivered.

Try the problem before reading the solution.

Questions to think about

1. Why must OTPs never share a queue with marketing?
2. A merchant's webhook endpoint is down for 6 hours. What happens to their events?
3. Where exactly do you stop a duplicate SMS?

Model solution

Sending once

Each delivery is a row with a unique dedup_key. Before calling the provider, the job locks the row and checks its status, so a retried job doesn't send twice. If the provider supports an idempotency key, pass it the notification id (Day 9).

SMS provider down

Each provider has its own circuit breaker (Day 10). When it opens, OTPs fail over to a second SMS provider. If both are down, the UI offers "send code by email instead", so jobs don't just pile up silently while the user waits for a code.

Merchant down for 6 hours

Backoff keeps you from hammering the endpoint. After N consecutive failures, mark it unhealthy and email the merchant. New events keep arriving as pending rows in MySQL, which is cheap compared with holding them in Redis memory. When the endpoint recovers, drain them in order or let the merchant replay.

Webhooks are at-least-once, so merchants must dedupe by event id, and your API docs should say so.

Figure 2: What happens to a merchant's webhooks while their endpoint is down, and after it comes back.

Support lookup

Query notifications by user and time. Provider callbacks fill in delivered_at and the status, so support can see whether the OTP was queued, sent, delivered or failed, along with any provider error.

Scale note

200k marketing pushes are fan-out on write in batches, which is fine. An in-app banner for everyone should be fan-out on read: stored once and shown to all users, without 200k rows.

Pause and think: A retried SMS job and the original both pass the dedup check at the same time. How does the design above still send only one SMS?

Answer: The job locks the delivery row before calling the provider and checks its status under that lock. Only one job sees "queued" and moves it to "sent". The other sees it's already handled and stops.

Answers to the questions

1. Marketing can drop 200k jobs into a queue at once. An OTP stuck behind them arrives after it has expired, so login and checkout break during every campaign.
2. Retries back off, the endpoint gets marked unhealthy, the merchant gets an alert email, and new events pile up as pending rows. On recovery they drain in order, or the merchant replays them, and the merchant dedupes by event id.
3. Mainly in two places. The unique dedup_key on insert stops duplicate rows. Locking the row and checking its status before the provider call stops a retried job from sending twice. On top of that, the provider's idempotency key covers a crash right after the send.

Interview one-liner

"Decouple events from delivery, fan out on write into deduplicated delivery rows, give each channel its own priority queues so OTPs never wait behind marketing, put a circuit breaker on each provider, and treat webhooks as signed, retried, replayable notifications."

Conclusion

Give OTPs and receipts their own high-priority lane before you build anything else, or every marketing campaign will break login and checkout. After that, the delivery row with its dedup key and status is what stops double sends and lets support answer "did it arrive?".

Try it yourself

In a Laravel app, create a notifications table with a unique dedup_key and a status column. Write one job per channel (just log the "send"), and put OTPs on a sms-high queue and marketing on a push-low queue.

Run a worker with --queue=sms-high,push-low, dispatch a few thousand marketing jobs and then one OTP, and time how long the OTP waits. Repeat with both on a single queue and compare.

---

This is part of my daily System Design Drills series.