System Design Drill Day 3: Queues and Async Jobs for Checkout

A customer clicks "Place order" and the spinner keeps spinning, because the checkout controller is waiting on an SMTP server somewhere. When that mail server has a bad day, checkout has one too.

This drill moves the slow parts of an online marketplace's checkout onto a Redis queue in Laravel. Along the way we'll cover retries, jobs that run twice, and what the customer sees while work is still pending.

Core concepts

Everything done synchronously inside a request, such as sending an email, notifying a merchant or resizing an image, adds to the response time and ties the endpoint's health to each of those dependencies. A queue breaks that link. The HTTP handler writes a small job and returns quickly, and a worker picks the job up shortly after.

Most queues deliver each job at least once. That sounds reassuring, but "at least once" also allows "twice": a worker can crash after doing the work but before acknowledging it, and the job comes back.

Failures get retried, usually with backoff, so the wait grows between attempts and a service that is already down isn't hammered. Because a job can run more than once, it also has to be idempotent. Running it again must not charge someone twice or send a second email.

In Laravel this means a Redis or database queue, jobs that implement ShouldQueue, and proper failed-job handling. The queue itself is simple to set up; most of the design work goes into idempotency.

Pause and think: Your queue promises at-least-once delivery. The email job crashes right after the SMTP server accepted the message but before you saved anything. What happens on retry?

Show answer

The job runs again and the customer gets a second email. At-least-once means you can't fully close that window, you can only make it small: check a flag like confirmation_email_sent_at before sending and set it straight after a successful send.

The design problem

When a customer places an order in the marketplace's checkout, stock has to be decremented, the customer gets a confirmation email, and the merchant gets notified so they can pack the order.

Decide what stays in the HTTP request and what goes on a Redis queue, and what the job payload carries. Choose a retry policy for when SMTP or the merchant notification channel is down. Show how a job that runs twice avoids sending two emails, and what the user sees while a job is pending or after it has failed for good.

Try the problem before reading the solution.

Questions to think about

1. Why is "at least once" the default for most queues, and what does that force you to design for?

2. When would you use one job, and when two separate jobs, for the customer email and the merchant notification?

3. How would you spot stuck or failed jobs in production without guessing?

Model solution

Keep the HTTP path thin

The request validates the cart and then, inside one DB transaction, creates the orders row and decrements stock with lockForUpdate. If there isn't enough stock (two buyers racing for the last unit), the transaction rolls back and the customer gets a clear "out of stock" instead of an order that has to be cancelled later. Stock is the one thing that must be right before you say yes, so it stays in the request, the same way Days 4 and 5 handle it.

Only after the commit are the jobs dispatched, and then the request returns 201 with the order id. Use ->afterCommit() on the dispatch, or set after_commit to true for the connection in config/queue.php; otherwise a fast worker can pick up the job before the order row is committed and find nothing. Email and merchant notifications stay out of the request.

Two jobs, small payloads

I'd put two follow-up jobs on a Redis queue, one for the customer email and one for the merchant notification, though one orchestrator job with two steps also works. The payload is order_id, possibly with user_id. The full order blob stays out, because the worker reloads from the DB and so works with current data instead of a snapshot.

Figure 1: Order and stock commit together in the request. Jobs are dispatched after commit, and workers handle email and the merchant notification.

Email job

The job loads the order and the user and sends "order placed". To make a second run harmless, add orders.confirmation_email_sent_at. If it's already set, the job returns without sending; otherwise it sends and then sets the column. A unique "mail sent" log row works just as well.

Allow 3 to 5 attempts with exponential backoff, then let the job land in failed_jobs and raise an alert. Endless automatic resends help nobody.

Figure 2: The email job checks the sent flag first, so a duplicate run does nothing.

Merchant notification job

The second job tells the merchant (or the warehouse) that a new order is waiting, by SMS, push or webhook. It needs the same protection as the email. Before sending, insert a row into an order_notifications table with a unique key on order_id + type; if the insert hits the unique key, a previous run already handled it and the job stops.

The same retry rules apply: a few attempts with backoff, then failed_jobs and an alert, so support can resend by hand if the merchant never heard about the order.

Checkout idempotency keys

Jobs aren't the only thing that can run twice. A customer on a shaky connection taps "Place order", the response gets lost, and the app sends the same request again. To stop that from creating a second order, the client generates a unique key per checkout attempt and sends it in an Idempotency-Key header. The server stores it in a column with a unique index alongside the order (or in a small idempotency_keys table) together with the response. A retry with the same key hits the unique index, and the server returns the stored response instead of placing the order again. Later drills lean on this key, and Day 9 covers it in depth.

What the user sees

A successful API response means the order was accepted. It doesn't promise the email has arrived. An order status such as confirmed or fulfillment_pending can make that visible. Admin and support staff should be able to see failed jobs and replay them, and checkout should never block on SMTP.

Answers to the questions

Show answer

1. Because it's the only promise a queue can keep cheaply. A job is removed only after the worker finishes and acknowledges it. If the worker dies first, the job becomes available again (on Laravel's Redis queue, after retry_after seconds) and another worker runs it. Exactly-once would need the queue and your side effect, like an SMTP send, to commit together, and they can't. So every job has to be safe to run twice.

2. Two jobs when the steps can fail independently, which they do here. If SMTP is down, the merchant notification shouldn't wait for it, and retrying a combined job would re-run the step that already worked. One job with two steps makes sense when the steps must run in order or share state, but then each step still needs its own idempotency check. For this checkout I'd use two.

3. Measure it. Alert when failed_jobs starts growing, and track queue depth and the age of the oldest waiting job per queue. If that age keeps climbing, workers are stuck or there aren't enough of them. php artisan queue:failed lists the failures, and on Redis queues Horizon puts wait times, throughput and failed jobs on one dashboard. Log the order_id in every job so you can trace a single order from start to finish.

Interview one-liner

"Sync is simpler but couples your latency and availability to email and other slow dependencies. Async improves p99 and resilience, as long as you own idempotency and dead-letter visibility."

Conclusion

Moving slow work to a queue fixes checkout latency almost for free, but it also means some jobs will run twice sooner or later. If a job can't cheaply detect that it already did its work, it isn't ready for production.

Try it yourself

In a Laravel project, create a ShouldQueue job that sends a fake email (just log it) and sets a sent_at column. Dispatch it, then dispatch the same job again for the same record, and check whether it logs twice.

Add the "already sent? return" check and repeat. For extra practice, set $tries = 3 and a backoff(), throw an exception on purpose, and watch the attempts end up in failed_jobs.

---

This is part of my daily System Design Drills series.