Background jobs and queues without the 3am pages

The first background job everyone writes looks fine. Take a task off the request path, push it to a queue, process it later. The problem is that background jobs and queues fail in ways synchronous code does not, and they fail when nobody is watching. A job that runs twice, runs out of order, or silently disappears will not show up in a unit test. It shows up at 3am when a customer notices they got charged twice. Most of the work in queue design is making those failure modes survivable instead of pretend they will not happen.

Assume every job runs at least twice

This is the single rule that prevents the most pain. Almost every queue worth using guarantees at-least-once delivery, not exactly-once. A worker can pick up a job, do the work, and crash before it acknowledges completion. The queue, having heard no ack, hands the same job to another worker. Now your email goes out twice, or your payment runs twice.

The answer is idempotency, which means a job produces the same result whether it runs once or five times. Sometimes that is a unique key on the database so a duplicate insert is a no-op. Sometimes it is checking whether the work was already done before doing it. It is more effort up front, and it is the difference between a worker crash being a non-event and being an incident. I design every job as if it will be delivered twice, because eventually it will be.

Retries need backoff and a ceiling

When a job fails, retrying is usually right. A downstream API blipped, a database connection dropped, the next attempt succeeds. But naive retries cause two specific disasters. The first is the retry storm: a downstream service is struggling, every job fails, and your workers immediately hammer it again, keeping it down. The second is the poison message: a job that can never succeed, retrying forever, burning a worker slot until you notice.

What I reach for on every queue:

• Exponential backoff with jitter, so retries spread out instead of stampeding the failing service all at once
• A hard retry ceiling, after which the job stops retrying and goes somewhere visible
• A dead letter queue for those exhausted jobs, so they are parked and inspectable instead of lost

A dead letter queue is not optional infrastructure. It is where you find out what is actually breaking, and a job vanishing into the void is far worse than a job sitting in a holding pen with its error attached.

Ordering is a promise you probably do not have

People assume jobs run in the order they were enqueued. With multiple workers pulling concurrently, they do not. Job B can finish before job A even if A was queued first. If your logic depends on order, like apply discount then charge card, concurrent workers will eventually run them backwards and you will get a bug that only reproduces under load.

You have a few options. Some queues offer ordered partitions, where jobs sharing a key process in sequence. Or you make the operations commutative so order stops mattering. Or you keep an explicit state machine that rejects an out-of-order transition rather than executing it. What does not work is assuming order you were never promised. That assumption is invisible in code review and obvious in production.

Visibility is the whole game

The reason queues page you at 3am is almost always that you could not see the problem building during the day. A queue silently filling up because workers are too slow is a time bomb. By the time latency is user-visible, you have a deep backlog that takes hours to drain even after you fix the cause.

I instrument three things from the start: queue depth over time, job processing latency, and the failure rate. Queue depth is the leading indicator. If it is climbing and not coming back down, you are falling behind, and you want to know that at 2pm when you can calmly scale workers, not at 3am when the backlog is already enormous. Alert on the trend, not just the catastrophe.

Keep jobs small and singular

A job that does five things is a job that can fail four steps in, leaving you in a half-done state that is miserable to recover. I keep jobs as small as the work allows and have them do one thing. If a flow needs several steps, I chain small jobs or use a workflow tool that tracks progress, so a failure resumes from where it stopped instead of redoing everything and re-triggering side effects.

The goal across all of this is boring queues. A good background job system is one you forget exists because it quietly handles its own failures, deduplicates its own retries, and tells you when it is falling behind long before it falls over. That boredom is earned. It comes from deciding, before you ship, that every job will run twice, fail sometimes, and run out of order, and building so none of that wakes you up.