The Queue Is the Product

A queue is easy to add and hard to own. The first version is always the same: a producer drops a message, a worker picks it up, everything is fast, and the demo works. Then a consumer deploys during a burst, a downstream API starts returning 503s, and someone asks what happened to the 4,000 invoices that were supposed to go out at 9 a.m.

The answer is usually in the queue. Not in the happy path. The queue is where the product actually lives.

The demo path and the failure path

The demo path has one message, one consumer, and one successful side effect. The failure path has duplicates, ordering problems, partial writes, poison messages, and a worker that dies halfway through a job. Most of the design work is choosing what happens on that second path.

The shape I use for most systems is deliberately boring:

• The producer writes the message and commits its database transaction.
• A queue holds the message with a visibility timeout.
• One or more consumers process it, with a retry budget.
• A dead-letter queue catches what the budget cannot fix.
• An operator surface makes the dead letters visible.

The interesting decisions are inside each step.

Idempotency is the contract

At-least-once delivery is not a bug you can configure away. It is the default semantics of most queues, including SQS. A message can be delivered twice because a consumer died after doing the work but before deleting the message, because a network partition hid the acknowledgement, or because a retry fired while the first attempt was still running.

That means every consumer needs an idempotency key and a durable place to record it. The key should be derived from the event, not from the delivery attempt:

type InvoiceEvent = {
  eventId: string;
  invoiceId: string;
  attempt: number;
};
 
async function handle(event: InvoiceEvent) {
  const key = `invoice:${event.invoiceId}`;
 
  if (await processed.has(key)) return;
 
  await processed.once(key, async () => {
    const invoice = await invoices.markIssued(event.invoiceId);
    await mailer.sendIssued(invoice);
  });
}

The important part is once. It needs to be atomic with the side effect, or at least recoverable if the process dies between the two. In Postgres, that usually means a unique constraint on the key and a transaction that writes the business row and the processed marker together. In a document store, it means a conditional write. In a file system, it means admitting you have a database and using one.

Visibility timeout is a design decision

The visibility timeout is how long the queue waits before assuming a consumer died. Set it too low and a slow job gets processed twice. Set it too high and a crashed job sits invisible for an hour while the dashboard stays green.

I start with the p99 processing time and add a margin, then I make the consumer extend the timeout for long jobs. The exact number matters less than the rule: the timeout must be longer than the slowest successful attempt, and shorter than the time a human is willing to wait for a retry.

There is also a queue-level consequence. A message that keeps failing should not block the messages behind it. That is what separate queues or a dead-letter path are for. One queue per job shape is usually enough; one queue for “everything asynchronous” is how you get a poison message holding up payroll.

Retries need a budget

Retries are not free. Each attempt consumes a worker, a database connection, a third-party rate limit, and the attention of anyone watching the error dashboard. A retry policy should have an explicit budget:

The rule I use is simple: retry only failures that are plausibly temporary. A validation error will not become valid on the third attempt. A missing row might appear if a previous message is still in flight, so it gets a short delay and a small budget. A 429 gets jitter, because retrying in lockstep is a polite way to stay rate-limited.

The dead-letter queue is a product surface

The DLQ is where the honest work goes. A message that failed every attempt is not an infrastructure detail; it is a customer waiting for an invoice, a shipment that did not move, or a password reset that never arrived. If nobody owns the DLQ, the product has a silent failure mode.

I want three things on the operator surface:

1. The message and its history. What was it, how many attempts, what changed between them.
2. A safe action. Replay, discard, or fix-and-replay, with an audit trail.
3. A count that is allowed to be zero. A DLQ dashboard that is always empty is either healthy or unobserved. The difference is whether someone checks it.

The best pattern I have used is a small “failed jobs” page in the admin surface, backed by the same queue metadata. It removes the need to open the AWS console during an incident, which is exactly when nobody should be learning IAM.

The checklist I use

Before a queue goes live, I want answers to these questions:

• What is the idempotency key, and where is it stored?
• What is the visibility timeout, and what happens when it expires?
• How many retries, with what backoff and jitter?
• Which failures are not retried, and who sees them?
• Where does the DLQ live, and who replays from it?
• What does the user see while the message is in flight?
• What happens if the consumer deploys mid-burst?
• How do we know the queue is draining, not just receiving?

None of these are exotic. They are the difference between a queue that moves work and a queue that hides it. The happy path is a demo. The queue is the product.¹