Message Queues

What a message queue does

A message queue lets one component produce work without requiring the consumer to be ready at the same instant. The producer places a message into a broker or queue, and a worker later consumes it. This decouples the timing and capacity of the two sides.

Queues are useful for email delivery, document processing, indexing, background jobs, media processing, and batch inference. A burst of traffic can be absorbed temporarily instead of forcing producers to wait synchronously for every consumer.

Delivery semantics

One of the first design questions is what delivery guarantee the system needs. At-most-once delivery favors avoiding duplicates but can lose work. At-least-once delivery retries failed processing but requires consumers to tolerate duplicates. Systems that appear to provide exactly-once behavior often achieve it with transactions or application-level deduplication rather than a universal guarantee.

Ordering is another concern. Some systems provide global ordering at high cost; others guarantee ordering only within partitions or keys. The application should define whether ordering is actually required.

Queues do not create capacity

A queue can absorb a burst, but it cannot make a slow consumer faster. If producers add 10,000 messages per second and workers complete only 4,000, backlog grows. The queue has converted immediate overload into waiting time.

This is why queue depth, oldest-message age, consumer throughput, and queue capacity are important operational signals. An unbounded queue can quietly turn an outage into hours of accumulated latency.

Failure handling

Real systems need policies for poisoned messages that repeatedly fail. A dead-letter queue can separate messages that exceed retry limits. Visibility timeouts, acknowledgements, leases, deduplication keys, and retention rules determine what happens when a worker crashes halfway through processing.

AI serving example

An inference scheduler can use a queue so it can batch similar requests, prioritize urgent work, and distribute jobs across available GPUs. If the queue becomes too deep, the system may autoscale more workers or reject low-priority requests.

For interactive workloads, however, the queue should have a clear latency budget. A user may prefer a fast overload response to a request that was accepted but will not run for several minutes.

The core idea

A message queue is a buffer and decoupling layer. It separates producers from consumers and smooths bursts, but the queue itself has capacity, latency, delivery semantics, and failure modes that must be engineered explicitly.