Backpressure

What backpressure means

Backpressure appears when a producer can generate work faster than a consumer can safely process it. Instead of allowing an unbounded buffer to absorb the difference, the downstream component communicates that its capacity is being exhausted and the upstream component slows, pauses, or rejects new work.

Without backpressure, excess work accumulates somewhere: memory grows, queues get longer, threads block, connections remain open, or requests wait until the system eventually fails.

Common forms

A bounded queue is a simple form of backpressure: once it reaches capacity, the producer must wait, drop work, or reject the item. Streaming protocols can expose demand or flow-control windows. Synchronous APIs can propagate pressure through blocking, cancellation, 429 responses, or overload errors.

TCP itself has flow-control mechanisms that prevent a sender from overwhelming a receiver's buffers, which illustrates the general principle: the consumer's ability to accept data limits the producer's ability to send it.

Backpressure vs rate limiting

Rate limiting is usually a policy-defined budget: a caller is allowed a certain amount of work regardless of instantaneous downstream pressure. Backpressure is state-dependent flow control: the consumer is saying that its current capacity is insufficient, so upstream production must change.

A production system can use both. Rate limiting protects shared capacity from excessive callers; backpressure prevents active pipelines from flooding a bottleneck that is already saturated.

Bounded queues and controlled degradation

A queue should usually have a meaningful upper bound. Once the bound is reached, the system needs a policy: reject low-priority work, shed load, sample events, slow producers, or return a visible overload response. Infinite buffering often makes an outage worse by turning failure into unbounded latency.

Example: streaming model output

Suppose a client reads generated tokens more slowly than the model server produces them. The streaming layer cannot safely buffer unlimited tokens in memory. It therefore applies flow control or pauses the producer until downstream buffers have room again.

The core idea

Backpressure is how a pipeline says "you are producing faster than I can consume". It keeps overload visible and bounded instead of hiding it inside ever-growing buffers.