Throughput

What throughput measures

Throughput is the rate at which a system successfully completes useful work. The unit should describe what counts as work: requests per second, tokens per second, messages per second, transactions per second, or jobs per minute.

Throughput is different from latency. Latency asks how long one operation takes; throughput asks how much completed work the system can sustain over time. A system can have low latency for small individual requests but poor throughput because it cannot process many requests concurrently.

Finding the bottleneck

Throughput is limited by the system's current bottleneck. Depending on the workload, the limiting resource may be CPU, GPU compute, GPU memory, memory bandwidth, disk I/O, network bandwidth, database connections, or an external dependency. Once the bottleneck saturates, feeding the system more work tends to create queueing rather than more useful throughput.

This is why performance testing should measure the throughput-versus-latency curve instead of looking for one maximum requests-per-second number. Often, throughput continues to rise until a resource approaches saturation, after which latency starts increasing rapidly.

Batching and AI inference

Batching can increase throughput by amortizing fixed overhead and improving accelerator utilization. Processing eight requests together may be much more efficient than running eight tiny kernel launches separately. The trade-off is waiting: the system may need to hold the earliest request briefly while enough work accumulates to form a useful batch.

Dynamic batching attempts to find a balance by collecting requests for a small scheduling interval and dispatching a batch once it reaches a target size or deadline. Larger batches can increase throughput while worsening tail latency and memory pressure.

Throughput is not acceptance

A service that accepts 10,000 requests per second but drops 20% of them does not have the same useful throughput as one that successfully completes all 10,000. In production, throughput should therefore be read alongside error rate, queue depth, latency, and saturation.

The same distinction appears in asynchronous systems. A queue may ingest messages much faster than workers process them. Input rate is not throughput; completed work is.

Example

A model server processes 40,000 tokens per second across four GPUs. Increasing concurrency to the point where queue depth grows does not necessarily increase useful throughput. If the GPUs were already saturated, additional concurrency may simply increase waiting time. The productive capacity was already near the bottleneck.

The core idea

Throughput answers how much useful work a system can finish per unit time. Capacity planning becomes meaningful when throughput is considered together with latency and the resources that constrain it.