Rate Limiting

What rate limiting actually does

Rate limiting is an admission-control mechanism. It defines how much work a caller is allowed to start over some period instead of allowing every request to compete for resources immediately. The limit may be expressed as requests per second, requests per minute, tokens per minute, concurrent requests, bytes per second, or another unit that maps to the scarce resource.

The important idea is that a service has finite capacity. Rate limiting turns that capacity into an explicit contract at the edge of the system so that overload is handled deliberately rather than by letting queues, memory usage, database connections, or worker pools grow until something fails.

Common algorithms

A fixed-window limiter counts requests in discrete intervals, such as one minute. It is easy to implement, but it can allow a burst at the boundary between two windows. A sliding-window design reduces that boundary effect by considering a moving interval.

A token bucket starts with a number of tokens and replenishes them at a configured rate. Each request consumes one or more tokens. This lets a client make a controlled burst up to the bucket capacity while preserving a long-term average rate. A leaky bucket instead behaves more like a queue with a controlled output rate, smoothing bursts more aggressively.

There is also a distinction between rate and concurrency. A caller can remain below a request-per-second limit while keeping thousands of expensive requests in flight. Concurrency limits therefore complement rate limits by bounding work that has already entered the system.

Where the limit is enforced

Rate limits can be enforced at an API gateway, reverse proxy, application server, service mesh, or directly inside an expensive subsystem. A public API may limit by API key, user, tenant, IP address, endpoint, model, or combinations of these dimensions. A multi-tenant platform often needs separate quotas so that one customer cannot consume the entire shared capacity pool.

In a distributed deployment, independent per-instance counters can accidentally multiply the effective limit. A ten-instance service with a local limit of 100 requests per second per instance can admit roughly 1,000 requests per second unless the architecture intentionally wants that behavior. Shared state, partitioned limits, or approximate distributed algorithms are used when a global quota is required.

What happens when the limit is exceeded

HTTP APIs commonly respond with 429 Too Many Requests. A useful implementation can include a Retry-After hint or equivalent metadata so a client knows that the rejection is temporary. The system should also record throttled requests separately from application errors; otherwise operators may misread deliberate load shedding as a software failure.

Rate limiting is not overload protection by itself

Rate limiting protects the boundary, but it does not automatically protect every downstream resource. One request may be cheap while another triggers a long-running database query or a large model generation. Production systems therefore combine rate limits with concurrency limits, timeouts, backpressure, queue bounds, and autoscaling.

Example: an AI inference API

Suppose a model provider allows a tenant 60 requests per minute and 8 concurrent generations. The request-rate limit protects the tenant's quota while the concurrency limit protects GPU memory and worker slots. When the tenant exceeds the request budget, the gateway returns 429. When all eight generation slots are occupied, additional requests wait briefly or are rejected according to the service's overload policy.

The core idea

Rate limiting makes resource consumption explicit and bounded. It is most effective when the limit is tied to the real bottleneck and combined with the rest of the system's capacity controls.