Autoscaling

What autoscaling does

Autoscaling is a feedback loop that changes the amount of serving capacity as workload demand changes. A scaler observes one or more signals—request rate, CPU utilization, memory pressure, queue depth, active requests, accelerator utilization, or a custom saturation metric—and adjusts capacity accordingly.

The goal is not simply to make a system bigger when traffic rises. It is to keep enough spare capacity to absorb demand without paying for a permanently oversized fleet.

Horizontal vs vertical scaling

Horizontal scaling changes the number of instances or workers. It is often the natural choice for stateless services because requests can be distributed among replicas. Vertical scaling changes the resources available to an instance, such as CPU, memory, or GPU capacity.

Horizontal scaling is constrained by startup time and the ability to distribute work. Vertical scaling can simplify some workloads but eventually hits a machine-size ceiling and can make failures larger because more capacity is concentrated in one instance.

Choosing the right signal

CPU is a poor autoscaling signal when the real bottleneck is queue depth, GPU memory, or active model generations. Queue depth can be a strong signal for asynchronous processing because it directly measures work waiting for consumers. Concurrency is useful when a service has a hard limit on the number of operations that can be active before latency rises sharply.

Latency can be used as an outcome signal, but scaling directly from noisy tail latency can create unstable feedback loops. The best metric is usually close to the resource's actual saturation point.

Why scaling is not instantaneous

A new VM or Pod may take time to schedule, download an image, initialize a model, allocate GPU memory, and warm caches. That means autoscaling reacts after demand has already changed. Systems with bursty traffic may need minimum warm capacity, prewarming, predictive scaling, or queue buffering to bridge this gap.

Scale-down is deliberately slower in many systems. Removing capacity too aggressively can cause oscillation: scale out, scale in, scale out again. Stabilization windows and cooldown periods smooth the control loop.

Example: AI inference

Imagine a model server that can sustain 30 concurrent generations per GPU. When active concurrency stays above 24 for several evaluation periods, the autoscaler adds another worker. When concurrency drops for a sustained interval, it removes one. The threshold is intentionally below hard saturation so the new worker can become ready before the existing fleet hits its limit.

The core idea

Autoscaling is capacity control through feedback. It works when the observed signal reflects true saturation and when the system accounts for the delay between requesting more capacity and that capacity becoming usable.