P99 / Tail Latency

What P99 means

P99 is the 99th percentile of a latency distribution. If a service reports a P99 of 800 ms, roughly 99% of observed requests completed in 800 ms or less and about 1% were slower. It is therefore a measure of the slow tail rather than the typical request.

Other useful percentiles are P50 (median), P90, P95, and P99.9. The farther into the tail you go, the more sensitive the measurement becomes to rare events such as queueing, garbage collection, cold starts, packet loss, retries, and overloaded dependencies.

Why the average can lie

A service can have a 100 ms average while its P99 is 4 seconds. That means the majority of requests look excellent while a significant minority are painfully slow. Interactive systems often care about the tail because users experience individual requests, not the arithmetic average.

Fan-out amplifies tails

In a distributed system, one user request may depend on many downstream operations. Even if each dependency is usually fast, the chance that at least one child lands in its slow tail grows as the number of dependencies increases. The slowest branch can determine the end-to-end latency. This is one reason distributed tracing and careful deadline propagation matter.

P99 is not an upper bound

P99 does not mean "requests will never exceed this value." One percent of observations are beyond it by definition. It is a percentile, not a hard safety limit. Percentiles also should not be averaged casually across servers or windows; histogram-based or mergeable distribution methods are safer for aggregation.

AI serving needs multiple latency measures

For generative models, total completion time is only one dimension. Time to first byte, time to first token, inter-token delay, and time to final token can tell very different stories. A model can have an excellent time-to-first-token while generating the rest too slowly.

Example

An inference service reports P50 time-to-first-token of 180 ms and P99 of 1.6 seconds. The average alone might suggest the service is healthy. A trace of the slow tail could reveal that the P99 requests are spending most of their time waiting in a full GPU queue rather than in model computation.

The core idea

Tail latency measures the unlucky requests that sit near the slow end of the distribution. P99 is useful because it exposes user-visible performance problems that an average can hide.