Observability

What observability means

Observability is the ability to infer what a running system is doing from the evidence it emits. The three common telemetry signals are metrics, logs, and traces. Metrics summarize behavior numerically, logs capture discrete events and context, and traces show how one request moved through the system.

The goal is not to produce the maximum amount of telemetry. The goal is to answer useful questions when behavior is unfamiliar: What changed? Which requests are affected? Which tenant, region, model, or dependency is responsible? Where is time being spent?

Metrics

Metrics reveal rates, errors, saturation, and distributions. Useful production signals can include request rate, error rate, queue depth, active concurrency, CPU/GPU utilization, cache hit rate, model load time, token counts, time to first token, and tail latency.

A single average often hides important behavior. For example, a service can have excellent average latency while a small fraction of requests takes several seconds. Percentiles and histograms expose that tail.

Logs

Structured logs make discrete events searchable and machine-readable. Request IDs, tenant IDs, endpoint names, error classes, and model identifiers can connect logs to higher-level telemetry. Care is required with sensitive data: raw prompts, tokens, credentials, and personal data should not be emitted casually.

Traces

Distributed tracing follows an individual request across service boundaries. It shows whether latency came from authentication, an API gateway, a queue, a database, a model server, or a downstream dependency. Traces become especially valuable in fan-out architectures where one slow child operation can dominate the final response.

Observability for AI systems

AI serving introduces metrics that ordinary web services may not need: tokens per second, queue time, batch size, time to first token, inter-token delay, accelerator memory pressure, model load duration, and cancellation rate. These signals distinguish compute saturation from queueing and distinguish model execution time from transport overhead.

Good observability is actionable

A dashboard is useful when it helps an engineer decide what to do next. An alert is useful when it identifies a condition that deserves intervention. High-cardinality labels can be powerful for slicing by tenant or model, but they also increase storage and query cost.

The core idea

Observability is the system's ability to explain its own behavior through telemetry. Metrics tell you what changed, logs provide event context, and traces explain where a request spent its time.