Distributed Tracing

Trace and span

A distributed trace represents one logical operation across many services. It is composed of spans, where each span represents a bounded unit of work such as an API gateway call, database query, queue wait, or model inference.

A trace has an identifier that is propagated between services. Parent-child relationships between spans preserve the path of the request, allowing operators to reconstruct not just what happened, but how long each stage took.

Why tracing matters

Suppose an API request takes 2 seconds. A server-level latency metric alone does not tell you whether the time was spent waiting in a queue, calling a database, communicating with another service, loading a model, or actually generating tokens. The trace breaks the 2 seconds into those components.

Tracing is also powerful for fan-out. If one request calls ten downstream services in parallel, the end-to-end latency is often determined by the slowest branch. A trace makes that relationship visible.

Context propagation

For tracing to work across service boundaries, trace context has to travel with the request. Standardized headers or instrumentation libraries propagate the trace identifier and parent span so downstream components can attach their work to the same trace.

Distributed tracing tools

OpenTelemetry is the usual instrumentation and telemetry layer. Its SDKs create spans in applications, and the OpenTelemetry Collector can receive, sample, enrich, batch, and export trace data to one or more backends. OpenTelemetry is not itself the place where teams normally search and retain traces.

Tool	Role
OpenTelemetry	Vendor-neutral APIs, SDKs, auto-instrumentation, and Collector pipeline
Jaeger	Open-source trace storage and investigation UI, often used for service graphs and trace search
Grafana Tempo	Trace backend designed to work with Grafana and object storage, with correlation to metrics and logs
Zipkin	Lightweight open-source tracing system with a simple collection and visualization model
Hosted platforms	Managed ingestion, retention, alerting, and cross-signal analysis from vendors such as Datadog, Honeycomb, New Relic, Elastic, or AWS X-Ray

A common production path is application instrumentation with OpenTelemetry, export through a Collector, and storage in Jaeger, Tempo, or a hosted backend. The important design choice is not the logo on the UI. It is whether the system preserves trace context, supports useful sampling, links traces to logs and metrics, and keeps sensitive payloads out of spans.

Sampling

Tracing every request can be expensive at high scale. Systems therefore use sampling strategies. Head-based sampling makes the decision early; tail-based sampling can keep traces that turn out to be errors or unusually slow after the complete trace is known.

Sampling should be designed so rare failures are not accidentally discarded. Many systems dynamically preserve errors and high-latency traces at a higher rate than routine successful traffic.

Security and data hygiene

Tracing metadata should help explain system behavior without becoming a second copy of sensitive application data. API keys, credentials, private prompts, and customer content should not be placed into spans casually.

Example

A request arrives at an edge gateway, which creates the root span. The gateway calls an auth service, then a scheduler, which waits 120 ms in a queue before calling a GPU worker. The GPU worker records model execution time and streams results back. The final trace shows exactly where the latency accumulated.

The core idea

Distributed tracing is request-level observability across service boundaries. It turns an opaque multi-service latency number into a causal timeline of work.