Distributed Tracing

Distributed tracing follows a single request as it passes through several services, recording where time is spent and where failures occur. It complements metrics, which show that something is slow, and logs, which show what each service did.

How it works

A trace represents one request end to end. It is made of spans, each a unit of work with a name, start and end time, a parent span ID, attributes, optional events and a status. A root span has no parent, and child spans form a tree that is usually drawn as a waterfall.

For spans from different services to join one trace, the span context (trace ID, span ID and flags) must travel with the request. This is context propagation. The W3C Trace Context standard defines the traceparent HTTP header for this, with four hexadecimal fields: version, trace ID, parent span ID and trace flags. Each service reads the header, creates its own spans under the incoming parent, and passes an updated header downstream.

A practical example

A checkout request carries this header into the payment service:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The payment service starts a child span with the same trace ID and sends the new span ID onward. Open the trace in a backend such as Jaeger or Grafana Tempo and the waterfall shows, say, a slow database call inside the payment span. Instrumenting with OpenTelemetry keeps the code vendor-neutral.

Common mistakes

• Breaking propagation. A message queue, async job or proxy that drops the headers splits one trace into several.
• Sampling carelessly. Head sampling decides early, so it cannot guarantee that traces containing errors are kept. Tail sampling decides after the trace is mostly complete, but needs stateful infrastructure and tuning.
• Adding high-cardinality or personal data as attributes without review.
• Tracing without correlating logs, which leaves engineers switching tools to find the matching log lines. See log aggregation.

When to use it and when not to

Tracing pays off in microservice systems where a request crosses many processes and latency is hard to attribute. It is also useful for finding repeated calls and serial work that could run in parallel. For a single application with one process, metrics, logs and a profiler are often enough, and tracing adds overhead and operational cost for little gain. Start with the services on your most important user journeys. Our work in this area is described under OpenTelemetry distributed tracing.

See also

• OpenTelemetry
• Log Aggregation
• Prometheus
• Service mesh
• Grafana Tempo and Jaeger trace storage