The question metrics cannot answer
Metrics tell you checkout latency has doubled. Logs from each service tell you what each one did. Neither tells you which of the nine services involved in a checkout request is responsible, or whether the delay is in the service itself, its database query, or a retry storm against a downstream dependency.
Distributed tracing exists for exactly this. It reconstructs one request across service boundaries. We add it to the monitoring and observability service once metrics and logs are healthy, because tracing multiplies their value rather than replacing them. For definitions, see our entries on distributed tracing and OpenTelemetry.
What good looks like
- Standard instrumentation. OpenTelemetry SDKs and OTLP, so you are not locked to a vendor.
- Context propagates everywhere. HTTP, gRPC and message queues carry the trace context.
- A Collector layer for batching, sampling and routing.
- Sampling that keeps the interesting traces.
- Signals linked. Click from a latency spike to an example trace, from a span to its logs.
- Meaningful attributes. Route, customer tier, deployment version, never secrets or personal data.
Architecture
A typical Kubernetes layout has applications sending OTLP to a node-local or sidecar Collector, which forwards to a gateway Collector that applies sampling and exports to Tempo:
receivers:
otlp:
protocols:
grpc: {}
http: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
batch: {}
k8sattributes: {}
tail_sampling:
decision_wait: 10s
policies:
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow
type: latency
latency: { threshold_ms: 1000 }
- name: baseline
type: probabilistic
probabilistic: { sampling_percentage: 5 }
exporters:
otlp/tempo:
endpoint: tempo.monitoring:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, tail_sampling, batch]
exporters: [otlp/tempo]
The tail_sampling processor and its policies belong to the Collector contrib distribution. The 5 percent baseline and one second threshold here are placeholders to show the mechanism; real values come from your traffic and latency objectives. TLS to Tempo should be enabled in production rather than left insecure as in this short example.
Sampling: the decision that controls cost
| Approach | How it works | Trade-off |
|---|---|---|
| Head sampling | Decide at the start of a trace, e.g. keep 10% | Cheap and simple, but may discard the error you needed |
| Tail sampling | Collect the whole trace, then decide | Keeps errors and slow requests, needs memory and routing by trace ID |
| Always on for low traffic | Keep everything | Fine for small systems, expensive at scale |
Tail sampling needs all spans of a trace to reach the same Collector instance, which means a load-balancing exporter tier in front of the sampling tier. That extra layer is real operational weight; at small scale, head sampling plus keeping all errors from the application SDK is often enough.
Instrumenting services
Start with auto-instrumentation. On Kubernetes, the OpenTelemetry Operator can inject agents into pods through annotations for Java, Python, Node.js and .NET:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "monitoring/default-instrumentation"
Then add manual spans around business operations, such as apply_discount or charge_card, and set attributes like order.tier. For Go, use the library instrumentation wrappers for net/http, gRPC and database drivers. Verify propagation across async boundaries: a message published to a queue loses its trace unless the context is injected into message headers.
Linking traces, logs and metrics
Three connections make tracing useful day to day:
- Logs carry the trace ID. A logging hook adds
trace_idto each line, and Loki derived fields turn it into a link. See the Loki page. - Metrics carry exemplars. A histogram bucket can hold a sample trace ID that Grafana shows as a dot on the latency graph.
- Tempo links back to logs by matching labels, as configured in data source provisioning.
Tempo or Jaeger
Tempo’s strength is cost: it keeps traces in object storage and avoids a heavy index, so you can retain more. It is query-light, so finding traces by attribute relies on TraceQL and metrics generated from spans. Jaeger offers rich native search and a dedicated UI, but running its storage backend (often Elasticsearch, OpenSearch or Cassandra) is a real task. In a Grafana, Loki and Prometheus stack, Tempo fits more naturally. Where an organisation already operates Jaeger well, we will not insist on migrating.
If you run a mesh such as Istio, proxies can emit spans for network hops, but the mesh cannot see inside your application, so application spans are still needed.
When not to do this
If you have a monolith, or two services and a database, tracing adds a pipeline to run for little gain; good logs and metrics will take you further. Tracing becomes high value when a request crosses several services owned by different people, when latency problems are hard to localise, or when async flows hide dependencies.
Failure modes
- Broken context propagation gives you orphaned single-span traces.
- Unbounded attributes such as full SQL with literals or user emails; both inflate cost and leak data.
- Collector overload without a memory limiter, dropping data silently.
- Sampling set once and forgotten while traffic grows tenfold.
- Tracing treated as a project rather than something new services get by default.
Starting small with tracing
Teams often delay tracing because instrumenting everything sounds large. It does not have to be. We usually begin with the one request path that matters most to the business, for example checkout or sign-up, and instrument only the services on it. Auto-instrumentation agents give useful spans for HTTP, database and messaging calls without code changes, and manual spans are added only where the automatic ones leave a gap.
Once that path is traced and the team has used it to solve a real incident, the value is obvious and the appetite to extend coverage grows naturally. Starting narrow also keeps the sampling and storage decisions simple while you learn what your traffic actually looks like, rather than guessing at capacity for a whole estate before a single trace has been read.
Delivery and handover
We deliver Collector and Tempo or Jaeger configuration in Git, instrumentation guidance per language, sampling policy with its rationale, Grafana links, and a validated trace of at least one critical journey from edge to database. Your developers keep ownership of manual spans. For metric instrumentation alongside this, see application metrics instrumentation.
How to tell it is working
Measure the share of requests with complete traces across all hops, how often incident write-ups cite a trace, time to locate the failing service, and span drop rates in the Collector. To plan tracing for your architecture, contact us.