When logs and metrics stop being enough
Metrics tell you latency rose. Logs tell you something in one service failed. Neither tells you that the checkout call spent most of its time waiting on a pricing service that was itself waiting on a slow database query. In a monolith you could read a stack trace. In a system of ten or fifty services, the path of a request is the missing piece, and distributed tracing provides it.
OpenTelemetry (OTel) is now the common standard for producing that data. It is part of our wider logging, ELK, Loki and OpenTelemetry service, and it feeds the storage and correlation work described on the trace storage and correlation pages. For background, see our glossary on OpenTelemetry.
What good looks like
- A request that crosses services appears as a single trace, with no gaps at service boundaries.
- Spans exist for inbound requests, outbound calls, database queries, queue publishes and consumers, plus a few named business operations.
- Spans carry consistent attributes: service name, version, environment, route, status.
- Telemetry flows through a Collector, not directly from each pod to the backend.
- Personal data is kept out of span attributes by design.
How it works in practice
Architecture
Applications run an OTel SDK and send telemetry over OTLP to a Collector. In Kubernetes we typically run a Collector as a DaemonSet or sidecar for local receipt and a small gateway deployment for sampling and export. The Collector exports to your trace backend. If you use the OpenTelemetry Operator, an Instrumentation resource can inject language agents into pods by annotation, which speeds rollout for Java, Python and Node.
# Collector pipeline fragment (illustrative)
receivers:
otlp:
protocols:
grpc: {}
http: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 20
k8sattributes: {}
batch: {}
exporters:
otlp/tempo:
endpoint: tempo-distributor.observability.svc:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, batch]
exporters: [otlp/tempo]
Place memory_limiter first and batch last in the processor list, as the Collector documentation recommends. TLS should be enabled outside a trusted cluster network, and the insecure flag shown is only for in-cluster traffic you have deliberately accepted.
Per-language approach
Java. The OpenTelemetry Java agent attaches with -javaagent and instruments common frameworks and drivers without code changes. Configuration is by environment variables such as OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT.
Python and Node. Auto-instrumentation packages wrap popular libraries such as Flask, Django, FastAPI, Express and database drivers. In Node the SDK must be initialised before application imports so libraries are patched. Both support manual spans.
Go. There is no agent in the same sense. You add instrumentation libraries such as otelhttp for servers and clients, otelgrpc and database wrappers, and create spans manually where needed.
tracer := otel.Tracer("checkout")
ctx, span := tracer.Start(ctx, "authorise-payment")
defer span.End()
span.SetAttributes(attribute.String("payment.provider", "stripe"))
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "authorisation failed")
}
Propagation is the hard part
Traces split when context does not cross a boundary. Typical culprits are message queues, background jobs, custom HTTP clients and API gateways that strip headers. The W3C traceparent header is the default format, and we test every hop: HTTP, gRPC, Kafka or SQS, scheduled jobs. A service mesh can add spans for network hops, but it cannot see inside your code, so it supplements rather than replaces SDK instrumentation.
Sampling
Three options exist. Always-on is simple and costly. Head sampling decides at the start of a trace, for example keep a fixed fraction, and is cheap but may discard the rare failure. Tail sampling in the Collector decides after the trace completes, allowing rules such as keep all errors, keep slow traces and sample the rest. Tail sampling needs all spans of a trace to reach the same Collector instance, so it needs a load-balancing exporter tier and more memory. We start with a modest head ratio, add tail rules once the basics are stable, and write the policy down.
Decisions and trade-offs
Auto or manual spans. Auto-instrumentation gives breadth quickly but produces generic span names. Manual spans give meaning. We use both, with manual spans limited to important operations rather than every function.
Vendor SDK or OpenTelemetry. OTel avoids lock-in and works with open-source or commercial backends. Where a vendor agent offers features you need and you accept the lock-in, we will say so, but we default to OTel.
Traces for every service. Instrument the critical request path first. Low-value batch jobs and internal tools can wait.
When not to do it yet. A monolith with one database often gets more from good logs and metrics. Tracing pays off as service count and call depth grow.
Common failure modes
- Broken parent context in async code, thread pools and queues. We test across those boundaries.
- High-cardinality span names such as raw URLs with IDs. We use route templates.
- Sensitive attributes like emails or tokens in spans. Redaction rules in the Collector handle this, as described in log and telemetry scrubbing.
- Collector out of memory under bursts. Limiters and horizontal scaling prevent it.
- Mixed SDK versions across services causing odd attribute names. We pin and track versions centrally.
Rolling out tracing without disrupting teams
The fastest way to stall a tracing programme is to ask every team to change code at once. We start with one critical user journey, typically the request path that generates the most support tickets, and instrument only the services on it. That gives a visible win and exposes propagation gaps early, such as a message queue that drops headers.
Once that path produces complete traces, we publish a short onboarding guide, then expand service by service in order of business importance. Libraries that support automatic instrumentation get it first because it needs the least code. Manual spans are added only where the automatic ones miss meaningful work, such as a pricing calculation or a batch step.
Along the way we check that adding instrumentation does not noticeably change latency, using a before and after comparison on the pilot path.
Delivery and handover
You get instrumented services merged through your normal pull request process, Collector manifests in your repository, a conventions document for span names and attributes, and a debugging session using a real slow request. Instrumentation for CI/CD and deployment markers can be linked to your CI/CD pipelines.
How to tell it is working
Traces for key journeys are complete end to end, p95 latency questions are answered by looking at a trace rather than guessing, the share of traces with orphan spans is low, and Collector queues and memory stay healthy. If you want a pilot on your busiest path, book a consultation.