The investigation that takes forty minutes instead of four
A common incident shape: an alert says checkout latency is high. Someone opens a trace and sees a slow call to the inventory service. They want the logs for that call. They copy the timestamp, pick the right service in the log tool, widen the time window, and scroll. Nothing identifies the request, so they guess between several lookalike lines. Half an hour goes on a join a computer could do instantly.
Correlation removes that gap. It is a small piece of engineering with a large effect on time to diagnose, and it connects the work described on our OpenTelemetry tracing and structured logging pages. It sits inside our wider logging, ELK, Loki and OpenTelemetry service.
What good looks like
- Every log line written during a request contains the same
trace_id, and thespan_idof the active span. - From a trace in Grafana you click a span and land in a log query scoped to that trace and time window.
- From an error log line you click the trace ID and open the full trace.
- Metrics panels can link onward to example traces where exemplars are enabled.
- A scripted test fails the build when correlation stops working.
How it works in practice
Step 1: put the IDs in the log line
The logging library must know about the active span. With OpenTelemetry, the current span context is stored in the request context. A logging hook reads it and adds fields. In Go with log/slog, you can wrap a handler:
type traceHandler struct{ slog.Handler }
func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
r.AddAttrs(
slog.String("trace_id", sc.TraceID().String()),
slog.String("span_id", sc.SpanID().String()),
)
}
return h.Handler.Handle(ctx, r)
}
Callers must pass the request context to the logger (logger.ErrorContext(ctx, ...)), which is a convention we put in the guidelines. In Python, the OpenTelemetry logging instrumentation can add trace and span IDs to the logging record, and structlog processors can do the same. In Java, the OpenTelemetry agent adds trace_id and span_id to the MDC for common logging frameworks. In Node, pino mixins or the OTel instrumentation for pino can inject them. Always verify with a real request rather than assuming.
Step 2: make the field searchable, safely
For Loki, keep trace_id in the log line as JSON and use a filter such as | json | trace_id="...", or store it as structured metadata in versions that support it. Never promote it to a label. For Elasticsearch, map it as a keyword field. In both cases the name should match across services, which is another reason for a shared standard.
Step 3: Grafana data source links
From logs to traces, a derived field on the Loki data source extracts the trace ID with a regular expression or JSON field and points at the Tempo data source:
# Grafana provisioning fragment for the Loki data source
apiVersion: 1
datasources:
- name: Loki
type: loki
url: http://loki-gateway.logging.svc
jsonData:
derivedFields:
- name: TraceID
matcherType: label
matcherRegex: trace_id
url: "$${__value.raw}"
datasourceUid: tempo
urlDisplayLabel: "Open trace"
The exact fields vary with the Grafana version, so we validate against the release you run, including whether to match on a label or on the log body. From traces to logs, the Tempo data source has a trace-to-logs setting that names the Loki data source, the tags to map (for example service.name to the app label), and a time shift so the query window covers the span. With Jaeger the same idea applies using its data source settings.
Step 4: sampling and edge cases
If a trace was sampled out, the link from a log line will fail. We reduce this by using tail sampling to retain errors and slow requests, as described in the OpenTelemetry page. We also look at the awkward paths: asynchronous jobs that continue a trace through a queue, background workers that should link to the originating request, and batch jobs with no incoming trace at all, where the job creates its own root span.
Decisions and trade-offs
Trace ID in every line or only errors. Every line is better for debugging. The cost is a few dozen bytes per line, which compresses well. Errors-only is cheaper but loses context, so we recommend all lines within request scope.
OTel log bridge or classic logger plus injection. The OTel logs SDK can send logs through OTLP with context attached, which is attractive for greenfield work. Many teams keep stdout JSON and a collector because it is simple and robust. We help choose and avoid running both paths for the same logs.
Request ID versus trace ID. Existing request IDs can remain, and the trace ID can be reused as the request ID so that support teams and engineers share one identifier.
When not to rush. If tracing covers only a few services, partial correlation can confuse people. We finish propagation first, then enable links broadly.
Common failure modes
- Context not passed, so logs inside goroutines, threads or callbacks lack IDs.
- Format mismatch, where traces use 32 hex characters and logs use a vendor format.
- High-cardinality labels from trace IDs.
- Clock skew making time-window queries miss lines. Time sync is checked on nodes.
- Links configured in the UI only, then lost on rebuild. We provision data sources as code via GitOps.
Delivery and handover
You receive the logger changes in your repositories, provisioned data source configuration, a CI test that sends one request and asserts the IDs line up, and a short guide with screenshots of the navigation paths. We run a debugging exercise with your on-call engineers using a past incident.
How to tell it is working
Engineers go from alert to the relevant log lines in a couple of clicks, the fraction of log lines inside requests that carry a trace ID approaches the expected share, and incident reviews mention correlation as a help rather than a gap. To get this working across your stack, book a consultation.