Storing and Searching Traces with Grafana Tempo or Jaeger

Producing traces is half the job

Once services emit spans, they have to land somewhere that is reliable, searchable and affordable. This is where tracing projects often stall. The instrumentation works in a demo, then volume grows, the backend struggles, someone turns sampling down to protect it, and the traces that explain incidents are the ones missing.

Trace storage is therefore a capacity and cost design problem as much as a tool choice. It follows the instrumentation described on our OpenTelemetry tracing page and sits inside the wider logging, ELK, Loki and OpenTelemetry service.

What good looks like

• A trace found by ID in seconds, and searchable by service, status, duration and key attributes.
• Retention short enough to be affordable and long enough to cover real investigations.
• Ingestion survives bursts through queueing and horizontal scale.
• The query interface sits behind single sign-on and respects team boundaries.
• Traces open from dashboards, alerts and log lines, not only from a separate tool.

Tempo and Jaeger compared

Both can ingest OTLP. Neither requires different application instrumentation, which is an advantage of using OpenTelemetry. Many teams on Grafana choose Tempo because trace-to-log links in Grafana are simplest when everything shares one interface.

How it works in practice

Tempo

Tempo has a distributor, ingesters, a compactor, queriers and a query frontend, usually deployed with the official Helm chart in a distributed or simplified mode, depending on scale. Blocks are written to the bucket, and retention is configured in the compactor.

# Tempo configuration fragment (illustrative)
storage:
  trace:
    backend: s3
    s3:
      bucket: example-tempo-traces
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1
compactor:
  compaction:
    block_retention: 168h
distributor:
  receivers:
    otlp:
      protocols:
        grpc: {}

Check key names against the Tempo version you deploy. The bucket, its IAM role and lifecycle rules are created with Terraform, and workloads authenticate using IAM roles for service accounts rather than static keys, which fits well on EKS.

A TraceQL query to find slow failing checkout spans looks like this:

{ resource.service.name = "checkout-api" && status = error && duration > 2s }

Jaeger

Production Jaeger runs collectors, query service and a storage backend. With Elasticsearch or OpenSearch it creates daily indices, so the retention and index management considerations from our ILM page apply. Version and storage support change over time, so we confirm the current options and the status of any in-memory or Badger storage, which are not suitable for production at scale.

Connecting to the rest of your stack

Grafana can query Tempo or Jaeger as a data source. We add trace links from error panels, exemplars from Prometheus metrics where supported, and trace-to-logs settings so a span opens the matching log lines. Details are on the trace and log correlation page.

Sizing and retention

Estimate from requests per second, spans per request, average span size and sampling ratio. The result is storage per day. We build a simple spreadsheet with your numbers and agree a retention that matches how people debug. Often a week of detailed traces, plus metrics derived from spans, is a better balance than a month of everything. The Collector’s span metrics connector can produce RED style metrics from traces, which keeps long-term trends cheap.

Decisions and trade-offs

Object storage or database. Object storage is cheaper and simpler to scale. Databases allow richer ad hoc search but add an operational burden.

Self-hosted or managed. Managed tracing (from Grafana Cloud or other vendors) removes cluster work. It makes sense for small teams, and the price per GB will decide. OpenTelemetry means you can move later.

Multi-tenancy. If several teams or customers share a backend, tenant IDs and per-tenant limits prevent one noisy service starving another.

When not to run your own. If you trace only a handful of services with modest volume, a managed tier is likely cheaper than the time to operate a distributed backend.

Common failure modes

• Ingesters out of memory during bursts. We set limits, autoscale distributors and use Collector queues.
• Search too slow on large time windows. Query frontends, sensible time ranges and attribute choices help.
• Retention mismatch between compactor and bucket lifecycle rules.
• Trace IDs not found because sampling discarded them. Logs referencing unsampled traces need an explanation in the UI, so we align sampling with logging.
• Open query endpoint. Authentication is applied before anything else.

Cost control: what we do before it becomes a problem

Trace storage grows with traffic and with how many spans each request produces, so cost surprises usually trace back to a few chatty services. Our first step is a span-volume breakdown by service, taken from the Collector’s own exported metrics. A health check endpoint or a polling loop that emits a span every second can account for a large share of storage while providing almost no debugging value.

The usual remedies are in this order. Drop spans for health checks and readiness probes at the Collector. Reduce the sampling rate for high-volume, low-risk routes while keeping every error and slow trace. Trim oversized attributes such as full request bodies, which belong in logs, not spans. Only after those do we consider shortening retention.

We write these decisions down per service, so a later rise in cost can be compared with a known baseline rather than argued from memory.

Delivery and handover

You receive deployment code, sizing assumptions, dashboards for backend health, and a runbook for upgrades and scaling. We test failure cases: restart an ingester, throttle the bucket, drop the Collector link. If you want us to operate it, that is part of Managed DevOps.

How to tell it is working

Ingest and query latency stay within targets, dropped spans are near zero, the monthly cost matches the model, and engineers choose to open a trace first when debugging latency. To compare options for your volume, contact us.

References

• Grafana Tempo, Grafana Tempo documentation
• TraceQL, Grafana Tempo documentation
• Introduction, Jaeger documentation