What OpenTelemetry is and why teams adopt it
OpenTelemetry is the common language for telemetry. It defines how applications describe traces, metrics and logs, provides SDKs for most popular languages, and ships a Collector that moves the data on. It deliberately stops there. There is no storage and no user interface, which is exactly why teams like it: instrument your code once, then choose and change backends without touching it again.
The usual reason for adopting it is lock-in. A proprietary agent embeds a vendor’s format and behaviour in every service, and switching later means re-instrumenting. With OpenTelemetry the instrumentation stays, and only Collector configuration changes. A second reason is trace quality: a single standard for context propagation means a request flowing through Java, Go and Node services still forms one trace. Our OpenTelemetry distributed tracing guide walks through a typical rollout, and the logging and telemetry service describes the engagement.
How we run OpenTelemetry in production
Collector topology
We usually run two tiers. A lightweight Collector as a DaemonSet or sidecar sits near workloads, receiving OTLP and adding Kubernetes metadata. A gateway Collector deployment behind a service does the heavier work: tail sampling, redaction and fan-out to backends. This keeps application configuration trivial (a single OTLP endpoint) and puts policy in one place.
A trimmed gateway pipeline looks like this:
receivers:
otlp:
protocols:
grpc: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
batch: {}
exporters:
otlp/tempo:
endpoint: tempo-distributor.tracing:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo]
The insecure TLS setting there is for an in-cluster example only. In production we enable TLS between tiers, and the memory limiter should always come first in a pipeline so back-pressure protects the Collector.
Instrumentation
Auto-instrumentation, through language agents or the Kubernetes Operator’s injection, produces spans for HTTP, gRPC, database and messaging calls. It is a good first week. After that, readability matters: span names should describe an operation (for example a route template, not a raw URL), and a few attributes such as tenant, plan or order type make traces searchable for business questions. We avoid putting personal data or unbounded values in attributes.
Correlation across signals
Traces become far more useful when logs and metrics line up with them. We inject trace and span IDs into structured logs, and keep resource attributes such as service name, namespace and version consistent across signals. Our write-up on trace and log correlation shows how this lets you jump from a slow span to the exact log lines.
Choosing trace storage
OpenTelemetry does not decide where traces live. Grafana Tempo and Jaeger are the common self-hosted choices, with Tempo favouring object storage and low cost, and Jaeger offering a mature UI and multiple backends. Metrics can flow to Prometheus and logs to Grafana Loki through the same Collector.
Sampling
Tracing everything is rarely affordable. Head sampling, configured in the SDK, keeps a fixed share of traces and is cheap. Tail sampling in the gateway waits until a trace completes and keeps all errors and slow requests plus a sample of the rest. Tail sampling needs all spans for a trace to reach the same Collector instance, so we place a load-balancing exporter in front of it. We decide the policy together with you, based on what incidents you need to explain, rather than copying a default.
Security and access
Telemetry can contain sensitive data: URLs with tokens, SQL statements, headers and user identifiers. We apply redaction and attribute filtering in the Collector, disable capture of request bodies, and restrict who can query traces. Transport between tiers uses TLS and authentication, and Collector endpoints are not exposed publicly. Credentials for backends are held in secrets rather than configuration files in Git.
Cost and capacity
OpenTelemetry is free software; costs come from Collector compute, network transfer and, above all, the volume your backend stores or bills for. Sampling, attribute discipline and dropping low-value spans such as health checks are the main controls. If you send data to a commercial platform, check how it prices ingested spans, custom metrics and logs before you widen collection.
Day-2 operations and failure modes
Common issues include broken context propagation through a queue or an unsupported library, Collector out-of-memory events during bursts, exploding metric cardinality from careless attributes and version drift between SDKs and the Collector. We run Collectors with limits and autoscaling, monitor their own metrics (dropped spans, queue length, export failures), pin versions and test upgrades. Because the project moves quickly and signal maturity varies, we check stability status for each component before relying on it.
Rolling it out without a big bang
Re-instrumenting everything at once is a mistake. We start with the services on the critical user path, enable auto-instrumentation, and confirm that traces connect across each hop before adding more. Gateways, message queues and background jobs are where context most often breaks, so we test those explicitly.
Where a vendor agent is already in place, the Collector can sit beside it during the transition, so both old and new pipelines receive data and engineers can compare them. Once confidence builds, the old agent is removed service by service. This keeps the change reversible and avoids a flag day.
When not to choose OpenTelemetry
If you have a handful of services, one cloud and are happy with a single vendor’s agent, adopting a vendor’s own tooling such as Datadog can be quicker, though you can still use OpenTelemetry later. Metrics-only needs are well served by Prometheus alone. If you want to explore a staged rollout, book a consultation.
Official documentation: opentelemetry.io