OpenTelemetry

Before OpenTelemetry, every observability vendor had its own SDK. Switching from one vendor to another meant re-instrumenting every service. OpenTelemetry standardises instrumentation: you add the OTel SDK once and route data to different backends through the OpenTelemetry Collector.

How it works

OpenTelemetry is a framework and toolkit, not a backend. It does not store or visualise data. It defines APIs and language SDKs that applications use to generate traces, metrics and logs, and OTLP, a standard protocol for the format of that data. Semantic conventions standardise attribute names, so an HTTP request is described the same way whichever language produced it. Auto-instrumentation components can generate telemetry for supported languages without code changes. The project belongs to the Cloud Native Computing Foundation and came from a merger of OpenTracing and OpenCensus.

The Collector is a separate process that receives, processes and exports telemetry. It is assembled from receivers, processors, exporters, connectors and extensions, which you wire together into pipelines. Running a Collector close to your services lets them offload data quickly, while the Collector handles retries, batching, encryption and filtering of sensitive data. It can run as an agent next to workloads or as a standalone collector service.

A practical example

A common Kubernetes layout has each service export OTLP to a Collector Service inside the cluster. A minimal Collector configuration receives OTLP and prints traces for testing, and you later swap the exporter for your chosen backend:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
exporters:
  debug:
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Changing backend then means editing the exporters and pipeline in one place, not redeploying every application.

Common mistakes

• Treating OpenTelemetry as a monitoring product. You still need backends for storage, querying and alerting, such as Prometheus for metrics.
• Defining a component but not listing it under the pipelines in the service section. Components that are configured but not used in a pipeline do nothing.
• Exporting from every service straight to a vendor, which recreates the lock-in the standard is meant to remove.
• Ignoring semantic conventions, which makes dashboards and queries inconsistent across teams.
• Adding unbounded values, such as user identifiers, as metric attributes, which can inflate storage cost in the backend.

When to use it and when not to

OpenTelemetry suits organisations with several services or languages, a wish to avoid vendor lock-in, or a need to correlate traces, metrics and logs consistently. It pairs naturally with distributed tracing across microservices. For a single small application that is happy with one vendor’s agent, the extra Collector layer may add more moving parts than benefit. Our OpenTelemetry technology page and logging and observability service give more context.

See also

• Distributed Tracing
• Prometheus
• Log Aggregation
• Grafana