Prometheus

Prometheus uses a pull model: it scrapes HTTP endpoints on your services, rather than services pushing metrics to a central collector. This makes it easy to add monitoring to any service by exposing a /metrics endpoint in the Prometheus exposition format.

How it works

Prometheus stores metrics as time series. Each series is identified by a metric name and a set of key-value labels, and each sample has a timestamp. The server finds targets through service discovery or static configuration, scrapes them at a set interval, and keeps the samples in local storage. Each server operates independently and does not depend on distributed or network storage.

Other parts of the ecosystem include client libraries for instrumenting code, exporters for systems that do not expose Prometheus metrics themselves, and the Alertmanager. Prometheus evaluates alerting rules against stored data and sends firing alerts to the Alertmanager, which handles grouping, routing and notification. PromQL is the query language, and it makes use of the label dimensions. Short-lived batch jobs that cannot be scraped can push to a Pushgateway, which Prometheus then scrapes.

A practical example

A minimal scrape configuration for one service:

scrape_configs:
- job_name: my-app
  scrape_interval: 15s
  static_configs: [{targets: ["my-app:8080"]}]

A PromQL query for the per-second request rate over five minutes, summed by status code label:

sum by (code) (rate(http_requests_total[5m]))

In Kubernetes you would normally use service discovery rather than static targets, and visualise results in Grafana.

Common mistakes

• Using high-cardinality labels, such as user identifiers or request paths with IDs. Every distinct label combination creates a new series, which increases memory and storage use.
• Applying rate() to a gauge, or averaging counters directly. Counters should be wrapped in rate() or increase().
• Treating the local storage as long-term, durable storage. Plan retention and use a remote storage system if you need history or a view across clusters.
• Using Prometheus where every event must be counted exactly, such as per-request billing. The project states it favours reliability and is not suited to workloads needing 100 percent accuracy.
• Alerting on causes rather than symptoms, which creates noisy pages.

When to use it and when not to

Prometheus is a strong default for infrastructure and service metrics in Kubernetes, especially for alerting on service level indicators (see SLI, SLO and SLA). It is less suited to log search, tracing, or exact accounting, where other tools fit better. For wider telemetry, consider combining it with OpenTelemetry. See our Prometheus technology page and monitoring service.

See also

• Grafana
• Alertmanager
• OpenTelemetry
• SLI, SLO and SLA