Alertmanager

How it works

Prometheus evaluates alerting rules and sends firing alerts to Alertmanager over HTTP. Alertmanager does not evaluate rules itself. It takes the alerts it receives and decides what to send, to whom and how often.

Four mechanisms matter most:

• Grouping: alerts with similar labels (for example the same alertname and cluster) are combined into one notification, so a large outage does not produce hundreds of messages.
• Routing: a tree of routes matches alert labels and sends each alert to a receiver such as email, Slack, PagerDuty or OpsGenie, or to a webhook.
• Inhibition: while one alert is firing, notifications for related alerts can be suppressed. A “cluster unreachable” alert can mute the symptoms it causes.
• Silences: matchers that mute alerts for a fixed period, typically created in the web interface during planned maintenance.

For availability, Alertmanager can run as a cluster using the --cluster-* flags. Prometheus should be configured with the list of all Alertmanager instances rather than a load balancer in front of them, so each instance receives every alert and the cluster deduplicates notifications.

A practical example

A minimal routing configuration that sends critical alerts to a pager and everything else to chat:

route:
  receiver: team-chat
  group_by: ['alertname', 'cluster']
  routes:
  routes: [{matchers: ['severity="critical"'], receiver: on-call-pager}]

receivers: [{name: team-chat}, {name: on-call-pager}]

Receivers here are placeholders. A real configuration adds the integration settings, such as a Slack webhook or a PagerDuty routing key, to each receiver.

Common mistakes

• Grouping by too many labels, which defeats the point of grouping and recreates alert storms.
• Having no consistent severity label, so routing cannot separate pages from tickets.
• Using silences as a permanent fix instead of repairing the noisy alert or its threshold.
• Load balancing Prometheus traffic across Alertmanager replicas instead of listing every replica.
• Alerting on causes rather than symptoms that users feel, which adds noise.

When to use it and when not to

Use Alertmanager whenever Prometheus alerts must reach humans reliably, particularly with several teams or escalation paths. If you rely on a hosted observability product that provides its own routing, you may not need it. Either way, the alerts only help if they are tied to meaningful objectives such as SLI, SLO, SLA targets.

Related reading: Grafana. Our Alertmanager severity routing and silences page describes how we approach this.

See also

• Prometheus
• SLI, SLO, SLA
• Grafana