Alertmanager Configuration: Severity Routing, Grouping and Silence Policies

Where noisy alerting comes from

Alert fatigue rarely comes from one bad alert. It comes from the way notifications are assembled. Every pod restart pages. A node failure produces forty alerts, one per workload on it. A planned deploy trips three warnings that someone dismisses with a thumbs-up. After a month the on-call engineer assumes any page is probably nothing, and the one that matters gets slow attention.

Alertmanager is the component that can prevent this, and it is configured badly in most setups we audit: a single receiver, no grouping, no ownership labels. Fixing it is a core part of our monitoring and observability service. For the rules side, the Prometheus alerting page and our alerting rules examples are the companion reading.

What good looks like

• Every alert has labels that route it: severity, team and ideally service.
• A page means act now. If it can wait until morning it is not a page.
• One incident, one notification. Related alerts group together.
• Root causes suppress symptoms.
• Silences are accountable. They expire and say why they exist.
• Configuration is tested and in Git. Nobody edits routes in a live pod.

Defining severity

We write this down before touching YAML, because the technical tree just encodes a decision:

Your own wording may differ. What matters is that the team agrees and that alerts are classified when written.

How it works in practice

The routing tree

A trimmed configuration that routes by severity and team, groups sensibly and sends everything else to a safe default:

route:
  receiver: default-slack
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="critical"']
      receiver: pagerduty-platform
      continue: false
      routes:
        - matchers: ['team="payments"']
          receiver: pagerduty-payments
    - matchers: ['severity="warning"']
      receiver: team-slack
    - matchers: ['alertname="Watchdog"']
      receiver: deadmans-switch
      repeat_interval: 1m

inhibit_rules:
  - source_matchers: ['alertname="ClusterDown"']
    target_matchers: ['severity=~"warning|critical"']
    equal: ["cluster"]
  - source_matchers: ['severity="critical"']
    target_matchers: ['severity="warning"']
    equal: ["alertname", "service"]

receivers:
  - name: default-slack
  - name: team-slack
  - name: pagerduty-platform
  - name: pagerduty-payments
  - name: deadmans-switch

Receiver bodies (webhook URLs, integration keys) are omitted here and come from secrets, covered in the PagerDuty, Opsgenie and Slack routing page. In the Prometheus Operator, this configuration is held in an AlertmanagerConfig resource or a secret, and we choose based on how your teams own their routes.

Grouping timings

group_wait is how long Alertmanager waits to collect alerts of a new group before the first notification. group_interval governs updates to an existing group. repeat_interval controls reminders for unresolved alerts. Short waits give faster pages but more fragments; long waits are calmer but delay the first signal. For critical routes we keep group_wait short, often tens of seconds.

Validating before deploy

amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=critical team=payments

The second command prints the receiver a given label set will reach. We put this in CI so a routing change that would drop pages fails the pipeline.

Silence policy

Silences are legitimate and necessary. The danger is the forgotten one. Our policy is simple:

1. Created with an author and a comment that names a ticket or reason.
2. Always has an end time, with a short default such as the length of the maintenance window.
3. Reviewed in the weekly operations check, with any silence older than a few days questioned.
4. Planned maintenance uses a silence matched on a precise label set, never a blanket match.

Decisions and trade-offs

Central or team-owned routes. One central file is simple and consistent but becomes a bottleneck. Per-team AlertmanagerConfig resources scale ownership, with a platform-owned top-level route that guarantees nothing escapes unrouted.

Alertmanager or the alerting built into Grafana. For Prometheus-based alerts, Alertmanager is mature and fits Git workflows. Grafana’s alerting is useful for mixed data sources. We avoid doing both for the same rule.

Inhibition risks. Over-eager inhibition can hide real problems. We use narrow equal labels and test with realistic failure scenarios.

When to stop tuning. If you have a small number of alerts and one on-call person, a basic route with grouping may be all you need.

Common failure modes

• Unlabelled alerts that fall to the default receiver and get ignored.
• Over-grouping that merges unrelated problems into one notification.
• No dead-man’s-switch, so a broken pipeline is silent. A Watchdog alert that always fires, routed to an external heartbeat check, fixes this.
• Single Alertmanager replica restarting during an incident.
• Secrets in config committed to Git.

A worked example: one node failure

Consider a node that loses power. Without design, every workload on it raises its own alert, plus a NodeNotReady alert, plus warnings from services that lost replicas. Five teams are paged separately and each starts investigating the same fault. With the tree above, grouping collapses the alerts that share a cluster and service into single notifications, and an inhibition rule keyed on the node alert suppresses the dependent symptoms. The platform on-call receives one page that names the cause, and application teams see only a quiet channel message that their replicas are rescheduling. The same failure, handled well, costs one interruption instead of five. This is the sort of scenario we replay in a test environment before sign-off, so you can see the behaviour rather than trust the YAML.

Delivery and handover

You receive the severity policy, the routing tree, inhibition rules, a silence procedure, CI validation and a short guide to adding a new team or receiver. We review volume and noise after two weeks of production traffic and again at the end of the engagement. If you also want help with the human side of rotations, see how it connects to on-call schedule integration.

How to tell it is working

Track pages per on-call shift, the share of pages that required action, alerts without a team label (target none), time-to-acknowledge, and silences older than their intended window. If engineers trust a page enough to respond quickly, routing is doing its job. To review your current Alertmanager setup, get in touch.

References

• Alertmanager configuration, Prometheus documentation
• Alertmanager, Prometheus documentation
• Alerting practices, Prometheus documentation