Alert Routing to PagerDuty, OpsGenie or Slack from Prometheus

Routing is where alerts become people

A perfectly tuned alert is worthless if it reaches nobody, or reaches everybody. We have seen companies where critical alerts landed in a Slack channel muted by half the team, and others where every warning triggered a phone call. Both are routing problems, not rule problems.

This page covers the last mile of the monitoring and observability service: connecting Alertmanager to the tools your people use. For reading on the underlying component, see the Alertmanager glossary entry.

What good looks like

• Right channel for the urgency. Pages for critical, chat for warnings, tickets for the slow burn.
• Clear ownership. Each service has one team that is paged, and an escalation path if they do not respond.
• Rich notifications. A responder can see what broke, how bad it is, and where to look, without opening a laptop first if possible.
• Closed loop. Resolved alerts close the incident or message automatically.
• No secrets in Git.
• Tested end to end before it is needed.

Choosing the destination

We do not push a vendor. If your team is two people and a phone number, a simple chat and SMS arrangement may be enough, and we will tell you if paying for a platform is premature.

How it works in practice

PagerDuty receiver

receivers:
  - name: pagerduty-platform
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-platform
        send_resolved: true
        severity: '{{ if eq .CommonLabels.severity "critical" }}critical{{ else }}warning{{ end }}'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          cluster: '{{ .CommonLabels.cluster }}'
          service: '{{ .CommonLabels.service }}'
          runbook: '{{ .CommonAnnotations.runbook_url }}'
          dashboard: '{{ .CommonAnnotations.dashboard_url }}'
        links:
          - href: '{{ .CommonAnnotations.runbook_url }}'
            text: Runbook

The routing_key_file option reads the integration key from a mounted file, so the key never appears in your configuration repository.

Opsgenie receiver

  - name: opsgenie-backend
    opsgenie_configs:
      - api_key_file: /etc/alertmanager/secrets/opsgenie-backend
        message: '{{ .CommonAnnotations.summary }}'
        priority: '{{ if eq .CommonLabels.severity "critical" }}P1{{ else }}P3{{ end }}'
        responders:
          - name: backend-team
            type: team
        send_resolved: true

Opsgenie priorities are P1 to P5 in its own system; they are unrelated to any incident severity numbering inside your company, so we map them explicitly.

Slack receiver

  - name: team-slack
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: "#alerts-platform"
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: |-
          {{ range .Alerts }}
          *{{ .Annotations.summary }}*
          Impact: {{ .Annotations.impact }}
          <{{ .Annotations.runbook_url }}|Runbook>  <{{ .Annotations.dashboard_url }}|Dashboard>
          {{ end }}

Slack messages are for people to read, so we keep them short and link out. A wall of labels teaches the channel to be ignored.

Secrets handling

On Kubernetes we sync integration keys from your cloud secret manager using External Secrets Operator, or store them encrypted with Sealed Secrets, and mount them into the Alertmanager pod. Rotation is a secret update plus a config reload. The general pattern is the same one described for ConfigMaps and Secrets.

Escalation design

Routing to a tool is half the job. The escalation policy inside it matters just as much. A typical policy pages the primary on-call, waits a defined interval for acknowledgement, then pages the secondary, then an engineering manager. We agree those intervals with you, enter them, and test a real escalation by letting one test page go unacknowledged.

Decisions and trade-offs

One integration per team or per service. Per service gives sharp ownership but many objects to manage. Per team is simpler. We usually choose per team with a service label for context.

Resolved messages. Auto-resolve is convenient but can hide flapping alerts. We enable it for paging tools so incidents close, and keep Slack resolved messages short.

Follow-the-sun and holidays. These are properties of schedules, not of Alertmanager. They need schedule hygiene in the paging tool, or an outsourced rotation. Our Managed DevOps service covers that where you want it.

Failure modes

• Wrong integration key for the wrong environment, so staging pages production on-call.
• Webhook rate limits from alert storms. Grouping prevents most of this.
• Muted channel. Review channel membership and notification settings.
• Unreachable endpoint. Egress rules blocking Alertmanager from the vendor. We check network policy and proxy settings.
• Test never run. The first real alert should not be the first test.

Testing routes safely

Routing changes are risky because a mistake produces silence rather than an error. We test every change in two ways. First, amtool is run in your pipeline to confirm that representative label sets reach the intended receiver. Second, a non-production Alertmanager sends real test notifications to a sandbox service and a test Slack channel, so formatting, links and escalation can be checked without waking anyone.

We also keep a short list of canonical test alerts, one per severity and team, that can be fired on demand. After any change to the tree, those alerts are replayed and the results compared with what was expected. This habit turns routing from something people are afraid to touch into something that can be improved routinely, which matters because your teams, services and ownership will keep changing.

Planned changes

When a team is added, renamed or merged, update the labels on its alerts and the route together in one pull request. Changing only one half is how pages end up in the default channel.

Delivery and handover

You receive working receivers for every route, templates, a secrets approach, a diagram of who is reached for each severity, and the results of an end-to-end test for each path. We keep the documentation next to the Alertmanager configuration so changes stay in sync.

How to tell it is working

Fire a synthetic critical alert monthly and time how long it takes someone to acknowledge. Review missed or duplicate pages, channels with unread alert backlog, and the percentage of pages with a runbook link. To get routing sorted out, contact us.

References

• Alertmanager configuration, Prometheus documentation
• Prometheus Integration Guide, PagerDuty documentation
• Integrate Opsgenie with Prometheus, Atlassian Support