Linking Runbooks from Prometheus Alert Annotations

The 3am problem

An alert fires at 03:12: “HighErrorRate on payments-api”. The engineer is half awake. They open a laptop, find the dashboard, scan the panels, and then ask: is this normal, what usually fixes it, who knows this service? Five or ten minutes disappear before any action, and they are the most expensive minutes of the incident.

A runbook link in the alert removes most of that. It is a small piece of the monitoring and observability service with an outsized effect on response time, because it travels with the page. It is also where monitoring meets operations, which is why the same idea appears in our wider Managed DevOps runbooks work.

What good looks like

• Every paging alert links to a runbook. No exceptions without a recorded reason.
• The link is in the notification itself, not three clicks away.
• The runbook matches the alert. Same name, same symptoms, same dashboard.
• Short and tested. A second engineer has followed it successfully.
• Owned. A named team reviews it.
• Enforced by automation, not memory.

The annotation standard

We agree a small, consistent set of annotations across every rule:

A rule following this standard:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-alerts
  namespace: monitoring
spec:
  groups:
    - name: payments
      rules:
        - alert: PaymentsHighErrorRatio
          expr: |
            sum(rate(http_server_requests_total{service="payments-api",status=~"5.."}[5m]))
            /
            sum(rate(http_server_requests_total{service="payments-api"}[5m])) > 0.02
          for: 10m
          labels:
            severity: critical
            team: payments
          annotations:
            summary: "Payments API 5xx ratio above 2% for 10 minutes"
            description: "Error ratio is {{ $value | humanizePercentage }} on {{ $labels.service }}."
            impact: "Customers cannot complete payments."
            dashboard_url: "https://grafana.example.com/d/payments/payments-api"
            runbook_url: "https://github.com/your-org/runbooks/blob/main/payments/high-error-ratio.md"

The description uses Prometheus template functions, and $value is the evaluated expression result. The example URLs are placeholders for your own domains.

Getting the link into the page

Annotations are only useful if the receiver shows them. In Alertmanager templates we include them in Slack text and in the details block of paging integrations, as shown in alert routing. Grafana’s own alert view also shows annotations, and many incident tools surface a runbook field. A quick test: fire a test alert and check whether you can reach the procedure in one tap from a phone lock screen notification.

Writing the runbooks

Start from the history

We pull the last months of firing alerts and the incident notes behind them. The questions we ask: what did the person check first, what finally fixed it, and what did they wish they had known? Those answers are the runbook. The template mirrors the one we use elsewhere: symptoms, impact, quick checks, actions, verification and escalation.

A compact example

# PaymentsHighErrorRatio

Owner: payments team. Last reviewed: 2026-09-20.

## Quick checks (two minutes)
1. Did a deploy land recently? `kubectl rollout history deploy/payments-api -n production`
2. Is the payment provider degraded? Check their status page.
3. Are database connections near the limit? See the "DB connections" panel.

## Actions
1. Recent deploy and errors began after it: `kubectl rollout undo deploy/payments-api -n production`
2. Provider outage: enable the fallback queue flag, then notify support.
3. Otherwise escalate to the payments developer on call.

## Verify
Error ratio under 1% for 10 minutes.

Kill the alert instead

Sometimes writing the runbook exposes that no meaningful action exists. “Wait and see if it resolves” is not a procedure. These alerts should be deleted, converted to a dashboard panel, or turned into a non-paging warning. Cutting noise this way is one of the most valuable by-products of the exercise, and it links back to the Alertmanager severity model.

Enforcing it in CI

Humans forget; pipelines do not. A simple check using yq and promtool can fail a pull request that adds a critical rule without a runbook:

# fail if any critical alert lacks a runbook_url
missing=$(yq -r '
  .spec.groups[].rules[]
  | select(.labels.severity == "critical" and (.annotations.runbook_url // "") == "")
  | .alert' rules/*.yaml)

if [ -n "$missing" ]; then
  echo "Critical alerts without runbook_url:"; echo "$missing"; exit 1
fi

promtool check rules rules/*.yaml

Adjust the file paths and extraction expression to match how your rules are stored, since a PrometheusRule wraps rule groups under spec, while a plain rule file does not. A weekly job that requests every runbook_url and reports non-200 responses catches dead links.

Decisions and trade-offs

Where to host. Git-rendered Markdown is durable and reviewable. A wiki is friendlier to edit but may share fate with your infrastructure. We often keep source in Git and mirror elsewhere. If you use a developer portal such as Backstage, the runbook can be surfaced there too.

One runbook per alert or per component. Per alert is precise. Per component reduces duplication when several alerts share a cause. We use per alert for paging rules and allow shared sub-procedures through links.

Automation. A runbook step that is safe and repetitive can become a script or pipeline job. The document should still explain when to use it. See how this builds on on-call schedule integration.

Failure modes

• Placeholders left in: “TODO: add steps”.
• Links to private spaces that on-call cannot open.
• Pinned commit URLs that show an old version.
• Runbooks nobody tested, assuming access the engineer does not have.
• Copy-paste runbooks whose commands target the wrong environment.

Delivery and handover

You receive the annotation standard, runbooks for your paging alerts, the template, CI checks, and a review schedule. Ownership is assigned to named teams, and we show your developers how to add one when they add a new alert.

How to tell it is working

Measure the share of paging alerts with a valid runbook link (target all), time from acknowledgement to the first correct action, and the number of pages resolved by someone seeing the service for the first time. If a new on-call engineer can handle a routine alert on their first shift, it is working. To get your runbooks wired in, contact us.

References

• Alerting rules, Prometheus documentation
• Template examples, Prometheus documentation
• Monitoring Distributed Systems, Google SRE book