Prometheus Alerting Configuration

Problems we solve

• Alert storms overnight, mostly noise, so engineers stop reading them
• No runbook links in alerts, on-call engineer stares at a metric with no idea what to do
• Alerting on CPU > 80% for 1 minute, catches nothing real, pages constantly on spikes
• No multi-window burn rate alerts, SLO breaches go undetected until customers complain
• Alert rules not in version control, modified via the Prometheus UI and lost on restart

Our approach

1. 01

   Alert audit

   We review every existing alert rule. We classify each as: actionable (keep), noisy (tune threshold or duration), redundant (deduplicate), or absent (missing coverage). You receive a scored report.

2. 02

   Rule refactoring

   We rewrite noisy alerts with appropriate for durations, add absent() detection for missing metrics, and remove synthetic thresholds in favour of rate-of-change and trend-based rules where appropriate.

3. 03

   SLO alerting

   We implement the multiwindow, multi-burn-rate alerting approach from the Google SRE Workbook. This pages you when you are burning through your error budget fast, not just when a single threshold is crossed.

4. 04

   Alertmanager routing

   We configure the routing tree: critical alerts go to PagerDuty with escalation policies, warnings go to Slack, informational alerts go to email. Inhibition rules suppress downstream alerts during known outages.

5. 05

   Runbooks and documentation

   Every alert gets a runbook link. We either write runbooks for your most critical alerts or document the process for your team to complete them. Runbooks are stored in Git alongside alert rules.

Results