Alert Fatigue: Auditing and Reducing Alert Noise

The real cost of noisy alerts

A noisy pager is not just annoying. Each unnecessary page interrupts sleep or deep work, teaches engineers that alerts are usually false, and quietly raises the odds that the one real page is ignored. Teams in this state often mute channels, add “just acknowledge it” habits or stop trusting the monitoring altogether. Meanwhile the dashboards and tools keep generating more.

Cleaning this up is one of the fastest wins in our SRE and incident response service. It costs little, improves on-call quality of life immediately, and makes every later reliability investment more effective.

What good looks like

Every paging alert passes three tests:

1. Urgent. It needs action now, not tomorrow.
2. Actionable. There is something the on-call engineer can do, described in a runbook.
3. Meaningful. It reflects, or reliably predicts, harm to users or the business.

Anything failing a test belongs elsewhere: a ticket for later work, a dashboard for context, a log line for forensics, or the bin. A healthy system has few pages, and most of them lead to real action.

How the audit works

Start with evidence

Opinions about which alerts are noisy are unreliable. Data is not. We pull firing history from Alertmanager and your paging tool and rank rules by volume. A query like this against Prometheus shows which alerts fire most often:

topk(15, sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[30d])))

Note that ALERTS records firing state per evaluation, so the count shows relative frequency rather than a number of distinct incidents. We combine it with paging-tool data on acknowledgements, resolution times and notes. A rule that fires hundreds of times and is always resolved without action is the first target.

Score and decide

For each rule we record whether humans acted, how quickly it cleared on its own, whether a runbook exists and whether it duplicates another alert. The outcome for each is one of:

Fix the structure, not only the numbers

Raising thresholds helps but often hides a design problem. Structural fixes matter more:

• Alert on symptoms, not causes. Page on error and latency burn rather than on CPU. Our guidance on burn rate alerts covers the replacement pattern.
• Use for durations sensibly. A short blip that self-heals should not page.
• Group related alerts. One incident should produce one notification, not fifty.
• Inhibit dependent alerts. If the node is down, suppress the pod alerts on it.

A compact Alertmanager example showing grouping and inhibition:

route:
  group_by: [cluster, service]
  group_wait: 30s
  group_interval: 5m

inhibit_rules:
  - source_matchers: [alertname="NodeDown"]
    target_matchers: [severity="warning"]
    equal: [node]

The timing values are illustrative. Right values depend on how quickly you need to know, and we set them with your on-call engineers.

Trade-offs and judgement

Fewer alerts, more risk? Removing an alert always carries some risk. We reduce it by replacing noisy cause alerts with symptom alerts that cover the same failures, keeping a small set of predictive alerts for things like disk, certificates and quotas, and watching incident data after each change.

Tuning versus rebuilding. If your alert estate grew by accretion with hundreds of rules, a partial rewrite around SLOs is often more effective than endless threshold tweaking. If you have a few dozen reasonable rules, tuning is enough.

Tool-specific limits. Cloud-native alarms in services like CloudWatch have fewer grouping and inhibition features than Alertmanager. Where needed we route alarms into a central place so shaping logic is in one location.

When not to bother. If your pager is quiet and every page is useful, you do not need an audit. A quick review once or twice a year is enough.

Common failure modes

• Copying community alert rules wholesale. Generic rules fire on things that do not matter for your product.
• Alerting on every error. Some background error rate is normal.
• No owners. Nobody feels entitled to delete an old alert.
• Silencing instead of fixing. Muting hides the problem until it hurts.
• One-off clean-up. Noise returns within months without a review habit.

For examples of well-formed rules, see our Prometheus alerting rules guide.

A worked example of triaging one noisy alert

Take a hypothetical alert that fires on any pod restart in production. The audit asks four questions in order. Did a human act on it? If the answer is mostly no, it is a candidate for removal. Did it correlate with customer impact? If restarts happened without any failed requests, the signal belongs on a dashboard, not in a pager. Does another alert already cover the same failure? If a latency or error-rate alert would have fired anyway, this one is a duplicate. Finally, would a ticket in working hours be enough?

The usual outcome is a demotion rather than deletion: the rule keeps firing, but its route changes from page to ticket, and a restart-loop condition (many restarts in a short window) keeps a page. Record every decision in the audit sheet with the reason, so the next engineer does not resurrect the old rule by accident.

Grouping and inhibition

Much noise is one fault reported many times. In Alertmanager, group_by collapses alerts for the same cluster and service into a single notification, and inhibition rules mute downstream symptoms while a root-cause alert such as a node-down or database-down is active. We review these settings in every audit because defaults rarely match how your services depend on each other.

Delivery and handover

We deliver the audit report listing every rule and its outcome, the updated rule files and routing configuration in your repository, a before and after comparison of page volume, and a short guide on adding new alerts so they follow the same standard. A recurring review slot is agreed so noise does not creep back. Teams on Managed DevOps get this as an ongoing practice.

How to tell it is working

Measure pages per on-call shift, the share of pages that led to action, out-of-hours interruptions and time to acknowledge. Add a qualitative check: ask the on-call engineers whether they trust the pager. If page volume falls while real incidents are still caught, and engineers stop dreading their shift, the audit has done its job. To start with a review of your current alerts, talk to us.

References

• Monitoring Distributed Systems, Google SRE book
• Alerting, Prometheus documentation
• Alertmanager, Prometheus documentation