Error Budget Dashboards and Burn Rate Alerting

Why threshold alerts fail

A classic alert says “error rate above 2% for five minutes”. It sounds sensible and behaves badly. A brief spike at 3am pages someone even though it consumed almost none of the month’s tolerance. A slow, steady 0.5% error rate never trips the threshold yet quietly exhausts the entire objective. The alert tracks a number, not the harm.

Burn rate alerting measures harm directly. It asks how quickly you are using up the unreliability your SLO allows, and pages only when that pace threatens the objective. It is one of the highest-value pieces of our SRE and incident response service, and it builds directly on the SLI and SLO definitions.

The idea in plain terms

Suppose the objective is 99.9% success over 30 days. The budget is 0.1% of requests. If the service fails at exactly 0.1% for the whole month, you finish at zero: a burn rate of 1. If it fails at 1.44% for an hour, you have burned 2% of the month’s budget in that hour, a burn rate of 14.4. That pace would empty the budget in about two days, so a human should look now.

The Google SRE workbook popularised a set of tiers that work well as a starting point. We tune them per service rather than copying them blindly:

The long window establishes that the burn is meaningful. The short window confirms it is still ongoing, so the alert stops soon after recovery.

A working Prometheus example

Assuming an SLI recording rule that yields an error ratio over several windows, the fast burn alert looks like this:

groups:
  - name: checkout-slo-burn
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          sli:checkout_error_ratio:rate1h > (14.4 * 0.001)
          and
          sli:checkout_error_ratio:rate5m > (14.4 * 0.001)
        labels:
          severity: page
          service: checkout
        annotations:
          summary: "Checkout is burning its 30 day error budget at 14.4x"
          runbook_url: "https://github.com/your-org/runbooks/blob/main/checkout-errors.md"

Here 0.001 is the budget fraction for a 99.9% objective. Hand-writing this for several windows and services is error prone, which is why we often generate the rules with Sloth or Pyrra from a short SLO definition and keep the generated output in version control. Our Prometheus alerting rules guide covers the syntax in more depth.

Dashboards that drive decisions

A good error budget dashboard answers four questions in one glance:

1. How much budget is left in the window, as a percentage and in minutes of full outage equivalent?
2. How fast are we burning right now, compared with the alert thresholds?
3. Which incidents or deploys consumed the budget?
4. What does the policy say we should do at this level?

In Grafana we build a per-service row with these panels and a summary page across all critical services. Annotations mark deploys and incidents so the cause of a steep drop is visible without detective work. The same dashboards support the monthly review and are useful in post-mortems.

The policy matters more than the maths

An error budget that nobody acts on is decoration. We write a short policy, agreed with product and engineering leadership, that says what changes as the budget falls. A typical shape:

• Plenty remaining: ship normally and take reasonable risks.
• Low: require extra review or staged rollouts for risky changes.
• Exhausted: pause non-essential feature releases and prioritise reliability fixes until the window recovers.

The policy should allow exceptions for security fixes and legal obligations. It should also be reviewed after the first quarter, because the first version is nearly always too strict or too vague.

Trade-offs and when this is overkill

Low traffic services. With very few requests, a handful of errors swings the ratio wildly and burn alerts become noisy. For these we use longer windows, synthetic probes to generate steady traffic, or a simple alert on consecutive probe failures.

Complexity. Multi-window rules are harder to understand than a plain threshold. We document each alert and keep the tiers consistent across services so on-call engineers learn them once.

Not every service needs it. Internal tools with relaxed expectations may only need a ticket-level slow burn alert. We reserve paging for journeys where users or revenue are hurt.

Dependencies. If your service burns budget because a third-party API is failing, the alert is still correct, but the fix may be a fallback rather than a code change. The runbook should say so.

Common failure modes

• Copying thresholds without checking traffic. Quiet services need different windows.
• Paging on slow burn. A slow leak belongs in a ticket queue, not at 3am.
• Missing runbooks. An alert with no next step wastes the early warning. See the runbook library.
• Budget policy never invoked. Leadership ignores it the first time it is inconvenient, and it stops meaning anything.

Worked example: reading a burn rate

Suppose a service has a 30-day availability objective. A burn rate of 1 means errors are arriving at exactly the pace that would use the whole budget by the end of the window. A burn rate of 10 would use it in a tenth of the time. That framing is why the same number can mean “ticket” or “page”: the higher the rate, the shorter the window you have before the budget is gone.

This also explains the paired short and long windows in the rule above. The long window proves the problem is sustained and not a blip. The short window proves it is still happening now, so the alert resolves quickly once you fix it instead of paging for an hour after recovery.

Low-traffic services

Burn-rate alerts behave badly when traffic is tiny: a handful of failed requests can look like a catastrophic rate. Options include widening the window, alerting on a minimum request count, adding a synthetic probe to generate steady traffic, or aggregating several related endpoints into one SLI. We choose per service and write the reason down, because this is the first thing a newcomer will question.

Delivery and handover

We deliver the rules and dashboards as code in your repository, the routing configuration for Alertmanager and your paging tool, the written budget policy, and a short guide on adding a new service. Your team can then extend the pattern without us. Where it helps, we also fold the review into a Managed DevOps cadence.

How to tell it is working

Track pages per week that required action, the share of incidents detected by burn alerts before customers complained, and the number of times the budget policy changed a release decision. If paging volume falls while customer-reported incidents also fall, the alerts are earning their keep. To discuss your services, contact us.

References

• Alerting on SLOs, Google SRE workbook
• Example Error Budget Policy, Google SRE workbook
• Alerting rules, Prometheus documentation