From “is it up?” to “is it good enough?”
Most monitoring answers a binary question: is the thing up. Customers do not experience the world that way. A checkout that responds in nine seconds is technically up and effectively broken. A login that fails for two percent of users is up for the other ninety-eight.
SLOs are the discipline that turns this into numbers, and error budgets are the mechanism that turns the numbers into decisions. They are part of the monitoring and observability service because they sit on top of the metrics you collect. The vocabulary is covered in our SLI, SLO and SLA glossary entry. If you need ongoing reliability practice around them, our SRE and incident response service goes further.
What good looks like
- Few and meaningful. Two or three SLOs per important service, tied to user journeys.
- Measured close to the user. Ideally at the load balancer, ingress or a synthetic probe, not deep in a backend that cannot see failures upstream of it.
- Agreed with the business. A product owner has signed off on what “good enough” means.
- Windowed and visible. A rolling 28 or 30 day window with budget remaining shown on a dashboard people actually open.
- Wired to alerts. Burn rate, not raw thresholds.
- Tied to policy. The team knows what changes when the budget is low.
Defining the SLIs
The two most common indicators are availability and latency.
Availability SLI: good requests divided by valid requests. Good usually means a non-5xx response. Be careful with client errors: a 404 caused by a bad user URL is not your failure, but a 429 caused by your own rate limits might be.
Latency SLI: the proportion of requests served faster than a threshold, for example under 300 ms. Using a threshold rather than a percentile makes it a ratio, which composes cleanly into an error budget.
Other indicators include freshness for data pipelines, durability for storage and correctness for batch jobs. We choose with you.
How it works in practice
Recording rules
Given the standard request metrics from instrumentation, an availability SLI over 5 minutes looks like this:
groups:
- name: checkout-slo
rules:
- record: sli:checkout_availability:ratio_rate5m
expr: |
sum(rate(http_server_requests_total{service="checkout", status!~"5.."}[5m]))
/
sum(rate(http_server_requests_total{service="checkout"}[5m]))
- record: sli:checkout_latency_good:ratio_rate5m
expr: |
sum(rate(http_server_request_duration_seconds_bucket{service="checkout", le="0.3"}[5m]))
/
sum(rate(http_server_request_duration_seconds_count{service="checkout"}[5m]))
Note the latency rule depends on a histogram bucket existing at le="0.3", which is why bucket choice is part of instrumentation design.
Multi-window burn rate alerts
For a 99.9% SLO the error budget fraction is 0.001. A page for fast burning uses a short and a long window together, so brief blips do not page and sustained problems do:
- alert: CheckoutErrorBudgetFastBurn
expr: |
(1 - sli:checkout_availability:ratio_rate5m) > (14.4 * 0.001)
and
(1 - avg_over_time(sli:checkout_availability:ratio_rate5m[1h])) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
team: payments
annotations:
summary: "Checkout is burning its 30 day error budget at over 14x the sustainable rate"
runbook_url: "https://github.com/your-org/runbooks/blob/main/checkout-slo.md"
The 14.4 factor comes from the widely used approach in the Google SRE workbook, where a burn that would consume a meaningful fraction of the monthly budget in about an hour is treated as an emergency. A slower burn rule uses a lower factor over longer windows and routes to a ticket instead of a page. Generators like Sloth or Pyrra emit these combinations for you from a short spec, which avoids copy-paste mistakes.
Dashboards
A good error budget panel shows the SLO target, current SLI, budget remaining as a percentage for the window, burn rate over the last hour and day, and a time series of consumption. Pairing it with the underlying request and error panels lets an engineer move from “budget is draining” to “which route” quickly. For 30 day windows with high-resolution data, we often pair this with long-term storage, since short local retention can cut the window short.
Making the budget matter
The dashboard is the easy part. The hard part is the policy. A workable one is short:
- Budget healthy: ship normally.
- Budget low: reliability work takes priority over risky feature releases for the affected service.
- Budget exhausted: freeze non-essential releases until recovery, and review causes.
This works only if leadership has agreed to it in advance. Otherwise the budget is a number on a screen.
Decisions and trade-offs
Request-based or time-based. Request-based SLIs handle uneven traffic better. Time-based measures like “minutes of downtime” are easier to explain to executives. We usually compute request-based and translate.
Low-traffic services. With few requests, one failure moves the ratio sharply. Synthetic probes or longer windows give steadier signals.
When not to bother. An early-stage product with a handful of users and no on-call may get more from basic uptime and error alerts. SLOs repay their cost once reliability becomes a trade-off against speed.
Failure modes
- Targets copied from marketing. Five nines on a single-region, single-database stack is fiction.
- Too many SLOs. Nobody reads thirty dashboards.
- Measuring the wrong place. A backend metric that ignores load balancer failures flatters the result.
- Alerting on every dip. Single-window thresholds reintroduce noise.
- No owner. Review SLOs each quarter.
Choosing the first SLOs
Teams new to SLOs often try to define them for every service at once and stall. We recommend starting with two or three user-facing journeys and one objective for each, usually availability and latency. Pick journeys where a failure would be noticed by customers or revenue, not internal components that happen to have the most metrics.
Draft the objective from observed behaviour, not aspiration. Look at how the service has actually performed over recent months, set a target that is slightly ambitious but achievable, and agree it with product as well as engineering. An objective nobody can meet teaches people to ignore the budget, and one that is trivially met tells you nothing. Revisit each target after a quarter of data, and expect to adjust it. The first version is a starting point for a conversation, and the dashboards simply make that conversation concrete.
Delivery and handover
You get an SLO document per service, rules in Git, dashboards, burn rate alerts routed to the right receivers, and a half-day workshop with product and engineering on the policy. We offer a review after the first full window to adjust targets with real data.
How to tell it is working
Look for fewer threshold-based pages, budget status referenced in release or planning conversations, and incidents that were caught by burn rate before customers wrote in. To start defining SLOs with your team, book a consultation.