Why SLOs come before everything else in SRE
Most teams that ask us about reliability already have dashboards. What they lack is an agreed answer to a simpler question: how reliable does this service need to be? Without it, every incident is argued from scratch, every release is either too cautious or too reckless, and the on-call engineer cannot tell a nuisance from an emergency.
Service level objectives give those arguments a shared reference. They turn “the site felt slow” into “the checkout journey missed its latency objective for 40 minutes on Tuesday”. That sentence can be prioritised, budgeted and fixed. This page explains how we define SLIs and SLOs as part of our SRE and incident response service, and where the approach does and does not fit.
What good looks like
A usable SLO has five parts: a user journey, an indicator, a target, a window and a consequence. Miss any of them and it degrades into a dashboard widget.
- Journey. Described in words a product manager would use, such as “a signed-in user can place an order”.
- Indicator. The ratio of good events to valid events, measured as close to the user as practical.
- Target. A percentage derived from history and tolerance.
- Window. Usually a rolling 28 or 30 days, so one bad day does not vanish at the end of a calendar month.
- Consequence. What the team agrees to do when the budget is spent, for example freezing risky releases and prioritising reliability work.
Choosing indicators that reflect users
Resource metrics describe your machines. SLIs should describe your users. A healthy CPU graph tells you nothing about whether a payment went through.
For request-driven services, two indicators cover most needs:
- Availability: the proportion of requests that return a non-error response.
- Latency: the proportion of requests faster than a threshold that users notice.
For pipelines and batch work, use freshness (how old is the newest processed record) and correctness (the proportion of outputs passing validation). For storage and queues, use durability and processing delay.
Where you measure matters. Measuring at the application can miss everything the load balancer drops. Measuring at the load balancer or ingress catches more, which is why we usually start there on Kubernetes and cloud load balancers, then add synthetic probes for journeys that span services.
A concrete example
Here is a Prometheus recording rule set for an availability and latency SLI on an HTTP service, using the standard request counter and latency histogram:
groups:
- name: checkout-sli
rules:
- record: sli:checkout_availability:ratio_rate5m
expr: |
sum(rate(http_requests_total{service="checkout",code!~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))
- record: sli:checkout_latency_under_500ms:ratio_rate5m
expr: |
sum(rate(http_request_duration_seconds_bucket{service="checkout",le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count{service="checkout"}[5m]))
The latency rule only works if a histogram bucket exists at the threshold you want, so we choose bucket boundaries deliberately during instrumentation. The objective itself is then declared separately, for example “99.9% over 30 days”, and a generator or a small set of rules produces the budget and burn calculations. Those feed the error budget dashboards and burn rate alerts.
Setting the target without guessing
We use a simple sequence. First, measure what the service achieved over the last weeks, including the worst days. Second, ask product what users will tolerate: would a few minutes of errors a month cause churn, or would an hour go unnoticed? Third, choose the lowest target that satisfies the business, because every extra nine costs engineering effort and slows delivery.
Two cautions. Do not set an SLO higher than the dependencies underneath it can support; a service built on a single database instance cannot honestly promise more than that instance delivers. And do not copy another company’s number. Their users, architecture and tolerance are not yours.
Decisions and trade-offs
Tooling. Hand-written Prometheus rules give full control but are easy to get wrong across multiple windows. Generators such as Sloth or Pyrra take an SLO definition and produce the recording and alerting rules, which reduces mistakes. For teams already on a commercial observability platform, built-in SLO features may be quicker, at the cost of lock-in.
Granularity. One SLO per user journey usually beats one per microservice. Per-service objectives multiply quickly and rarely map to what customers feel.
When not to do this yet. If you have no reliable request metrics, no on-call process and a single service, a full SLO programme is premature. Start with basic monitoring and a small incident process, then return to SLOs when there is something to decide.
Common failure modes
- Vanity targets. A round number nobody can meet, so it is ignored.
- Internal-only indicators. Measuring queue depth instead of whether the user got a response.
- No owner. SLOs without a named person drift and die.
- No consequence. If missing the objective changes nothing, it is a report, not a control.
- Over-segmentation. Dozens of objectives with unclear priority.
We avoid these by keeping the first set small, writing the consequence down in the workshop, and recording an owner in the SLO file itself.
Worked example: a checkout service
For a checkout service, the user journey is “customer submits an order and gets a confirmation”. Candidate indicators: the proportion of order requests that return a non-error response, and the proportion that complete within a latency threshold agreed with product. Both are measured at the load balancer or ingress, because that is closest to what customers experience.
Exclusions matter. Health checks, internal test traffic and requests rejected for client mistakes should not count against the objective. Write those rules next to the SLI definition so everyone computes it the same way.
Reviewing objectives
Objectives are not carved in stone. We schedule a review after the first full window and then each quarter: was the target too loose to be useful, too tight to be affordable, or just right? Changes go through the same pull request process as the rest of the monitoring configuration, with the reason recorded.
Delivery and handover
You receive a short SLO document per critical journey, the Prometheus rules and Grafana dashboards in your repository, and a one-page policy describing how budgets influence release decisions. If you run the practice yourselves, we train the owners and sit in on the first review. If you want ongoing help, SLO reviews fit naturally into a Managed DevOps engagement.
How to tell it is working
Look for behaviour, not charts. Are release and reliability trade-offs now discussed using the budget? Do incident reviews reference the objective? Has the number of alerts that fire without user impact fallen? Is the SLO list reviewed on schedule? If the answer to most is yes, the objectives are doing their job. To scope a workshop for your services, book a consultation.