What Is SRE? A Practical Guide to Site Reliability Engineering
DevOps / SRE Engineer & CEO, Cloud DevOps Solutions
Site reliability engineering (SRE) is a discipline that applies software engineering principles to infrastructure and operations. Google coined the term around 2003, and it has since become one of the most adopted approaches to running production systems at scale. If DevOps is a culture, SRE is one concrete way to implement it.
This guide covers the core concepts, practices and tooling behind SRE, and explains how it relates to DevOps, platform engineering and traditional operations.
SRE meaning and definition
A site reliability engineer treats operations as a software problem. Instead of manually configuring servers or reacting to alerts, SRE teams write code to automate infrastructure, define reliability targets in measurable terms, and use data to decide where to invest engineering effort.
The key idea: reliability is a feature, and it competes for engineering time the same way any other feature does. SRE gives teams a framework to balance reliability work against feature development using error budgets.
SRE vs DevOps: what is the difference?
DevOps and SRE are complementary, not competing. DevOps is a set of cultural practices and principles (shared ownership, automation, continuous delivery, learning from failure). SRE is a specific role and framework that implements many of those principles with concrete practices.
| Aspect | DevOps | SRE |
|---|---|---|
| Origin | Industry movement, multiple contributors | Google, documented in the SRE book |
| Focus | Culture, collaboration, delivery speed | Reliability, measurable targets, engineering |
| Key metric | Deployment frequency, lead time | Error budget, SLO attainment |
| Automation goal | Automate the delivery pipeline | Automate away toil (repetitive ops work) |
| On-call | Varies by organisation | Structured, with caps on interrupt-driven work |
| Failure response | Blameless culture | Blameless postmortems with action items |
In practice, many organisations use both: DevOps principles guide the culture, and SRE practices provide the engineering rigour for running production. A team can adopt SRE without a dedicated SRE title, and a DevOps engineer can apply SRE practices to their work.
SLIs, SLOs and SLAs: measuring reliability
SRE makes reliability measurable through three related concepts.
Service Level Indicators (SLIs)
An SLI is a quantitative measure of some aspect of the service. Common SLIs include:
- Availability: the proportion of requests that succeed (HTTP 5xx excluded)
- Latency: the proportion of requests served within a threshold (for example, p99 below 300ms)
- Throughput: requests per second the service handles without degradation
- Error rate: the proportion of requests that return an error
- Freshness: for data pipelines, how recently the output was updated
Good SLIs are chosen from the user’s perspective. An internal health check passing is not the same as a user being able to complete a checkout.
Service Level Objectives (SLOs)
An SLO is a target for an SLI over a rolling window. For example: “99.9% of API requests return a non-error response within 300ms, measured over 30 days.” That 99.9% is the SLO. The remaining 0.1% is the error budget.
SLOs should be:
- Based on user expectations, not infrastructure capability
- Achievable, because an SLO you cannot meet erodes trust
- Reviewed quarterly, because user expectations and system capabilities change
Service Level Agreements (SLAs)
An SLA is a contract between a provider and a customer that includes consequences (refunds, credits) for missing the target. SLOs are internal engineering targets. SLAs are external commitments. Your SLO should always be stricter than your SLA, giving you a buffer before contractual penalties apply.
Error budgets: balancing reliability and velocity
The error budget is the most powerful concept in SRE. If your SLO is 99.9% availability over 30 days, you have a budget of roughly 43 minutes of downtime. As long as you have budget remaining, you can ship features, run experiments and take risks. When the budget is exhausted, the team shifts focus to reliability work.
Error budgets solve the eternal tension between “ship faster” and “do not break things.” They replace arguments with data:
- Budget remaining: prioritise feature development and deploys
- Budget low: slow down, add testing, improve rollback
- Budget exhausted: freeze non-critical changes, focus on reliability improvements
This framework aligns product and engineering teams around the same numbers. Product managers understand that pushing a risky release costs error budget that could block future releases.
Error budget policies
An error budget policy documents what happens at different budget levels. For example:
- Above 50% remaining: normal development velocity
- 25% to 50% remaining: require rollback plans for all deploys
- Below 25% remaining: production freeze except for reliability fixes
- Budget exhausted: all engineering effort goes to reliability until the budget recovers
The policy should be agreed in advance by engineering leadership and product, so decisions during an incident are not political.
Toil reduction
Toil is manual, repetitive, automatable operational work that scales linearly with service growth. Restarting a pod when it crashes is toil. Investigating why it crashes and fixing the root cause is engineering.
Google’s guideline is that SRE teams should spend no more than 50% of their time on toil, with the rest going to engineering projects that reduce future toil. Common toil reduction targets include:
- Manual deployments: replace with CI/CD pipelines and GitOps workflows
- Certificate renewals: automate with cert-manager or AWS Certificate Manager
- Scaling responses: replace manual scaling with HPA, KEDA or autoscaling groups
- Log-in to check status: replace with dashboards and alerting
- Ticket-driven access grants: replace with self-service RBAC and time-bound access
- Configuration changes: codify with Terraform, Ansible or Helm values
Track toil in your ticketing system. If you cannot measure it, you cannot prove it is shrinking.
Incident management
Incidents are inevitable. SRE provides a structured approach to handling them.
Incident response process
- Detect: monitoring and alerting catch the problem before users report it
- Triage: determine severity, assign an incident commander
- Mitigate: restore service first, investigate root cause second
- Resolve: apply the fix, verify recovery, close the incident
- Learn: run a postmortem, track action items
The incident commander coordinates communication, delegates investigation and makes decisions about mitigation. They do not need to be the most senior engineer; they need to stay calm and keep the response organised.
For more on structuring this process, see our SRE and Incident Response service.
Severity levels
Define severity levels before an incident happens:
| Level | Description | Response time | Example |
|---|---|---|---|
| SEV1 | Service down, all users affected | Immediate, all-hands | Payment processing unavailable |
| SEV2 | Significant degradation, many users affected | Within 15 minutes | API latency above 5s for 30% of requests |
| SEV3 | Minor degradation, limited impact | Within 1 hour | One region returning elevated errors |
| SEV4 | Cosmetic or low-impact issue | Next business day | Dashboard showing stale data |
Communication during incidents
Use a dedicated incident channel (Slack, Teams). Post updates at regular intervals even when there is no new information. Stakeholders tolerate uncertainty better than silence.
Blameless postmortems
A postmortem analyses an incident to prevent recurrence. The blameless part means the review focuses on systems and processes, not on who made the mistake. The goal is to understand why the system allowed the failure, not to find someone to blame.
A postmortem document should include:
- Summary: what happened, when, how long it lasted, who was affected
- Timeline: minute-by-minute sequence of events, detection and response
- Root cause: the underlying technical or process failure
- Contributing factors: what made the incident worse or harder to detect
- What went well: detection, response or mitigation that worked
- Action items: specific, assigned, with deadlines
Action items must be tracked and completed. A postmortem without follow-through is paperwork, not learning. Review postmortem action items in weekly engineering standups until they are closed.
On-call practices
On-call is part of SRE, but it must be sustainable.
On-call guidelines
- Rotation size: at least eight engineers in the rotation so no one is on-call more than once every eight weeks in primary
- Interrupt budget: cap on-call interrupt work at 25% of an engineer’s total time
- Escalation paths: clear escalation to a secondary on-call and then to management
- Runbooks: documented procedures for every alert so on-call engineers are not debugging from scratch at 3am
- Compensation: on-call should be compensated with time off in lieu or pay, not treated as a free add-on to the role
- Handoff process: outgoing on-call briefs incoming on-call on current issues
If your on-call is regularly paging engineers for non-actionable alerts, the alert quality is the problem. Every alert should be actionable, urgent and about a user-facing symptom. Delete the rest.
Observability: metrics, logs and traces
You cannot manage reliability without observing the system. SRE relies on three pillars of observability.
Metrics
Time-series data about system behaviour: request rate, error rate, latency percentiles, CPU, memory, queue depth. Metrics are cheap to store, fast to query and the foundation for alerting and SLO tracking.
Tools: Prometheus, Grafana, Datadog, CloudWatch.
See our Monitoring and Observability service for implementation details.
Logs
Structured, timestamped records of events. Logs answer “what happened” when metrics tell you something is wrong. Centralise logs so engineers do not SSH into machines to read them.
Tools: Elasticsearch, Loki, CloudWatch Logs, Fluentd, OpenTelemetry.
See our Logging and Tracing service for architecture patterns.
Traces
Distributed traces follow a request across services, showing where time is spent and where failures originate. Essential for microservices architectures where a single user request touches dozens of services.
Tools: Jaeger, Tempo, OpenTelemetry, AWS X-Ray.
The golden signals
Google’s SRE book identifies four golden signals to monitor for any service:
- Latency: how long requests take (distinguish successful from failed)
- Traffic: demand on the system (requests per second, sessions)
- Errors: the rate of failed requests (explicit errors and requests that succeed but return wrong data)
- Saturation: how full the service is (CPU, memory, disk, queue depth)
If you monitor nothing else, monitor these four.
Capacity planning
SRE teams plan capacity to ensure services can handle expected and unexpected load without degradation.
Capacity planning involves:
- Load testing: simulate production traffic patterns to find breaking points
- Growth modelling: project traffic growth from historical data and business forecasts
- Headroom: maintain enough spare capacity to handle a traffic spike or the loss of one availability zone
- Cost awareness: over-provisioning wastes money, under-provisioning causes outages. Right-size based on data, not fear
Review capacity quarterly. Cloud autoscaling helps, but it has limits: cold-start times, API rate limits and dependencies that do not scale automatically (databases, third-party APIs).
How to start an SRE practice
You do not need to hire a team of 20 SREs to start. Begin with the practices that deliver the most value for your current pain points.
Phase 1: measure (weeks 1 to 4)
- Define SLIs for your most critical user journeys
- Set initial SLOs based on current performance (do not aim for perfection)
- Set up dashboards showing SLO attainment and error budget burn rate
- Start tracking incidents and toil in your existing ticketing system
Phase 2: respond (weeks 5 to 8)
- Write runbooks for your top 10 most common alerts
- Establish an on-call rotation with escalation paths
- Run your first blameless postmortem for the next significant incident
- Implement alerting on SLO burn rate instead of raw thresholds
Phase 3: automate (weeks 9 to 16)
- Identify your top three sources of toil and automate them
- Implement CI/CD if you have not already
- Add canary deployments or progressive rollouts to reduce blast radius
- Introduce error budget policies agreed with product stakeholders
Phase 4: mature (ongoing)
- Expand SLOs to cover more services
- Run regular production readiness reviews for new services
- Track toil reduction over time and report it to leadership
- Adopt chaos engineering to proactively find weaknesses
Common SRE mistakes
- Setting SLOs at 100%: nothing is 100% reliable. An impossible target makes the framework useless.
- Alerting on symptoms instead of user impact: CPU at 80% is not an incident if users are unaffected.
- Treating postmortems as blame sessions: people stop reporting honestly and the organisation stops learning.
- Ignoring toil: if every sprint is consumed by operational work, you never invest in reducing it.
- SRE as a rename for ops: if the team has no time for engineering projects, it is an operations team with a new name.
SRE tooling overview
| Category | Tools |
|---|---|
| Monitoring | Prometheus, Grafana, Datadog, CloudWatch |
| Logging | Elasticsearch, Loki, Fluentd, OpenTelemetry |
| Tracing | Jaeger, Tempo, AWS X-Ray, OpenTelemetry |
| Incident management | PagerDuty, Opsgenie, Grafana OnCall |
| SLO tracking | Sloth, OpenSLO, Datadog SLOs, Nobl9 |
| Chaos engineering | Litmus, Gremlin, AWS Fault Injection Simulator |
| IaC | Terraform, Pulumi, Ansible |
When to bring in outside help
Building an SRE practice is straightforward in theory but hard in practice. Common situations where external expertise accelerates the process:
- You are setting up monitoring and alerting from scratch and want to avoid the trial-and-error phase
- Your team is drowning in toil and needs someone to build the automation while they keep the lights on
- You are preparing for a compliance audit and need to demonstrate incident management processes
- You want to define SLOs but are not sure where to set them or how to instrument your services
We help teams build SRE practices that match their size and maturity. See our SRE and Incident Response service or book a working session to discuss your situation.
Key takeaways
Site reliability engineering gives teams a data-driven framework for running production systems. The core ideas are simple: measure reliability with SLIs and SLOs, use error budgets to balance velocity against stability, reduce toil through automation, and learn from every incident through blameless postmortems.
You do not need to adopt everything at once. Start by measuring what matters to your users, respond to incidents with structure, and automate the work that keeps your team from engineering. The practices compound over time, and the result is a team that ships faster because the systems underneath are more reliable.
Related service
Need hands-on help? Our certified engineers deliver this as a managed service.
SRE & Incident Response