Setting Up a Sustainable On-Call Rotation

Why on-call goes wrong

Most on-call problems come from design, not from the tool. A single “DevOps person” is paged for everything. Alerts fire for conditions nobody can act on. A rotation exists on paper but one senior engineer quietly covers every bad night. Within a year the team is tired, resentful, or a key person leaves.

A well-built rotation is the opposite: predictable, shared, and rarely disturbing. This page covers how we build one as part of our SRE and incident response service, including the awkward trade-offs for small teams.

What good looks like

• Shared fairly. Load is spread across everyone qualified, with swaps and overrides easy to arrange.
• Reachable. A page produces a phone call or push that breaks through do-not-disturb, and the schedule shows who is on right now.
• Backed up. Every page has an escalation: if the primary does not acknowledge within a set time, the secondary is paged, then a manager.
• Equipped. The on-call engineer has access, runbooks and authority to act, including rollback.
• Bounded. Alert volume is capped by process. If a shift regularly exceeds a sensible number of interrupting pages, that triggers work on alerts, not more heroics.
• Reviewed. Handover notes and a weekly look at the previous shift’s pages.

Choosing the paging tool

The three tools we implement most are PagerDuty, OpsGenie and Grafana OnCall. All can model schedules, escalation policies and routing from monitoring tools.

Vendors change plans and roadmaps, so we verify the current state at the time of selection rather than relying on a comparison written earlier. If your monitoring already lives in Grafana, the lowest-friction option is often Grafana OnCall. If you need rich reporting and many integrations, PagerDuty is hard to beat.

Designing the rotation

Small teams (two to four engineers)

Weekly rotations of one person are exhausting at this size. Options that work better:

• Split the day and night, so nobody carries both.
• Use a primary-and-secondary model where the secondary only answers if the primary misses.
• Reduce what can page at night to a short list of truly severe conditions, and let everything else wait until morning.
• Consider outsourced coverage for out-of-hours.

Larger teams

Weekly shifts with a primary and a secondary, plus a separate rotation for incident commander if you have a formal incident process, keep load balanced. Multi-region teams can add follow-the-sun handoffs.

Shift hygiene

Handover happens at a fixed time with a written note: open issues, risky changes planned, flaky alerts to watch. Swaps are recorded in the tool, not agreed in chat only, so the schedule remains the single source of truth.

Alert routing and escalation

A page must reach someone who can act. We map each service to an owning team in the paging tool and route from Alertmanager or cloud alarms using labels:

route:
  receiver: default-ticket
  group_by: [alertname, service]
  routes:
    - matchers: [severity="page", team="platform"]
      receiver: pagerduty-platform
    - matchers: [severity="page", team="payments"]
      receiver: pagerduty-payments
    - matchers: [severity="ticket"]
      receiver: jira-ticket

receivers:
  - name: pagerduty-platform
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-platform-key

The key is read from a mounted secret file rather than written into configuration. The escalation policy then lives in the paging tool: for example, notify the primary immediately, escalate to the secondary if unacknowledged after a few minutes, then to the engineering manager. Exact timings are agreed with you. The wider escalation structure is described in escalation paths and communication.

Common failure modes

• The hero pattern. One person quietly takes every page. Fix it with overrides, visible load reports and a manager who checks.
• Unreachable pages. Silent mode, wrong phone number, notification rules never tested. We test every person’s contact path.
• Pager for everything. Low-value alerts train people to ignore the pager. Pair rotation setup with an alert quality audit.
• No authority. The on-call engineer cannot roll back without waiting for approval. Agree pre-authorised actions in advance.
• No time off after a bad night. Agree a recovery rule, such as a late start after a night with several interruptions.

Handover and shift hygiene

A rotation is only as safe as its handoff. At each change, the outgoing engineer posts a short note: open incidents, alerts that were silenced and why, risky changes in flight, and anything flaky to watch. The incoming engineer acknowledges it. This takes five minutes and prevents the classic failure where a silenced alert quietly outlives the reason for silencing it.

Compensation and time off

Out-of-hours duty is real work. Whether you pay a stipend, grant time off in lieu, or both is a decision for your leadership and local employment rules, and we do not prescribe it. We do recommend writing it down before the first rotation starts. Teams that leave this vague tend to lose their best engineers first.

Reviewing load

Each month, look at pages per shift, night pages, and who carried the heaviest weeks. Uneven load is a rota design problem or an alerting problem, and both are fixable.

When not to run your own rotation

If you have one or two engineers, or your product has no real out-of-hours need, a formal rotation may cost more than it returns. Business-hours support with a clear “best effort” policy for nights can be honest and sufficient. If you do need round-the-clock cover and cannot staff it fairly, an external team under Managed DevOps is often better than burning out a handful of developers.

Delivery and handover

You receive the configured paging tool, schedules, escalation policies and routing rules (as Terraform where the provider supports it), a one-page on-call guide, a handover note template and a tested contact list. We stay alongside for the first rotations to adjust load.

How to tell it is working

Track pages per shift, out-of-hours pages, acknowledgement time, and the proportion of pages that needed action. Add a human measure: ask on-call engineers each quarter how the shift felt. Falling page volume with steady or improving recovery time is the signal that routing, alerts and runbooks are working together. To plan yours, book a consultation.

References

• Being On-Call, Google SRE book
• On-Call, Google SRE workbook
• Grafana OnCall documentation, Grafana Labs