A routed alert still needs a person
Everything upstream of this page, the metrics, rules and routing, ends in one hope: that a capable human sees the alert quickly and knows it is theirs. In practice this fails for unglamorous reasons. The schedule was last updated in spring. The person on the rota left the company. The only phone number belongs to someone on a flight. Or one engineer has carried the pager for six weeks because nobody built a rotation.
On-call is a human system with a technical interface. In the monitoring and observability service we set up that interface and help you design the rotation behind it. If you would rather have an outside team own the rota, our Managed DevOps service includes that, and so does SRE and incident response for more reliability-focused engagements.
What good looks like
- Always a named owner. At any hour there is a primary and a backup, and the tool knows who.
- Fair and sustainable. Load is shared, quiet weeks are protected, and burnout is treated as a risk.
- Escalation that works. If the primary does not acknowledge, the next level is paged automatically.
- Easy overrides. Swapping a shift takes a minute, not a ticket.
- Context in the page. Severity, service, dashboard and runbook, as described in alert routing.
- Handoffs. A short written summary at shift change.
Choosing the tool
| Option | Fit | Caution |
|---|---|---|
| Grafana OnCall | Teams already on Grafana who want schedules close to dashboards | Verify current licensing, support and maintenance status for your chosen edition |
| PagerDuty | Larger teams, complex escalation, incident workflow | Per-user licensing |
| Opsgenie | Atlassian-centred teams | Check the vendor’s current product plans |
| Slack plus phone tree | Very small teams, low volume | Weak guarantees, limited escalation |
We have no preference that outranks your needs. Vendors change plans, so we confirm the current state of whatever we recommend at the time of the engagement, and keep integration through Alertmanager webhooks standard so that switching later is a configuration change.
How it works in practice
Wiring Alertmanager to an on-call webhook
Grafana OnCall and similar tools provide an Alertmanager integration URL. The receiver is straightforward:
receivers:
- name: oncall-platform
webhook_configs:
- url_file: /etc/alertmanager/secrets/oncall-platform-url
send_resolved: true
max_alerts: 20
The route that feeds it matches the critical severity, as laid out in the Alertmanager routing page. The URL contains a token, so it lives in a secret file, not in the repository.
Schedules and escalation as code
Where a Terraform provider is available, schedules and chains can be declared. This illustrates the shape for Grafana OnCall via its provider; check the provider documentation for the exact resource names and arguments in your version:
resource "grafana_oncall_schedule" "platform_primary" {
name = "platform-primary"
type = "calendar"
time_zone = "Europe/London"
}
resource "grafana_oncall_escalation_chain" "platform" {
name = "platform-critical"
}
resource "grafana_oncall_escalation" "notify_primary" {
escalation_chain_id = grafana_oncall_escalation_chain.platform.id
type = "notify_on_call_from_schedule"
notify_on_call_from_schedule = grafana_oncall_schedule.platform_primary.id
position = 0
}
Day-to-day shift swaps stay in the UI or calendar, because forcing every swap through a pull request is a quick way to make people avoid the system.
Designing the rotation
Primary, secondary and escalation
The primary acknowledges and works the alert. The secondary is the safety net if the primary does not respond within the agreed interval, typically a matter of minutes for critical alerts. A final level reaches an engineering lead. We write the intervals down and test them.
Time zones and coverage
A team in one time zone covering nights has a hard problem. Options include follow-the-sun with an overseas group, restricting paging to the most severe alerts out of hours, or using an external provider. Be honest about what your headcount supports. A rotation that pages a three person team every third night will fail regardless of tooling.
Handoffs and learning
At shift change the outgoing engineer posts what fired, what is still fragile, and any silences in place. A short weekly review looks at pages that were avoidable. Those become alert tuning or automation tasks, which closes the loop with the runbooks.
Decisions and trade-offs
Outsource or own. Owning on-call builds knowledge in your team but demands people and discipline. Outsourcing gives coverage faster, with less context. Many teams split: developers cover application alerts, platform specialists cover infrastructure.
Developer on-call. Teams that run what they build tend to fix root causes faster. It needs training and fair compensation.
Too many tools. Slack, a pager, a status page and a ticketing system each need integration. We keep the chain short.
Failure modes
- Stale schedules with departed staff.
- Notification settings that bypass nothing: engineers’ phones set to silent without the tool’s override.
- Only one contact method configured.
- Escalation to nowhere, a chain whose final step is an unmonitored mailbox.
- No test pages, so the first real escalation is the first run.
Testing the chain before you need it
An escalation chain that has never fired is a hypothesis. We run a structured test before handover: a synthetic critical alert is sent through Alertmanager and must reach the primary on every configured channel, including push notification, SMS and phone call where your tool supports them. We then deliberately leave it unacknowledged to confirm the secondary is paged at the agreed interval, and again to confirm the final level is reached. Each step is timed and recorded.
We repeat a shorter version after any change to schedules or contacts, and recommend a quarterly drill for the same reason fire drills exist. The checks that catch problems are simple ones: a phone on silent, an expired mobile number, a departed engineer still listed, a webhook URL rotated without updating the receiver. Finding these on a quiet afternoon is cheap. Finding them at three in the morning during a real incident is not.
Delivery and handover
You receive configured schedules, escalation chains, integrations with routing by label, onboarding instructions for each engineer (install the app, set contact methods and override rules), and a short on-call guide. We run a test page through every level before sign-off.
How to tell it is working
Track time to acknowledge, the proportion of pages that escalate beyond the primary, pages per engineer per week, after-hours pages, and engineer feedback on load. If acknowledgement is quick and nobody dreads their week, the integration is working. To set up on-call properly, book a consultation.