The cost of improvising at 3am
When an alert fires and there is no procedure, the on-call engineer improvises: searching old chat threads, guessing which dashboard matters, and deciding alone whether a risky action is safe. Recovery time becomes a function of who happens to be on shift and how much of the system they remember.
An incident runbook library removes that variance. It does not make engineers unnecessary. It gives them a reliable starting point so judgement is spent on the new part of the problem rather than rediscovering the old part. It is a core piece of our SRE and incident response service, and it is what makes a new on-call rotation viable.
What “every tier-1 service and alert” means in practice
Coverage is defined by two lists, and both must be complete.
- Tier-1 services. Anything whose failure stops customers using the product, loses revenue or risks data. Each needs runbooks for its common failure modes: bad deploy, dependency down, saturation, data issue.
- Paging alerts. Every alert that can wake someone must link to a procedure. If an alert has no runbook, either write one or ask whether the alert should page at all.
This two-list rule is easy to audit. A script can read your alert rules and report any paging rule without a runbook_url, which becomes a standing check in CI.
The template
Consistency reduces cognitive load. Ours:
# Orders API latency high
Owner: platform team | Reviewed: 2026-09-20 | Page severity: P2
## Trigger
Alert OrdersLatencyBurn is firing, or support reports slow order pages.
## Impact
Customers can place orders but pages are slow. Abandonment risk.
## Check first (3 minutes)
1. Recent deploy? `kubectl rollout history deploy/orders -n prod`
2. Saturation: CPU, memory and connection pool panels on the Orders dashboard.
3. Dependencies: database latency panel, payment provider status.
## Actions
1. Deploy in the last hour: `kubectl rollout undo deploy/orders -n prod`
2. Pool saturated: scale out `kubectl scale deploy/orders -n prod --replicas=<current+2>`
3. Database slow: follow the "Database load" runbook.
## Verify recovery
p95 latency under the SLO threshold for 15 minutes.
## If this fails
Escalate to the payments developer on call, then the engineering manager.
Notice what is absent: long explanations, theory and optional reading. Those belong in design docs.
Building from evidence
We do not draft a hundred speculative pages. The inputs are:
- The last year of incidents and near misses. Each post-mortem contains a timeline of what was checked and what fixed it, which is a free first draft.
- Alert history sorted by frequency and by whether a human took action.
- Interviews with the engineers who “just know” how things break. Capturing this is half the value.
- The architecture: queues, caches, databases, certificates, DNS and third-party dependencies each have predictable failure modes.
Typical runbooks in a tier-1 library cover deploy and rollback, database failover and restore, connection and queue saturation, certificate and DNS expiry, node loss and pod eviction on Kubernetes, credential rotation, and cloud region problems. For restore and disaster scenarios, they link to the procedures from our backup and disaster recovery service.
Linking runbooks to alerts
The runbook arrives with the page. In Prometheus, an annotation carries the URL, and the paging tool shows it in the notification:
- alert: OrdersLatencyBurn
expr: slo:orders_latency:burn_rate1h > 6 and slo:orders_latency:burn_rate5m > 6
labels:
severity: page
annotations:
summary: "Orders latency is burning error budget at 6x"
runbook_url: "https://github.com/your-org/runbooks/blob/main/orders-latency.md"
A CI check that fails when a rule with severity: page lacks runbook_url keeps the library honest as new alerts are added. See our alerting rules guide for more patterns.
Testing so the steps are trustworthy
An untested procedure has unknown quality. We validate in layers:
- Cold read. A teammate who did not write it follows it and notes each hesitation.
- Access check. Confirm the on-call role actually has permission for every command. This catches more problems than anything else.
- Drill. Inject a controlled failure in a safe environment and resolve it using only the runbook. This links directly to game days.
- Post-incident correction. After any real incident, update the document within the week.
Decisions and trade-offs
Runbook versus automation. If a procedure is ten safe commands, script it. But do not delete the runbook; it still explains the trigger, the verification and the failure path.
Depth versus maintenance. Each page is a liability. We write for failures that occur or would be severe, and avoid documenting hypothetical edge cases nobody has seen.
Where to host. A wiki is pleasant to edit but may be unreachable in an outage. Git is durable and reviewable. A common compromise is Git as source of truth with a rendered copy elsewhere.
When this is overkill. A solo founder running a single small service needs a README with deploy and restore steps, not a library.
Common failure modes
- Wall-of-text pages nobody reads under stress.
- “Check the logs” with no location or search term.
- Authors leave and owners are never reassigned.
- Dead links in alerts erode trust in every runbook.
- Runbooks that assume admin rights the on-call role does not have.
Example: a small service-tier table
A short register tells everyone where effort goes. Tiers are a business judgement, so the owner of each service agrees them with us in a workshop.
| Tier | Meaning | Runbook expectation |
|---|---|---|
| 1 | Customer-facing or revenue-critical | Every paging alert and the common failure modes, drilled |
| 2 | Important internal or supporting service | Top failure modes, reviewed after incidents |
| 3 | Low impact, easy to rebuild | A README with deploy and restore steps |
Writing for the engineer who did not build it
The test of a good step is whether someone from another team could follow it. We avoid team jargon, name the exact dashboard and cluster context, and state what a healthy result looks like next to each command, so the reader can tell success from silence.
Delivery and handover
You receive the repository of runbooks, the template, the CI check for alert coverage, an owner register and a review schedule. For new services, the rollout checklist includes a runbook before it goes live. Everything is plain Markdown in your own repository, readable without us.
How to tell it is working
Measure the share of paging alerts with a linked runbook (aim for all), time from acknowledgement to first correct action, unnecessary escalations, and gaps found per drill. The best test is simple: can a newer on-call engineer resolve a common incident on their first shift? If so, the library is working. To scope one for your stack, talk to us.