The problem runbooks solve
Every startup has knowledge that lives in one person’s head: how to restart the queue consumers without losing jobs, which migration must run before a deploy, why the cache must be flushed after a certain config change. It works until that person is asleep, on holiday or has left. Then a routine failure becomes a long outage because the fix is a guess.
Runbooks turn private knowledge into shared, tested procedure. They are the backbone of a safe on-call rotation and a key part of our Managed DevOps service, because no outside team can be effective at 3am without them.
What good looks like
A strong runbook has these properties:
- Findable. Linked directly from the alert that triggers it.
- Short. Ten steps, not ten pages. Background lives elsewhere.
- Exact. Commands can be copied and run. No “check the logs” without saying which logs and what to look for.
- Verifiable. It says how to confirm the problem is resolved.
- Safe. It warns about destructive steps and includes rollback.
- Owned. A named person or team keeps it current.
- Tested. Someone other than the author has followed it successfully.
The template we use
Consistency matters more than perfection, because a familiar layout reduces cognitive load when stressed. Ours looks like this:
Runbook title: Checkout API returning 5xx
Owner: platform team Last reviewed: 2026-09-14 Severity when firing: P1
## When to use
Alert CheckoutHighErrorRate is firing, or customers report failed payments.
## Impact
Customers cannot complete purchases. Revenue is affected.
## Quick checks (2 minutes)
1. Recent deploy? `kubectl rollout history deploy/checkout -n production`
2. Pod health: `kubectl get pods -n production -l app=checkout`
3. Dependency status: payment provider status page, database dashboard.
## Actions
1. If a deploy happened in the last hour, roll back:
`kubectl rollout undo deploy/checkout -n production`
2. If pods are crash-looping, read `kubectl logs --previous` and escalate to dev on-call.
3. If the database connection count is at its limit, follow the "DB connections" runbook.
## Verify
Error ratio below 1% for 10 minutes on the Checkout SLO dashboard.
## Escalate
Platform lead, then engineering manager. Page the payments developer for code faults.
In your repository the title line is simply the file heading. Every field has a purpose, and the “Quick checks” section is deliberately time-boxed to stop aimless wandering.
How we build the set
Start from reality, not theory
We do not write a hundred speculative documents. We start with the alerts that page, the incidents of the last year, and the components whose failure would hurt most. A post-mortem is a free runbook outline: the timeline already lists what was checked and what fixed it.
Components that nearly always need one
- Application deploy, rollback and hotfix.
- Database failover, restore from backup and connection saturation.
- Cache and queue problems.
- Kubernetes node loss, pod eviction storms and failed upgrades (see cluster operations).
- Certificate and DNS expiry.
- Secret and credential rotation.
- CI/CD outage or stuck pipeline (see pipeline ownership).
- Cloud provider incident and regional failover.
- Backup verification and disaster recovery, which pairs with our backup and disaster recovery service.
Link alerts to procedures
An alert without a runbook forces improvisation. In Prometheus, the link rides in an annotation, as we show in the monitoring page and our alerting rules guide:
annotations:
summary: "Checkout 5xx ratio above 2% for 10 minutes"
runbook_url: "https://github.com/your-org/runbooks/blob/main/checkout-5xx.md"
Paging systems surface that link in the notification itself, so the engineer sees the procedure within seconds.
From documents to automation
A runbook with ten manual commands is a candidate for a script. The sequence we follow is document, then test, then automate the parts that are safe and repetitive. Examples include a script that drains a node, a pipeline job that restarts a service with a health check, or a button in an automation tool such as Rundeck. Automation is not a substitute for the document. The runbook explains when to run the automation and what to do when it fails.
Be careful with automating destructive steps. A script that deletes volumes needs confirmation prompts and dry runs, and it should never run on an alert alone.
Testing runbooks
An untested runbook is a hypothesis. We validate in three ways:
- Peer walkthrough. A second engineer follows it cold and notes every point of confusion.
- Game day. We inject a controlled failure in non-production, such as killing a database replica, and have on-call resolve it using only the runbook.
- Real incident review. After a live incident, we check where reality diverged from the document and update it that week.
Trade-offs and when not to over-document
Documentation decays. Every page is a maintenance liability. We write runbooks for failures that happen or would be severe, not for everything. Stale documentation is worse than none because it breeds false confidence.
Where to host. A wiki is easy to edit but may be unreachable during an outage if it shares infrastructure. Git is durable and reviewable. We often keep source in Git and mirror to a wiki.
Judgement over scripts. Some failures need diagnosis that a procedure cannot capture. Runbooks give structure for the common cases and a clear escalation route for the rest.
When it is overkill. A single developer running one small service may need only a README with deploy and restore notes.
Common failure modes
- Wall-of-text documents. No one reads them at 3am.
- Placeholder commands. Steps that say “run the script” without a path.
- Orphaned owners. The author left; nobody reviews.
- Stale credentials and links. Dead links in alerts erode trust in all of them.
- Access gaps. The runbook says “log in to the console” but the on-call account lacks permission. Drills expose this.
Delivery and handover
We deliver the runbook repository, the template, the alert links and a review schedule. Ownership is recorded for each document. Updates after incidents are part of the routine, and new components added to your platform get a runbook as part of their rollout. The set is yours, in your repository, readable without us.
How to tell it is working
Measure the share of paging alerts with a runbook (target all of them), time from acknowledgement to first correct action, number of escalations that were unnecessary, and how often drills expose gaps. If a new on-call engineer can resolve a common incident on their first shift, the runbooks are working. To scope a documentation set, talk to us.