Why runbooks fail when you need them
Most teams have documentation about their systems. Fewer have documentation designed for an emergency. The usual disaster recovery document describes the architecture in loving detail and skips the part that matters: what to type first. It assumes the reader knows which account, which region and which profile to use. It was written by the engineer who left last year, and it refers to a server that no longer exists.
A runbook is the opposite. It is an operational document that starts with a trigger and ends with a verified outcome. It is the written form of what you have decided in your recovery objectives, and a core deliverable of our backup and disaster recovery service.
What good looks like
A good runbook can be followed by a competent engineer who has never touched the service, with no one to ask. It is precise about order, honest about time, and clear about who decides. It also states when to stop and escalate. If a step can destroy data, the runbook says so, plainly, before the step.
The structure we use
Every service runbook follows the same layout, so people know where to look:
- Purpose and scope. What service, which scenarios, what it does not cover.
- Targets. The RTO and RPO for this service and its tier.
- Trigger and decision. Who may declare a disaster and the criteria, including the cost of failing over versus waiting.
- Roles and contacts. Incident lead, service owner, communications, with deputies.
- Prerequisites. Access, accounts, tools, keys, and where to find them safely.
- Recovery steps. Numbered, ordered, with commands and expected output.
- Verification. Concrete checks that prove the service works for users.
- Rollback and failback. How to undo and how to return to the primary later.
- Communication. Status page text and stakeholder updates.
- History. Date and result of every drill and real use.
Example of a good step
A weak step reads “Restore the database.” A usable step reads like this:
### Step 4: Restore the orders database (est. 25 to 45 min)
Owner: database on-call. Do not start before step 3 confirms the DR VPC exists.
1. Find the latest recovery point in the DR vault:
aws backup list-recovery-points-by-backup-vault \
--backup-vault-name dr-vault --region eu-west-2 --max-results 5
2. Start the restore using the saved metadata file in the repo at dr/orders-restore.json.
3. Wait until the new instance status is "available".
Expected result: instance orders-dr is available, data no older than the RPO.
If the newest recovery point is older than 1 hour: stop and tell the incident lead.
Verify: run dr/checks/orders.sql and confirm row count is within 1 percent of last known value.
The details, such as the exact command, the 1 hour threshold and the expected time, come from your environment and your drills. The point is the shape: owner, precondition, command, expected result, stop condition, verification.
Automate what repeats
Long pasted command sequences are a source of typos. Where a step is mechanical, we turn it into a script, a Terraform workspace or a pipeline job and have the runbook call it. Infrastructure for the recovery site should come from code, with the runbook covering decisions, data restore and verification. Teams that use GitOps for clusters often recover applications by pointing the controller at the recovery cluster, which turns many runbook steps into a single, reviewable action. Stateful pieces, such as persistent volumes, still need explicit restore steps.
Decisions and trade-offs
Runbook or automation. Full automation is faster and less error prone, but harder to build and to trust for rare events. We usually automate the mechanical 80 percent and keep humans at the decision points.
Where to store it. In Git next to the code gives review and history. It must also be accessible if your main hosting is unavailable, so keep an offline export or a copy in a separate system.
Granularity. One runbook per service plus a top-level sequence for a full-site event. The sequence says “recover identity, then data, then application tier” and links out.
When not to write one yet. A service that does not exist in recovery terms, for example a stateless tool you can redeploy from a pipeline, needs a two-line note, not a runbook.
Common failure modes
- Stale commands and ARNs. Resource names change, the document does not. Reduce this by parameterising and storing identifiers in code.
- Hidden access dependencies. The only person with the right role is on holiday.
- No decision criteria. Everyone waits for someone else to call it.
- Only the author has read it. Unreviewed runbooks hide assumptions.
- No failback section. You recover, then discover that returning to normal is its own project.
- Circular dependency. The runbook lives in a tool that needs the recovered service to open.
Keeping runbooks accurate over time
A runbook starts decaying the day it is published. Infrastructure changes, people leave and tools are renamed. We build maintenance into how the runbooks are produced rather than relying on goodwill.
- Versioned with the code. Runbooks live in the same repository as the infrastructure, so a pull request that changes a recovery path can update the runbook in the same review.
- Ownership by name. Every runbook has an owner and a review date, and the owner is notified when the systems it describes change materially.
- Executable where possible. Steps that can be scripted become tested scripts, and the runbook calls them. A script that runs weekly in a test is far less likely to rot than a paragraph of instructions.
- Exercise-driven edits. Each game day or restore test ends with a short list of corrections, applied immediately.
- Offline copy. A printable or separately hosted copy exists, so the runbook is readable when your wiki or cloud console is not.
A good test of a runbook is handing it to an engineer who did not write it and watching where they hesitate. Every hesitation is a missing detail. We run this review before handover and again after the first live exercise, with results feeding your business continuity plan.
Delivery and handover
We deliver the runbooks in your repository or documentation system, using the standard layout, together with the scripts they reference. We run a tabletop session where the team reads the runbook aloud and challenges it, followed by a live drill, then update the text with everything we learn. The business continuity plan links to each runbook. For day-to-day operational runbooks we also cover this in the Managed DevOps service.
How to tell it is working
A new on-call engineer completes a drill using only the runbook. Each runbook shows a last tested date within the agreed interval. Drill findings are fixed within days, and measured recovery time in drills falls within the targets. To get your most critical service documented and tested, contact us.