The cost of knowledge that lives in one head
Every platform has a person who knows why the ingress has that odd annotation, which node pool must never be scaled to zero, and what to do when the CI deploy hangs. That person is a risk. They go on holiday, change teams or leave, and the first real incident afterwards is slower and more dangerous than it should be.
Kubernetes runbooks and operational documentation turn that knowledge into something a team can share. It is the last item in our Kubernetes consulting scope, and it is also what lets you take over a platform we built, or hand day-to-day operation to our Managed DevOps team with confidence. For the wider context, see what managed DevOps means.
Runbooks versus documentation
They are different artefacts and need different writing.
| Runbook | Operational documentation | |
|---|---|---|
| Purpose | Act during a specific failure or task | Understand how the platform works |
| Reader state | Stressed, short on time | Calm, learning |
| Length | One screen or two | As long as needed |
| Trigger | An alert or a scheduled task | Onboarding, design questions |
| Style | Imperative steps and commands | Explanation and diagrams |
Mixing them produces pages too long to follow at 3am and too thin to learn from.
What good looks like
- Every paging alert links directly to a runbook.
- Runbooks are consistent, so a reader always knows where to find the verification step.
- Commands are exact and safe to copy, with notes on what they change.
- The platform docs have a current architecture diagram, an access guide and a decision log.
- Someone other than the author has followed each critical runbook successfully.
- Pages have owners and a last-reviewed date.
The runbook template
We use one template for everything, in Markdown, so pages are quick to write and review:
# Runbook: Pods in CrashLoopBackOff
**Alert:** KubePodCrashLooping | **Severity:** warning | **Owner:** platform team
**Last reviewed:** 2026-10-08
## Impact
Which users or services are affected, and how urgent it is.
## Confirm
kubectl get pods -n <namespace> --field-selector=status.phase!=Running
kubectl describe pod <pod> -n <namespace> # check Last State and Events
kubectl logs <pod> -n <namespace> --previous
## Mitigate
1. If a recent deploy caused it: roll back in Git (revert the commit) and let the controller sync.
2. If OOMKilled: raise the memory limit through a pull request, do not edit live.
3. If a dependency is down: follow the runbook for that dependency.
## Verify
Pods Ready, restart count stable for 15 minutes, error-rate panel back to normal.
## Escalate
Platform on-call, then the owning team (see service catalogue).
Placeholders in angle brackets are filled in by the reader at the time. The important design choices: confirm before acting, prefer reversible actions, and say where to escalate.
Which runbooks we write first
We prioritise by blast radius and frequency, taking input from your alerts and past incidents. A typical first set:
- Pods crash looping, pending, or stuck terminating.
- Node not ready, node pool at capacity, disk pressure.
- Ingress 502, 503 and 504 errors, and certificate expiry.
- Failed or stuck deployment, and GitOps sync failures.
- DNS resolution problems inside the cluster.
- Persistent volume attach failures.
- Cluster upgrade procedure and rollback options. See upgrade planning.
- Access requests and break-glass procedure, linked to RBAC design.
- Restoring from backup, aligned with your disaster recovery plan.
These tie back to alerts defined in cluster monitoring and to the response process in SRE and incident response.
Operational documentation
The platform guide answers different questions: what is deployed, why, and how do I change it? We typically produce:
- Architecture overview with a diagram of network, identity and add-ons.
- Environment map: clusters, accounts, owners and purposes.
- Deployment path: how code gets to production, from commit to running pod.
- Access guide: how people get permissions and how to request elevated rights.
- Upgrade and maintenance calendar.
- Decision log: short records of why major choices were made.
- Onboarding checklist for new engineers, tested by having one follow it.
Diagrams written in Mermaid or similar text formats can live in the repository and be reviewed like code, which keeps them from drifting.
Testing documentation
Docs that nobody has tried are guesses. We test them in three ways:
- Cold read. An engineer who did not write the page follows it in a safe environment and notes every hesitation.
- Game day. We trigger a controlled failure (kill a node, expire a test certificate) and see whether the runbook leads to recovery.
- Post-incident update. After any real incident, the relevant page is corrected as part of closing it out.
Trade-offs and honest limits
- Not everything needs a runbook. If a failure is rare, low impact and easy to diagnose, a short note beats a formal procedure.
- Automation can replace some steps. If a runbook says “restart the pod”, consider whether a controller or alert-driven automation should do it. Keep the page to explain the automation.
- Tools matter less than ownership. A plain Markdown folder with owners beats a polished wiki nobody updates.
- Documentation is not a substitute for simplicity. If a procedure needs forty steps, the platform may need fixing.
- Application-level docs belong with the teams who own those services.
Common failure modes
- Runbooks that say “investigate the issue” without saying how.
- Commands that only work on the author’s machine or in one namespace.
- No link between alert and page, so nobody finds it.
- Documentation written once at project end and never touched.
- Secrets or credentials pasted into pages. Reference where they are stored instead.
Delivery and how to tell it works
You receive the runbook set and platform guide in your repository, linked from your alerts, with an owner table and a review schedule. We run a short walkthrough with your on-call engineers. It works when new on-call engineers resolve common incidents without calling the original builder, mean time to diagnose falls, and pages stay current because updating them is part of the incident process. If you want to see what a finished runbook looks like for your stack, get in touch.