Resilience you have not tested is an assumption
Most architecture diagrams promise resilience: multiple replicas, multi-zone clusters, automatic failover, retries. Few teams have watched those mechanisms actually work. The first real test is often a 3am outage, when you discover the failover needs a manual step, the retry logic amplifies the problem, or the alert that should have fired was never wired up.
Chaos engineering moves that discovery to a time of your choosing. It is not about breaking things for sport. It is the scientific method applied to production readiness, and it is part of the proactive side of our SRE and incident response service.
What chaos engineering is, and is not
It is a controlled experiment: form a hypothesis about steady-state behaviour, inject a realistic failure, observe, and learn. A good hypothesis sounds like: “If one of three replicas of the orders service is killed, user-facing error rate stays under the objective and the pod is replaced within a couple of minutes.”
It is not random destruction, load testing in disguise, or a stunt to impress executives. It is also not a substitute for fixing known problems. If you already know your database has no replica, you do not need an experiment to prove it.
Are you ready?
We check four prerequisites before planning any injection:
- Observability. You can see error rate, latency and saturation for the affected service. Without this you cannot tell whether the experiment hurt.
- An incident process. Someone is in charge, and there is a way to abort. See incident roles.
- Basic redundancy. Testing a system with a single point of failure only confirms the obvious.
- Runbooks. There are written procedures to follow, which the game day then tests. See the runbook library.
If these are missing, we recommend fixing them first. A tabletop discussion is a gentle way to start while the foundations are built.
Designing a game day
Pick failures that match real risk
Choose from your own incident history and architecture, not from a generic list. Common, valuable candidates include:
- A pod or node dies during peak usage.
- An availability zone becomes unreachable.
- A dependency, such as a payment API, slows down or returns errors.
- DNS fails or a certificate expires.
- A database primary fails over.
- A bad deploy needs rollback under time pressure.
- Credentials are rotated or revoked unexpectedly.
Make it safe
Every experiment needs: a stated hypothesis; a limited blast radius (one service, a small percentage of traffic, one non-critical environment); a time window; an automatic or manual abort condition tied to an SLO signal; a known rollback; and a notice to everyone who might otherwise treat it as a real incident. Pausing an experiment must be easy and practised.
A concrete experiment
On Kubernetes, Chaos Mesh lets you declare a pod failure experiment as YAML. This one kills a single pod of a target service once:
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: orders-pod-kill
namespace: chaos-testing
spec:
action: pod-kill
mode: one
selector:
namespaces:
- staging
labelSelectors:
app: orders
Before running it, confirm the Deployment has more than one replica and a sensible PodDisruptionBudget, then watch your SLO dashboard. If the error rate breaches the abort threshold you set, delete the experiment object and investigate. The same idea applies to cloud-level faults using AWS Fault Injection Service or Azure Chaos Studio, which integrate with the provider’s permissions and stop conditions.
Tooling choices
| Option | Fits | Watch out for |
|---|---|---|
| Chaos Mesh | Kubernetes-native teams wanting declarative experiments | Requires cluster permissions and care in shared clusters |
| LitmusChaos | Kubernetes teams wanting a library of experiments and workflows | More moving parts to operate |
| AWS Fault Injection Service | AWS workloads, including EC2, EKS and networking faults | Scoped to AWS resources and IAM design |
| Azure Chaos Studio | Azure workloads | Azure only |
| Gremlin | Teams wanting commercial support and guardrails | Licence cost |
Capabilities evolve, so we verify current features when choosing. For Kubernetes shops we often begin with a small number of manual, scripted experiments before introducing a full platform.
Running the day
A typical session: a short briefing, steady-state check, injection, observation by responders using only the runbooks and dashboards, and a hot debrief. Assign roles as in a real incident, including a scribe, so findings are captured. Time-box the session and keep a clear end signal. Afterwards run a brief review in the style of a blameless post-mortem: what we expected, what happened, and what to fix.
Trade-offs and honest limits
Staging is not production. Different data, scale and configuration can hide real behaviour. Use staging to build confidence and tooling, then consider carefully scoped production experiments for specific hypotheses.
Cost versus value. Game days take engineer time. They pay off when they reveal real gaps, so focus on the highest-risk assumptions first.
Culture. Experiments will expose uncomfortable truths. If blame follows findings, people will stop volunteering for them. Keep it blameless.
Capacity and cost. Fault injection can trigger autoscaling or failover that costs money. Plan for that.
Common failure modes
- Running without a hypothesis, so nothing is learned.
- No abort condition.
- Testing the system but not the people.
- Findings recorded and never fixed.
- Doing it once and calling the system resilient forever.
Delivery and handover
We deliver a game day plan with hypotheses, safety rules and a calendar, the experiment definitions in your repository, facilitation of the first sessions, and a findings report with prioritised fixes. We train your engineers to run future sessions. Pairing with SLO work gives you the objective signals that define success and abort.
How to tell it is working
Track the number of weaknesses found and closed per game day, whether repeat runs of the same experiment improve, recovery time in drills, and runbook gaps discovered. Over time, real incidents should surprise you less. To plan a first game day, contact us.