Incident Response with Defined Severities and Real Post-Mortems

Why incident response needs structure

When production breaks, the technical problem is often smaller than the organisational one. Four engineers join a call, two make changes at the same time, nobody tells customers anything, and the fix is forgotten by Monday. The cost is not just downtime. It is repeated downtime, because the lesson was never written down.

Incident response gives you a rehearsed way to act under pressure. It sits alongside 24/7 monitoring and on-call coverage as the part of our Managed DevOps service that you only notice when something goes wrong. For the standalone discipline, see our SRE and incident response service.

What good looks like

A mature incident process has five properties.

1. Severity is defined before it is needed. Criteria use customer impact, not engineer opinion.
2. Roles are separate. One person coordinates, others investigate, one communicates.
3. Mitigation beats diagnosis. Restore service first.
4. Communication has a rhythm. Updates arrive on a schedule even when there is nothing new.
5. Learning is tracked. Actions have owners and dates and are reviewed until closed.

Severity levels in practice

The table below is a starting point. We adapt the wording to your product during onboarding.

We do not publish resolution time guarantees for incidents, because honest resolution depends on root cause. What we do commit to is how fast a human engages, how often you hear from us, and that the problem is reviewed afterwards.

How an incident runs

Detection and acknowledgement

An alert reaches the on-call engineer through PagerDuty or OpsGenie. They acknowledge, open a dedicated channel, and set an initial severity. If the first person is unsure, they escalate rather than debug alone for twenty minutes.

Command and control

For P1s we name an incident commander. Their job is to keep the timeline, decide who does what, and stop parallel conflicting changes. They do not type commands unless the team is tiny. A simple opening message looks like this:

SEV: P1  |  Impact: checkout returning 502 for ~40% of requests
IC: on-call (platform)   Ops: second engineer   Comms: account lead
Hypothesis: bad deploy at 02:14 UTC. Plan: roll back, confirm error rate.
Next update: 15 minutes.

Mitigate

The usual levers are rollback, feature flag off, failover, scaling out, or draining a bad node. Most are one command if the platform was built for it, which is why pipeline ownership and infrastructure as code matter to incident speed. A rollback on Kubernetes can be as small as:

kubectl rollout undo deployment/checkout -n production
kubectl rollout status deployment/checkout -n production

Communicate

Stakeholders want three things: what is broken, what you are doing, and when they will hear next. We keep updates short and free of jargon. If you run a public status page, we publish there as well.

Close and learn

After service is restored we confirm stability, stand down, and schedule the post-mortem within a few working days while memory is fresh.

The post-mortem

A good post-mortem is blameless and specific. People are not the cause of failures; systems that allowed a single mistake to reach production are. Ours includes:

• Summary and impact: who was affected and for how long.
• Timeline: detection, acknowledgement, mitigation and resolution times.
• Contributing factors: usually several, such as a missing alert, an untested migration and a risky deploy window.
• What went well: so we keep doing it.
• Actions: each with an owner, a priority and a due date, tracked in your issue tracker.

We chase open actions in the weekly check-in until they close. A post-mortem whose actions are ignored is just a diary.

What the first thirty minutes look like

The first half hour sets the tone for the whole incident, so we keep it disciplined. In order:

1. Acknowledge the page and say so in the channel, so nobody wonders whether anyone is awake.
2. Check the three questions: is the user affected, what changed recently, and is it getting worse.
3. Look at deploy markers and recent infrastructure changes before looking at code. Most incidents follow a change.
4. Pick the lowest-risk mitigation that plausibly helps, announce it, and apply it.
5. Verify using the same signal that triggered the alert, not a gut feeling.
6. Post the first stakeholder update, even if it only says what is known and what is not.

The hardest habit to build is stopping to think. Under pressure, people try five things at once and cannot tell which one worked. The commander’s quiet job is to enforce one change at a time and note what each did.

Handling third-party and cloud provider failures

Sometimes the cause is outside your system: a cloud region degrades, a payment provider fails, or a DNS vendor has an outage. We still run the process, because customers still need updates and you may have options such as failing over, degrading gracefully or switching a feature off. We record the dependency in the post-mortem and look at whether a backup and disaster recovery design or a fallback path would have limited the impact.

Trade-offs and when this is not enough

Response is not resolution. If your incidents are caused by application bugs or a database schema nobody understands, we can stabilise and diagnose, but the code fix is yours. We say so in the post-mortem instead of hiding behind infrastructure.

Customer status pages need your decisions. We can draft and publish updates, but what you disclose publicly, especially around security, is a business call we will ask you to make.

Very small teams. If you have one service and a handful of users, formal severity tiers may be heavier than you need. A shared on-call and a simple checklist would do.

Common failure modes

• Severity inflation. If everything is P1, nothing is. Examples in the definitions keep tiers honest.
• Hero debugging. One engineer goes silent for an hour. The commander role exists to prevent it.
• Action item decay. We track follow-ups in the monthly report so slippage is visible.
• No test. Process untested is process imagined. We run a game day early in the engagement to rehearse a P1 safely.

How to tell it is working

Track time to acknowledge, time to mitigate, repeat incidents per cause, and the percentage of post-mortem actions closed on time. Over a few months, incident count may stay flat while severity falls. That is a healthy sign: the system is failing in smaller, safer ways.

If you want to see how this would fit your team, talk to us or review the engagement models.

References

• Managing Incidents, Google SRE Book
• Postmortem Culture: Learning from Failure, Google SRE Book
• SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management, NIST