Incident Management Process Design | Cloud DevOps Solutions

Incident Management: Commander, Comms and Scribe Roles

An incident management process defines who does what when production breaks. We introduce an incident commander who coordinates, a communications lead who updates stakeholders, and a scribe who records the timeline, plus a simple lifecycle and tooling so responders stop talking over each other.

Key takeaways

Chaotic incidents are mostly coordination failures, not technical ones.

Separate roles: one person decides, one communicates, one records, others investigate.

The commander does not debug. Their job is to keep the response organised.

A light process used every time beats a heavy one nobody remembers.

Practise with small incidents so the habit exists before the big one.

What an unmanaged incident looks like

You know the pattern. An alert fires. Six people join a channel and each starts investigating something different. Two roll back the same release. Nobody tells the customer success team what is happening, so they find out from angry emails. A senior engineer is deep in logs while the CEO asks for an update every ten minutes. Later, nobody agrees on the order of events.

The technical fault might have taken minutes to fix. The coordination failure turned it into an hour. A defined incident management process fixes that, and it sits at the centre of our SRE and incident response service.

The core roles

Incident commander

The commander runs the response. They keep a mental model of the situation, ask for status, assign actions and make the call on mitigations such as rollback or failover. They do not debug. The moment a commander starts typing commands, nobody is watching the whole incident.

Communications lead

This person owns updates to stakeholders and customers: the internal channel, executives, support and the public status page. They speak on a regular cadence even when there is no news, because silence causes more anxiety than “still investigating”.

Scribe

The scribe records a timestamped timeline: what was observed, what was tried, who decided what. Good notes shorten the review and make handovers painless.

Responders (subject matter experts)

Engineers investigating specific parts: database, network, application. They report findings to the commander and avoid side conversations.

In small teams one person may hold two roles during a minor incident. For serious ones, separating them is worth the cost.

The lifecycle

A simple, repeatable sequence beats an elaborate flowchart.

Declare. Anyone can declare an incident. Declaring early is encouraged, and there is no penalty for a false alarm.
Assemble. Create the channel, name the commander, communications lead and scribe, and link the relevant dashboards and runbooks.
Mitigate. The priority is restoring service, not finding the root cause. Rollback, failover, feature flag off, or scale out first.
Resolve. Confirm recovery against the objective signal, such as the SLO dashboard, and announce the end.
Review. Schedule a blameless post-mortem with owners for follow-up actions.

How deep each stage goes depends on the severity. A low-severity ticket-level issue may need only a named owner. A critical outage triggers all the roles.

Tooling that supports the process

You do not need much. We typically set up:

A paging tool integration that can open an incident and create a dedicated channel automatically.
A pinned message template in the incident channel.
A status page with templates for investigating, identified, monitoring and resolved.
A shared timeline document or thread convention for the scribe.

A simple channel opening message, which can be posted by a bot or pasted by hand, keeps roles clear:

INCIDENT DECLARED  |  Severity: SEV2  |  Opened: 14:07 UTC
Summary: Checkout error rate elevated since 14:02
Commander: @priya   Comms: @alex   Scribe: @sam
Dashboards: Checkout SLO board   Runbook: checkout-errors.md
Updates: every 20 minutes in this channel and on the status page
Rule: investigation chatter in thread, decisions and updates in main channel

The “every 20 minutes” value is an example. Cadence is agreed per severity.

Decisions and trade-offs

Heavy versus light process. Large companies adopt formal frameworks with many role types. For most startups a four-role model and a one-page guide is plenty. Process that nobody remembers during an incident is worse than a simple one used reliably.

Fixed versus rotating commanders. A permanent commander builds skill but creates a single point of failure and burnout. Rotating among trained engineers spreads the capability. We recommend rotation with shadowing for newcomers.

Command authority. The commander needs authority to make decisions quickly, including pausing a risky fix. Agree this with leadership in advance so a senior executive does not override the response mid-incident without taking the role.

When it is too much. Teams of one to three engineers handling rare incidents can run a minimal version: an owner, a written timeline and a customer update. Grow it as the team and the stakes grow.

Common failure modes

The hero commander. The most skilled engineer both commands and debugs, then the response collapses when they are stuck.
No handover. Long incidents need relief. The commander hands over explicitly and the new one confirms.
Too many people. Everyone joins, nobody leads. The commander can ask observers to leave or stay silent.
Premature root cause hunting. Mitigate first, investigate later.
Skipping the review. If incidents end without learning, the process is theatre.

A first ten minutes checklist

The opening minutes decide whether an incident feels controlled. We give the commander a short, rehearsed sequence rather than expecting improvisation.

Declare the incident out loud in the channel, with the severity and a one-line description.
Name yourself or a colleague as commander, and name the scribe and communications lead.
Ask for current customer impact, not theories about cause.
Start a single thread for decisions and keep investigation chatter in a separate one.
Set the first update time, for example “next update in 20 minutes”, and keep it even if nothing has changed.

Handing over command

Long incidents outlast people. A handover is explicit: the outgoing commander summarises impact, current theory, actions in flight and open risks, and the incoming commander repeats it back before taking over. The scribe records the change with a timestamp. Without that ritual, two people assume the other is in charge, or nobody is.

Keeping responders in lane

The commander coordinates and does not debug. If the commander is the only person who understands the failing system, that is a signal to name a technical lead for the investigation and keep the coordination role separate, even in a small team.

Delivery and handover

You receive role cards, the lifecycle guide, channel and status page templates, the configured integrations, and a small training session including at least one tabletop exercise. We then observe your first live incidents and refine the process with you. Where you want us to carry the pager, the same process applies under Managed DevOps.

How to tell it is working

Look at time to declare, time to assign a commander, time to first stakeholder update and time to mitigate, plus whether timelines are complete. Equally important are softer signals: responders say incidents feel calmer, leadership knows where to find updates, and reviews produce actions rather than arguments. To set this up for your team, book a consultation.

References

How we deliver it

Review how incidents run today

We read recent incident channels and timelines to see where coordination, ownership and communication broke down.

Define roles and handoffs

We write short role cards for commander, communications lead, scribe and responders, including who can take each role.

Design the lifecycle

Declare, assemble, mitigate, resolve and review, with clear entry and exit criteria for each stage.

Set up the tooling

Dedicated incident channels, a status page, a timeline template and paging integrations are configured.

Run tabletop exercises

We simulate incidents with your team so people practise the roles without real customer impact.

Observe real incidents

We coach in the first live incidents, then adjust the process to what actually happens.

Frequently asked questions

What does an incident commander do?

The commander owns the response, not the fix. They keep a clear picture of what is known, assign tasks, decide on mitigations such as rollback, and make sure communication happens. Good commanders stay out of the terminal so they can see the whole situation and keep people coordinated.

Do small teams need incident roles?

Yes, in a lightweight form. Even with four engineers, naming who leads, who updates stakeholders and who writes the timeline prevents duplicated work and silent gaps. One person can hold two roles in a minor incident, but for serious ones, separating them noticeably improves clarity.

What is the role of the scribe?

The scribe records a timestamped log of observations, decisions and actions as they happen, usually in the incident channel or a shared document. This makes the post-incident review accurate, helps late joiners catch up and removes the need to reconstruct events from memory.

Who should be incident commander?

It should be someone trained in the role, not necessarily the most senior engineer or the person who knows the failing system best. Rotate the role among trained people so the skill spreads. The on-call engineer can start as commander and hand over when the incident grows.

How is this different from the severity matrix?

Severity says how serious an incident is and what response level it triggers. The process describes how the response is run once triggered. They work together: higher severities invoke the full role structure, while minor incidents may need only an owner.