Escalation Paths and Incident Communication Templates

Two problems that look like one

When production fails, two things must happen in parallel. The right people must be pulled in quickly, and everyone else must be told something sensible. Teams often handle the first reasonably and the second poorly, or neither. An engineer discovers at hour two that the database specialist was never paged. The CEO learns about the outage from a customer tweet. Support improvises answers that later contradict the status page.

Escalation paths solve the first problem and communication templates solve the second. Both are standard parts of the incident management work in our SRE and incident response service.

Escalation paths

What an escalation path is

An escalation path is a pre-agreed ladder. If the first person does not respond, or the incident outgrows them, the next rung is contacted automatically. A typical ladder looks like this:

1. Primary on-call engineer for the service.
2. Secondary on-call, if the primary does not acknowledge in time.
3. Subject matter expert or owning team lead for the affected component.
4. Engineering manager or head of platform.
5. Executive and communications, for major customer or revenue impact.

There are two kinds of escalation, and they should not be confused. Functional escalation brings in more skill: the database expert, the network specialist, the vendor’s support engineer. Hierarchical escalation brings in more authority or awareness: a manager who can approve a costly mitigation or tell the board.

Triggers should be written down

Hesitation to escalate is a common cause of long incidents. People worry about waking someone or looking incompetent. Removing the judgement call helps:

• Escalate to a specialist if there is no mitigation progress after an agreed period.
• Escalate to management when the severity reaches a stated level.
• Escalate to a vendor when evidence points to a third-party fault.
• Always escalate suspected security or data exposure to the designated owner immediately.

The specific times are set with you and depend on your coverage. We avoid prescribing universal numbers.

Configuring it in the paging tool

In PagerDuty, OpsGenie or Grafana OnCall, an escalation policy is a first-class object. Defining it as code keeps it reviewable. A Terraform sketch for PagerDuty:

resource "pagerduty_escalation_policy" "platform" {
  name      = "Platform production"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.primary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.secondary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.eng_manager.id
    }
  }
}

The delay values are placeholders to show the mechanism. Resource names will differ in your account. The full rotation design is described in on-call rotation setup.

Communication templates

Know your audiences

Status page template

Public updates move through a standard sequence: investigating, identified, monitoring, resolved. Each should be short and free of speculation. For example:

Investigating: We are aware that some customers are seeing errors when
placing orders. Our engineers are investigating. We will post another
update by 15:30 UTC.

Identified: We have identified the cause as a faulty configuration change
and are rolling it back. Some customers may still see errors. Next update
by 15:50 UTC.

Resolved: Order placement has returned to normal. We will publish a summary
once our review is complete.

The times in this example are illustrative. Notice what it avoids: no technical jargon, no blame, no promise of a fix time that may slip, and a commitment to the next update.

Internal and executive updates

Executives want three things: impact, what is being done, and what they need to decide. A template with those three headings, sent at the agreed cadence, stops repeated interruptions to responders. A separate support brief carries approved customer wording and workarounds so the front line does not guess.

Trade-offs and judgement

Transparency versus risk. Honest, timely updates build trust, but over-detailed technical statements can be wrong and can create legal exposure for security events. Have legal or leadership review the wording for security incidents and keep drafts ready.

Automation versus control. You can connect monitoring to the status page so it updates automatically. This is fast but can announce false positives. Many teams prefer a human to publish, with templates ready.

Public versus private status. Some enterprise contracts require direct notification instead of a public page. Design for the commitments you actually have.

When to keep it simple. A tiny team with few customers may only need a contact list, a ladder of three names and two or three message templates.

Common failure modes

• Contact details out of date or reachable only on office hours.
• Escalation policy loops back to the same unreachable person.
• No decision about who talks to customers.
• Updates stop when the news is bad.
• Templates written once and never rehearsed.

Delivery and handover

You receive the escalation map, the configured policies, ready-to-use templates in your status and chat tools, a contact register with a review date, and a rehearsal run with your team. For round-the-clock operations this ladder is the one our Managed DevOps engineers follow too.

How to tell it is working

Look at time to reach the right specialist, time to first stakeholder update, how many updates arrived on schedule, and the volume of support tickets raised during incidents. Qualitative signals matter: executives stop interrupting the channel, and support confidently quotes the same line as the status page. To design yours, book a consultation.

References

• Managing Incidents, Google SRE book
• Incident Response, Google SRE workbook
• Incident Response Recommendations and Considerations for Cybersecurity Risk Management, NIST SP 800-61 Rev. 3