Outsourced On-Call: We Take the 3am Page, Not Your Developers

The hidden cost of developer on-call

When a small team shares the pager, the cost does not appear on any invoice. A three-person engineering team cannot sustain a healthy rotation: someone is always the night’s responder, sleep suffers, weekends become provisional, and the best people eventually leave. Product work slows because the person on call is tired or interrupted. Meanwhile infrastructure problems are handled by developers whose skills lie elsewhere, so recovery is slower than it needs to be.

Outsourced on-call changes who holds the pager. It is a core element of our Managed DevOps service, working alongside 24/7 monitoring and the incident process.

What good on-call looks like

Whoever carries the pager, the characteristics are the same:

• Pages are rare and meaningful. If the pager goes off weekly for non-issues, the system is wrong, not the people.
• There is a primary and a secondary. A single person with no backup is a single point of failure.
• Everyone knows the escalation path. No guessing who to call.
• Procedures exist. The responder is never alone with an unfamiliar system; they have runbooks.
• Follow-up is mandatory. Every page produces either a fix, a downgrade or a deletion of the alert.
• Load is shared fairly and monitored. Out-of-hours pages are counted, not forgotten.

Google’s SRE guidance has long treated sustained high page load as a signal to fix the system rather than to ask humans to endure it. We take the same view.

How the arrangement works

Scope: what we own and what we triage

The boundary is the most important part of the contract. A typical split:

We write this table with you, and revisit it as the system changes. Ambiguity at 3am is expensive.

Escalation policy

In PagerDuty or OpsGenie, an alert routes by service to a schedule. If the primary does not acknowledge within a short window, it escalates to the secondary, then to a named manager. A separate developer schedule exists but is last in line, and is used deliberately. Conceptually:

escalation_policy: platform-production
  - level: 1
    target: managed-devops-primary
    timeout: 5m
  - level: 2
    target: managed-devops-secondary
    timeout: 10m
  - level: 3
    target: managed-devops-lead
    timeout: 15m
  # developer escalation is triggered by the responder, not automatically

The exact timings are agreed with you. This sketch shows structure only; PagerDuty and OpsGenie each configure it through their own interface or Terraform provider.

Access that is safe and fast

An on-call engineer who cannot reach production is useless; one with unlimited access is dangerous. We use single sign-on with MFA, role-based access scoped to the platform, audited sessions, and a break-glass role for emergencies that alerts when used. Our compliance work reviews this access regularly. All credentials remain in your accounts.

Onboarding before we carry the pager

We never take the pager on day one. The sequence is: read existing documentation and past incidents, shadow your current responders, write the initial runbooks, review the alerts for noise, and rehearse with a game day. Only then does first-line duty move to us. For a period afterwards your team remains as backup so any gap shows up safely.

The handoff to your developers

Sometimes the problem really is in the code. The goal is to wake a developer only when it is the fastest route to recovery, and to arrive with everything they need:

• What is failing, since when, and how many users are affected.
• What changed recently (deploys, config, dependencies).
• Graphs, log excerpts and trace IDs.
• What has already been tried and what is the next question.

Often we can mitigate with a rollback or a feature flag and leave the investigation for the morning. That outcome keeps both the service and your team’s sleep intact.

Decisions and trade-offs

Full outsourcing versus hybrid. Some teams prefer us on nights and weekends while their developers take daytime application issues. Others want us as the single first line. Both work. Hybrid keeps developers close to production behaviour; full outsourcing maximises rest.

Follow-the-sun versus single timezone. A distributed engineering group avoids night work by handing over across timezones. It needs enough people in each region to be sustainable. For a small team it is rarely practical, which is a main reason clients bring us in.

Depth of system knowledge. An external responder knows your system less than the engineers who built it. That is why runbooks, onboarding and a feedback loop matter, and why we are candid that the first weeks need extra attention.

When not to buy this. If you have one low-risk internal service and customers who tolerate downtime, a simple alert to your own team may suffice. Outsourced on-call earns its place when downtime is costly and your team is too small to cover nights sustainably.

Common failure modes

• Vague scope. Disputes over whose problem it is waste the first twenty minutes of an incident.
• No access. The responder cannot act. Test it with a drill.
• Alert noise inherited. We tune alerts early, so we do not exhaust the on-call engineer on nuisance pages.
• Poor communication. Developers learn about a problem from customers. We post updates in your channels.
• No learning. The same page repeats. Page review is part of the monthly report.

How to tell it is working

Watch acknowledgement time, pages per week out of hours, the share of pages resolved without developer escalation, and repeat pages for the same cause. A good sign is falling page volume and developers reporting uninterrupted nights. For commercial structure, see our engagement models, or book a consultation to scope coverage.

References

• Being On-Call, Google SRE Book
• On-Call, Google SRE Workbook
• Managing Incidents, Google SRE Book