Disaster Recovery Architecture: Pilot Light, Warm Standby or Active-Active

The cost of recovering quickly

Disaster recovery is a negotiation between time and money. If you can tolerate a day of downtime, you can keep only backups and rebuild afterwards. If you can tolerate minutes, something must already be running in a second place. There is no clever architecture that avoids this: faster recovery means paying for capacity, replication and operational work in advance.

The four standard strategies, popularised in AWS’s disaster recovery guidance and applicable on any cloud, are backup and restore, pilot light, warm standby and multi-site active-active. We help you pick per service tier, using the targets agreed in the workshop, and build them as part of our backup and disaster recovery service.

The four patterns compared

Backup and restore

You keep backups copied to another region and rebuild everything from code after a disaster. It is a sound choice for tier 3 services. Its weakness is time: you must provision infrastructure, restore data and validate before users return. If infrastructure is not fully in code, this pattern is far slower than it looks on paper.

Pilot light

The data layer runs continuously in the recovery region, replicated from the primary, while application servers are absent or stopped. On failover you create compute from your templates, scale it up and switch traffic. The name comes from a pilot light on a boiler: small, always on, ready to ignite the rest. Recovery time is dominated by how quickly you can build and warm the compute, and by promotion of the replica database.

Warm standby

A reduced version of the full environment runs all the time, perhaps at a fraction of production capacity, and can serve traffic immediately at small scale. On failover you scale up. Because the stack is already running, you also test it every day with a trickle of real or synthetic traffic, which reveals problems earlier than a dormant environment would.

Active-active

Both regions serve live traffic. Failure of one means the other absorbs the load, so recovery can be close to seamless. The price is architectural: data must be writable or reconcilable in both places, caches and sessions must tolerate it, and deployments must be coordinated. Managed globally distributed databases help but impose their own consistency model. This is a design decision for new systems more than a retrofit.

Data is the hard part

Compute is easy to recreate. State is not. For each store we decide how it replicates and what lag we accept:

• Relational databases: managed cross-region replicas or global database features promote a replica in the recovery region. Replication is asynchronous in most designs, so a regional failure can lose the most recent transactions, which is your effective RPO.
• Object storage: replication rules with versioning, as in cross-region backup replication.
• Queues and caches: often rebuilt or drained, not replicated. Be explicit about what is lost.
• Secrets and keys: available in the recovery region before you need them.

Build the recovery site from code

A recovery environment that someone created by hand last year is not a recovery environment. It should be defined with the same Terraform modules as production, parameterised by region and size. A small variable controls the pattern:

variable "dr_mode" {
  type        = string
  description = "pilot_light or warm_standby"
  validation {
    condition     = contains(["pilot_light", "warm_standby"], var.dr_mode)
    error_message = "dr_mode must be pilot_light or warm_standby."
  }
}

locals {
  app_desired_capacity = var.dr_mode == "warm_standby" ? 2 : 0
}

For Kubernetes, a second cluster is configured by GitOps so application manifests appear automatically, as we describe in GitOps with Argo CD and Flux and for AWS in our EKS consulting. Persistent data still needs its own replication or restore path.

Traffic then has to move, which is covered by Route 53 and Traffic Manager failover.

Decisions and trade-offs

Pick per tier. Most estates end up with a mix: warm standby for the revenue path, pilot light for core tools, backup and restore for the rest.

Single region, multi-AZ first. Multi-AZ already tolerates most infrastructure faults. Add multi-region when a regional event would be unacceptable to the business.

Managed DR services. Tools such as AWS Elastic Disaster Recovery and Azure Site Recovery replicate servers and orchestrate failover, useful for lift-and-shift estates. Cloud-native applications are usually better served by data replication and code.

When not to build a DR site. If your RTO is measured in days and your infrastructure is code, tested backups in another region are enough. A standby environment that nobody operates costs money and decays.

Common failure modes

• Drift between production and DR. Production gets a new service, DR does not.
• Capacity limits. The recovery region has quotas that block scaling to production size. Request increases ahead of time.
• Missing dependencies: certificates, container images, secrets, allow-lists and third-party callbacks tied to the primary region.
• Untested promotion. Database failover works on paper, then takes three times longer.
• No failback plan.

Delivery and handover

You receive the chosen pattern per tier with a written cost range, the recovery environment as code, replication configured and monitored, and a failover and failback procedure in the runbooks. We run a live drill and report measured recovery time against target.

How to tell it is working

Replication lag is monitored and within budget, the recovery environment matches production in every drill, measured failover meets the RTO, and a failback has been done at least once. To size a recovery design for your platform, book a consultation.

References

• Disaster recovery options in the cloud, Disaster Recovery of Workloads on AWS, AWS Whitepapers
• Architecture strategies for disaster recovery, Microsoft Azure Well-Architected Framework, Microsoft Learn