Having a recovery site is not the same as reaching it
A common disaster recovery surprise: the standby environment is healthy and waiting, yet users still hit the failed primary because nobody can move traffic quickly. The DNS change needs a console login that is behind the broken identity provider. The TTL is a day. The health check tests the wrong thing and either never triggers or flaps constantly.
Traffic management is the last mile of recovery, and it deserves the same design effort as data. It connects the recovery environment to your users, and it is a standard part of our backup and disaster recovery service.
What good looks like
- Public names resolve to the right place within a measured, known delay.
- Health checks reflect user experience, with sensible thresholds.
- The automatic and manual cutover rules are written and agreed.
- The secondary has valid certificates, capacity and data before traffic arrives.
- Failover and failback have been exercised.
How it works on AWS
Failover records
Route 53 failover routing uses a pair of records for one name. The primary is tied to a health check; if it fails, Route 53 returns the secondary. In Terraform:
resource "aws_route53_health_check" "primary" {
fqdn = "origin-primary.example.com"
port = 443
type = "HTTPS"
resource_path = "/healthz/deep"
request_interval = 30
failure_threshold = 3
}
resource "aws_route53_record" "app_primary" {
zone_id = aws_route53_zone.main.zone_id
name = "app.example.com"
type = "A"
set_identifier = "primary"
health_check_id = aws_route53_health_check.primary.id
failover_routing_policy {
type = "PRIMARY"
}
alias {
name = aws_lb.primary.dns_name
zone_id = aws_lb.primary.zone_id
evaluate_target_health = true
}
}
resource "aws_route53_record" "app_secondary" {
zone_id = aws_route53_zone.main.zone_id
name = "app.example.com"
type = "A"
set_identifier = "secondary"
failover_routing_policy {
type = "SECONDARY"
}
alias {
name = aws_lb.secondary.dns_name
zone_id = aws_lb.secondary.zone_id
evaluate_target_health = true
}
}
The domain names are examples. The important detail is the health check path. A shallow /healthz that returns 200 whenever the web process runs will say “healthy” while the database is gone. A deeper check should exercise dependencies that matter, but not so deep that a minor dependency failure triggers a full failover.
Route 53 health checkers probe from the public internet, so private endpoints need a CloudWatch alarm based check. For stricter control over cutover, Route 53 Application Recovery Controller provides routing controls that you flip deliberately, which helps avoid unsafe automatic switching when state is involved. AWS Global Accelerator is another option where fast shifting at the network layer is needed, since it avoids DNS caching, but it applies only to supported endpoint types.
How it works on Azure
Azure Traffic Manager returns DNS answers by a chosen method. For disaster recovery, priority routing lists endpoints in order: all traffic goes to the highest priority healthy endpoint, and falls to the next when monitoring marks it degraded. Monitoring probes a path and port you choose. Like Route 53, Traffic Manager works at the DNS level and does not proxy traffic. For HTTP workloads that need faster, connection-aware failover, Azure Front Door is often a better fit, but it introduces its own dependency and cost.
The TTL reality
Failover completes only when clients ask again. A TTL of 60 seconds sounds quick, yet resolvers and browsers can cache longer, and long-lived connections do not re-resolve at all. You can check what the world sees with:
dig +noall +answer app.example.com
dig +trace app.example.com
We set TTLs low enough for recovery but not so low that DNS cost and resolver load become a problem, and we measure real client convergence in a drill rather than trusting the number on the record. Lowering TTL on the day of a disaster is too late, because the old long TTL is already cached.
Decisions and trade-offs
Automatic or manual. A front end behind a load balancer failing to a warm standby is a fair candidate for automatic failover. If the secondary database is a replica that needs promotion, an automatic DNS switch to an unpromoted database makes things worse. We pair automatic DNS with manual data promotion, or use controlled switches.
Failure of the health check itself. Flapping, or a check that fails because of a network issue between the checker and your site, can move traffic needlessly. Use sensible thresholds, multiple regions of checkers and alarms on the check status.
DNS versus load balancing layers. DNS failover is a coarse tool. Within a region, load balancers handle instance failure. DNS is for site-level failure. Techniques like blue-green deployments use the same routing mechanisms for releases, which is a useful side benefit.
When not to use it. If everything is in one region and the business accepts a manual rebuild, you may not need automated failover. A documented, rehearsed manual switch costs nothing to run.
Common failure modes
- Secondary certificate expired or missing, so users get errors on the recovery site.
- Hard-coded endpoints in clients or partners’ allow-lists.
- DNS provider is itself in the failed region. Keep DNS hosting independent of what you are protecting.
- Health check calls an endpoint behind authentication.
- Failback forgotten. Traffic returns before data has been synchronised back.
Testing failover without causing an outage
Failover configuration is notoriously easy to get subtly wrong, and the first real test should not be a real incident. We validate in stages.
- Dry run on a test record. Create a non-production hostname with the same health check and routing policy, then fail the primary deliberately and observe the answer change.
- Measure the real delay. Record how long it takes from the fault to a client receiving the new answer, including health check detection, record TTL and resolver caching. The slowest part is usually client and resolver behaviour, not the DNS service itself.
- Check the secondary under load. Confirm the standby environment can serve production traffic, not just pass a health check. A healthy but undersized standby fails slowly.
- Practise failback. Returning traffic is a separate decision. We document when to fail back, and whether to do so manually to avoid flapping between sites.
DNS failover also does not fix data problems. If the standby holds stale or inconsistent data, redirecting users makes things worse, so routing changes are tied to the data steps in your DR environment architecture and runbooks.
Delivery and handover
Records, health checks and alarms are Terraform in your repositories. The runbook for each service includes the cutover and failback steps, access paths that do not depend on the failed region, and a measured convergence time. Our ingress and platform work ensures clusters expose the health endpoints these checks need.
How to tell it is working
Health checks are green in normal operation and have never flapped without cause. A drill moves test traffic to the secondary and back, with the measured time written in the runbook. To design failover for your public endpoints, contact us.