The backup nobody has restored
Backup monitoring usually tells you that a job ran. It rarely tells you that the result is usable. Jobs can succeed while writing an empty file, a corrupted archive, or data encrypted with a key that was later rotated away. Permissions change, a database version moves on, a script quietly stops capturing a new table. The first time anyone finds out is the day they need the data.
Automated restore testing closes that gap. It is the single most valuable practice in our backup and disaster recovery service, because it converts belief into evidence.
What good looks like
- Restores run on a schedule without anyone remembering to start them.
- They run in isolation, so they cannot damage production or leak data.
- The result is validated with real checks, not just an exit code.
- Duration is recorded and compared with the RTO.
- Failures create tickets and are fixed, with the runbook updated.
How it works in practice
The sandbox
Restores land in an environment designed for them. Ideally this is a separate account or subscription with its own network, no peering to production, and a budget alarm. Restored systems must not behave like production: outbound email, payment calls, webhooks and queue consumers are disabled or pointed at stubs. Otherwise a restored database could send thousands of real notifications.
If the data is sensitive, the sandbox needs the same encryption and access controls as production, because it holds a full copy.
The workflow
A scheduler such as EventBridge or a pipeline cron triggers an orchestrator, for example Step Functions or a CI workflow. The orchestrator restores the latest recovery point, waits, runs validation, records metrics and tears the resources down. On AWS, Backup restore testing plans can handle selection, restore and cleanup for supported resource types, and we use them where they fit. For databases and unusual resources, a custom workflow calls the same restore steps your runbook documents.
A minimal CI schedule for a PostgreSQL dump test looks like this:
name: restore-test-postgres
on:
schedule:
- cron: "0 4 * * 0" # weekly, Sunday 04:00 UTC
workflow_dispatch: {}
jobs:
restore:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: ${{ secrets.SANDBOX_DB_PASSWORD }}
ports: ["5432:5432"]
steps:
- uses: actions/checkout@v4
- name: Fetch latest backup
run: ./scripts/fetch-latest-backup.sh # uses short-lived OIDC credentials
- name: Restore and time it
run: |
start=$(date +%s)
pg_restore --no-owner --jobs=4 --dbname="$SANDBOX_DB_URL" latest.dump
echo "restore_seconds=$(( $(date +%s) - start ))" >> "$GITHUB_OUTPUT"
- name: Validate
run: psql "$SANDBOX_DB_URL" -f scripts/validate.sql
Real data usually exceeds what a hosted runner can handle, so large databases restore into temporary cloud instances instead. The structure stays the same: fetch, restore, time, validate, destroy.
Validation that means something
A passing job is not proof. Checks we typically add:
- Row counts for key tables compared with the source within an expected margin.
- A freshness check: the newest record is within the RPO window.
- Checksums or sampled queries on critical tables.
- Schema and extension presence.
- A smoke test that starts the application against the restored data and calls a health endpoint.
For object storage and file backups, we restore a sample and verify checksums. For Kubernetes workloads, a tool such as Velero restores into a scratch cluster or namespace, and we check pods become ready.
Measure time
Each run records how long the restore took. Over weeks this builds a trend: a restore that took 40 minutes last quarter and takes 3 hours now is a warning, even if it still succeeds. These numbers feed your Grafana dashboards and alerts in the same way as other service health signals.
Decisions and trade-offs
Full restore or sampled. Restoring everything every week is expensive for large data sets. A common compromise is sampled restores weekly and a full restore of each tier quarterly.
Native service or custom workflow. Use the native feature when it supports the resource, because it is less to maintain. Custom workflows give full control for databases, application validation and multi-step recovery.
Where to run it. Same region as production tests the common path. Running from the copy in another region or account tests the one you will rely on in a disaster. We do both over time.
When not to automate yet. If you have no documented restore procedure, automating chaos helps nobody. First do one manual restore, write it down, then automate it.
Common failure modes
- Green status, empty data. Validation is skipped or too shallow.
- Sandbox not isolated, so a restored system contacts real services.
- Test uses a different method than the real recovery, so it proves nothing about the runbook.
- Cost surprise from large resources left running. We destroy resources automatically and set budgets.
- Failures ignored. The alert exists, but nobody owns it.
A worked example: one failed test, end to end
Imagine the weekly test restores last night’s database snapshot into the sandbox, and the row-count check passes but the application smoke test fails with a permissions error. Without automation, nobody would have known until a real incident. With it, the workflow opens a ticket with the restore logs attached, the owner finds that a recent change moved the application role into a new schema, and the restore procedure is amended the same day. The next scheduled run passes, and the fix is recorded in the runbook with the date.
The point is the loop, not the tooling: a test fails safely, a named person is told, the procedure changes, and the next test proves the change. We also keep a short log of every failure and its cause, because patterns (for example, repeated key access problems) tell you where your recovery design is weakest.
Delivery and handover
You receive the sandbox environment, the scheduled workflows as code, validation scripts, a dashboard and an alert route. A short guide explains how to add a new data store to the test suite. Results feed the disaster recovery runbooks, which are corrected whenever a test finds a gap. For alert design patterns, see our guide to Prometheus alerting rules.
How to tell it is working
Every critical store has a recent successful, validated restore. Measured restore times sit within RTO with headroom. Failures are rare and closed quickly. Most importantly, your team can answer “when did we last restore this?” with a date. To set this up, book a consultation.