The deploy that fails at 4:50 on a Friday
Every team eventually ships a bad release. What separates a five minute blip from a two hour incident is not whether the bug existed, but how quickly the system returns to a working state. If the answer is “someone finds the previous version, remembers the right commands and hopes the config still matches”, the response is slow and stressful.
Automated rollback makes recovery a designed behaviour of your CI/CD pipeline instead of an improvisation. This page explains how we build it, including the awkward cases such as database changes, where rollback is not as simple as it sounds.
What good looks like
- Fast, reliable detection. The pipeline knows within minutes whether a release is healthy, using signals that distinguish real failure from noise.
- A precise target. The last good version is identified by immutable digest or revision, not by
latest. - A tested revert. The mechanism has been exercised against a deliberately broken release.
- Backward-compatible data changes. The previous version can run against the current schema.
- Visibility. Every automatic rollback produces an alert and a record stating what failed.
- A fallback to humans. Ambiguous situations escalate rather than guessing.
How it works in practice
Layer one: the platform stops bad rollouts
Kubernetes already provides the first layer. A rolling update only terminates old pods as new ones become ready, so a version that crashes on startup or never passes its readiness probe never receives traffic, and the rollout stalls. Setting maxUnavailable conservatively and defining proper readiness probes is the foundation, and many teams have probes that do not test anything meaningful. We review them as part of the work.
If the rollout stalls, progressDeadlineSeconds marks it as failed, and the pipeline can then act on that status.
Layer two: the pipeline verifies and reverts
After deployment, the pipeline should wait for the rollout, run smoke tests, and revert if either fails. With Helm, the --atomic flag does the core of this:
helm upgrade --install checkout ./charts/checkout \
--namespace shop \
--set image.tag="$IMAGE_DIGEST" \
--atomic \
--timeout 5m \
--history-max 10
With --atomic, Helm waits for resources to be ready and, if the upgrade fails or times out, rolls back to the previous release automatically. That covers startup failures. It does not cover a release that starts cleanly but misbehaves, which is why a post-deploy smoke test step follows:
./smoke-test.sh https://staging.example.com || {
helm rollback checkout 0 --namespace shop --wait
exit 1
}
Revision 0 in helm rollback means the previous revision. We prefer explicit smoke tests that exercise a real user path (log in, create a record, read it back) over a generic ping, because a service can return HTTP 200 on its health endpoint and still be broken.
Layer three: progressive delivery
For higher-risk services, canary releases limit exposure. A tool such as Argo Rollouts or Flagger sends a small share of traffic to the new version and queries Prometheus for error rate and latency. If the metrics exceed thresholds, the rollout aborts and traffic returns to the stable version automatically, with no one watching a dashboard. Well-written alerting rules double as analysis queries. This approach suits services with enough traffic to measure; a low traffic service may never produce a statistically meaningful signal in the canary window.
Blue-green deployment is the simpler alternative: run both versions, switch traffic, and switch back if needed. It costs double capacity during the release but gives the fastest reversal.
GitOps rollbacks
With a GitOps controller such as Argo CD, the desired state lives in Git. Rollback is a revert commit, which the controller applies. This is slower than an in-cluster undo but fully audited, and it avoids the trap where someone rolls back in the cluster and the controller immediately redeploys the bad version. Our GitOps tutorial for EKS shows the model, and the Helm and Argo CD deployment page covers how rollbacks fit in.
The hard part: data
Code rolls back easily. Data does not. If a release drops a column, renames a table or rewrites records, the old code may fail against the new schema, and no deployment tool can undo the lost data.
The accepted approach is expand and contract:
- Expand: add the new column or table without removing anything. Old and new code both work.
- Migrate: deploy code that writes to both and reads from the new structure.
- Contract: in a later release, once rollback of the earlier code is no longer plausible, remove the old structure.
This discipline is the single biggest factor in whether automated rollback is safe. Where destructive migrations are unavoidable, we mark those releases as non-rollbackable, take a backup beforehand (see backup and disaster recovery) and require approval.
Decisions and trade-offs
Roll back or roll forward. If the fix is a one-line change that you understand and the pipeline takes ten minutes, rolling forward can be quicker and avoids reverting unrelated changes bundled in the release. The pipeline should support both without drama.
Automatic versus approved rollback. Hard failures (pods never ready, smoke test fails) are safe to automate. Soft failures (error rate rises a little) are riskier, because noise can trigger needless reverts or flapping. We start with automation on hard signals and widen it as trust grows.
Cost. Canary and blue-green approaches add complexity or capacity. For a small service with modest traffic, --atomic plus smoke tests is often enough. We say so rather than selling progressive delivery everywhere.
Common failure modes
- Useless readiness probes that always return success.
- Rollbacks never tested, which fail the first time they matter.
- Configuration not versioned with the code, so the old image runs with new settings.
- Flapping, where a noisy metric triggers repeated rollbacks.
- Silent rollbacks that nobody notices, so the same broken change is redeployed.
- Mutable tags like
latest, which make “previous version” ambiguous.
Delivery and handover
We define failure signals with your team, implement detection and revert for each deploy target, and rehearse in staging with a deliberately broken release, including a database scenario. Rollback events post to your chat channel, as described in deployment notifications. You receive a runbook covering automatic behaviour and the manual commands for when automation is the wrong tool.
How to tell it is working
Track mean time to restore for deployment failures, the share of failed releases recovered automatically, change failure rate, and the number of rollback rehearsals completed per quarter. A falling time to restore with a stable or lower failure rate is the outcome to look for. To discuss your release process, book a consultation.