The deployment that works until it does not
Many teams deploy by SSH-ing into a server, pulling code and restarting a service, then repeating on the next server. It works for months. Then a deployment half-succeeds on three of six servers, the load balancer sends users to a mix of old and new code, and nobody knows which servers are on which version.
An Ansible rolling deployment replaces that ritual with a repeatable procedure. This page describes how we build those playbooks for virtual machine and bare metal applications under our Ansible service, and, just as importantly, when to use something else.
What good looks like
- Users do not see an error or a dropped session during a normal release.
- At most a small, defined share of capacity is out of service at any time.
- A failure stops the rollout automatically.
- Every server reports the version it runs, and the pipeline records it.
- Rollback is a rehearsed, boring operation.
How it works in practice
The rolling pattern
The core idea fits in one play. serial limits how many hosts are handled at once, and pre and post tasks bracket the deployment:
- name: Rolling deploy of the web application
hosts: web
serial: "25%"
max_fail_percentage: 0
become: true
pre_tasks:
- name: Take host out of the load balancer
ansible.builtin.include_tasks: tasks/lb_disable.yml
- name: Wait for in-flight requests to drain
ansible.builtin.wait_for:
timeout: 30
roles:
- role: app_deploy
vars:
app_version: "{{ release_version }}"
post_tasks:
- name: Verify local health endpoint
ansible.builtin.uri:
url: "http://localhost:8080/healthz"
status_code: 200
register: health
retries: 12
delay: 5
until: health.status == 200
- name: Return host to the load balancer
ansible.builtin.include_tasks: tasks/lb_enable.yml
With max_fail_percentage: 0, the play stops if any host in a batch fails, and later batches never start. The wait_for here is a simple pause. In a real design we replace it with a query of the load balancer’s connection or target state, so draining ends when it is actually finished.
Controlling the load balancer
For AWS, the amazon.aws collection deregisters and registers targets in an Application Load Balancer target group, and the deregistration delay does the draining. For HAProxy, tasks set the server state through the admin socket. For Azure, a load balancer’s backend pool membership or an Application Gateway setting can be driven through the Azure collection. The playbook delegates these tasks to the controller with delegate_to: localhost, because the load balancer is an API, not the host.
Immutable release directories
Deploying in place leaves mixed files and makes rollback hard. We prefer:
/opt/app/
releases/
2026-10-01-1.8.2/
2026-10-08-1.9.0/
current -> releases/2026-10-08-1.9.0
shared/config/
The role unpacks the new release, links shared configuration, runs a smoke command, switches the current symlink atomically and restarts the service. Rolling back means pointing the symlink at the previous directory and restarting.
Health checks that mean something
A port that accepts a connection is not a healthy application. The health endpoint should check what the application needs to function, such as its database connection, without becoming so heavy that it takes the service down. The playbook then waits for that endpoint using until and retries. Wider metrics, such as error rate during the rollout, are better watched with Prometheus and Grafana, and a spike can trigger a human pause.
Database changes
Rolling deployments mean old and new code run side by side for a while, so schema changes must be compatible with both. The safe pattern is expand then contract: add the new column nullable, deploy code that handles both, backfill, and only in a later release drop what is unused. Migrations run once, from a single host, with run_once: true, not in every batch.
Decisions and trade-offs
Rolling or blue-green? Rolling uses existing capacity and is simple, but runs two versions at once. Blue-green deployment switches traffic between two full environments, which gives faster rollback and cleaner cutover at the cost of double capacity during the switch. We pick based on your cost tolerance and how awkward mixed versions are for your application.
Ansible or the platform’s native deployment. For containers on Kubernetes, use the native rollout and a GitOps tool, as covered by our GitOps service. Ansible is the better fit for VMs, appliances and legacy software.
Canary first or straight to batches. A first batch of one host is cheap protection. We add it as the default.
When not to build this. If you have one server and can tolerate a minute of downtime at 3 a.m., a rolling pipeline is overbuilt. Be honest about your availability needs.
Common failure modes
- Health check that always passes. The rollout marches through a broken release. We test it with a deliberately failing build.
- Load balancer and playbook disagree. Targets re-enabled before they are ready. Wait on the load balancer’s own view.
- Concurrent migrations. Two hosts migrate at once. Use
run_onceand locks. - Configuration drift between hosts. A release works on five servers and not the sixth. The baseline role prevents this.
- Secrets in the release artefact. Config and secrets come from Vault-managed variables, not the build.
Delivery and handover
We deliver the deployment role and playbook, load balancer integration for your platform, a rollback playbook, a staging rehearsal with a deliberately broken release, and pipeline wiring. Deployments are typically triggered through AWX job templates or your CI system, which keeps a record of who released what.
Rehearsing the rollback
A rollback you have never run is a hope, not a plan. We rehearse it in a staging environment on a schedule, including the awkward case where a database migration has already run. The result is a short, tested list of what can be reversed automatically and what needs a human decision.
How to tell it is working
- Release windows disappear because releases stop being events.
- A broken release is stopped on its first batch and costs one host, not an outage.
- You can state the running version on every server in seconds.
- Rollback is exercised in staging regularly, and is quick.
To review how your current deployments behave under failure, book a consultation.
References
- Playbook Example: Continuous Delivery and Rolling Upgrades, Ansible Community Documentation
- Controlling playbook execution: strategies and more, Ansible Community Documentation
- Error handling in playbooks, Ansible Community Documentation
- ansible.builtin.uri module, Interacts with webservices, Ansible Community Documentation