Ansible OS Patching Automation | Cloud DevOps Solutions

Automating OS Patching with Ansible and Maintenance Windows

Automated patching with Ansible applies operating system updates in controlled rings: check the host, drain it, update packages, reboot only if required, verify services and return it. Schedules and approvals keep every change inside an agreed maintenance window, with a report as evidence.

Key takeaways

Patching is a process problem first: rings, windows, checks and rollback matter more than the module used.

Reboots are the risky part; detect when one is needed and handle services deliberately.

Staged rings (test, then non-critical, then critical) turn the first failures into cheap ones.

Security-only patching and full patching are different policies and should be chosen on purpose.

The output that auditors and managers want is a report of what was patched, when and with what result.

Patching is easy to automate and hard to do safely

Running dnf update on a server is trivial. Running it across two hundred servers without taking down a service, without patching both nodes of a pair together, without a reboot at 2 p.m., and with evidence that it all happened, is a process. Most patching programmes fail on the process: a window is missed, a reboot is forgotten and a kernel vulnerability stays live for months, or an update is applied everywhere at once and the one broken package takes the platform with it.

Ansible is well suited to patching because it already knows your inventory, can inspect hosts, and can sequence actions. This page describes the approach we use within our Ansible configuration management service.

What good looks like

Servers are placed in rings, and patches always flow through them in order.
Each ring has a maintenance window, and automation will not start outside it.
Clustered or paired servers are never down together.
A failed host stops the run and raises an alert, and is not silently skipped.
After every cycle there is a report that can go straight to an auditor.

How it works in practice

Rings and windows

A typical ring design:

Ring	Contents	When
0	Test and canary hosts	Day 1
1	Non-critical production, internal tools	Day 3
2	Customer facing services, one node per group at a time	Day 7, in the agreed window
3	Stateful systems and special cases	Handled with a service-specific runbook

The delay between rings is the safety net: a bad update shows itself in ring 0 or 1. Rings are inventory groups, usually built from tags or facts, which ties directly to inventory design.

The playbook skeleton

- name: Patch ring 1
  hosts: patch_ring_1
  serial: 2
  max_fail_percentage: 0
  become: true

  pre_tasks:
    - name: Confirm services healthy before patching
      ansible.builtin.include_tasks: tasks/precheck.yml

  tasks:
    - name: Apply security updates (RHEL family)
      ansible.builtin.dnf:
        name: "*"
        security: true
        state: latest
      when: ansible_os_family == "RedHat"

    - name: Apply updates (Debian family)
      ansible.builtin.apt:
        upgrade: dist
        update_cache: true
        cache_valid_time: 3600
      when: ansible_os_family == "Debian"

    - name: Check whether a reboot is required (RHEL family)
      ansible.builtin.command: needs-restarting -r
      register: reboot_check
      failed_when: false
      changed_when: reboot_check.rc == 1
      when: ansible_os_family == "RedHat"

    - name: Reboot when required
      ansible.builtin.reboot:
        reboot_timeout: 900
      when: reboot_check.rc | default(0) == 1

  post_tasks:
    - name: Confirm services healthy after patching
      ansible.builtin.include_tasks: tasks/postcheck.yml

For Debian family hosts the reboot check looks for /var/run/reboot-required with ansible.builtin.stat. The dnf module’s security: true option applies security updates only, which is policy decision one: security-only keeps change small, full updates keeps versions current. Many clients use security-only monthly and full updates quarterly.

Pre and post checks

Checks are where the safety lives. Before patching: are disks sufficiently free, is the host in the load balancer, is the cluster healthy, is a recent backup present? Afterwards: are the expected services running, does the health endpoint return success, are there new alerts? For systems behind load balancers we reuse the drain and enable steps from our rolling deployment pattern. For data stores, backup checks before touching anything are non-negotiable.

Maintenance window controls

The window is enforced in three places. AWX schedules start the job only at the right time. A guard task at the top of the play compares the current time to the window and aborts if it is outside. And a monitoring silence is created for the affected hosts so on-call is not paged by planned reboots, then removed afterwards. Alert silencing with Alertmanager must be scoped narrowly, otherwise a real incident hides during patching.

Reporting

Each run records hosts touched, packages updated with before and after versions, reboot status and failures. We use callback plugins or a final task that writes structured results, and AWX keeps the job history. Monthly summaries show compliance with your patch SLA: for example, the share of servers patched within your internal target after a vendor release.

Decisions and trade-offs

Ansible versus native cloud patching. Cloud patch managers are low effort inside a single cloud. Ansible wins in hybrid estates and where custom checks matter.

Automatic updates versus orchestrated. unattended-upgrades and similar tools are simple and fine for low-risk servers. They give you little control over order and no pre-checks, so we use them selectively.

Live patching versus reboot. Kernel live patching reduces reboots but is vendor-specific and does not replace them entirely. We treat it as a bridge, not a policy.

When to rebuild instead. If your servers are stateless and image-based, rebuilding from a patched image is cleaner than patching in place. Patching remains necessary for long-lived hosts.

Common failure modes

Patching both nodes of a pair. Prevented by serial and group design.
Reboot loops or hosts that never return. The reboot module times out and the play stops. We ensure console or out-of-band access exists.
Held or pinned packages. Version locks block updates silently. The run reports them.
Kernel patched, never booted. The system claims to be patched while running the vulnerable kernel. The reboot check and report catch it.
Silence left on. Alerts muted indefinitely. The playbook removes the silence in an always block.

Delivery and handover

You receive the playbooks, ring definitions, AWX schedules (see our AWX deployment), the report format, and a patching runbook for exceptions. Hardening updates complement it: the baseline role controls automatic update settings, and compliance checks verify patch levels.

A note on emergency patches

Plan for the out-of-cycle case before you need it. A critical vulnerability will not wait for the monthly window, so the same playbook should accept a narrow package list and a tighter batch size, and still run through health checks and the approval step.

How to tell it is working

Time from a vendor patch release to full rollout is predictable and within policy.
Patch failures are caught in ring 0 or 1, not in production.
No one is awake at midnight pressing buttons.
The audit report writes itself.

To plan a patching programme for your fleet, contact us.

References

How we deliver it

Classify the estate

Servers are grouped by criticality and tolerance, and clustered services are identified so they are never patched together.

Agree the policy

Windows, ring order, security-only versus full updates, reboot rules and exceptions are written down and approved.

Build the playbook

Pre-checks, snapshot or backup hooks, package updates, reboot handling and post-checks are written as tested tasks.

Schedule with controls

AWX schedules or pipeline triggers run each ring inside its window, with limits on concurrency and a stop on failure.

Verify and report

Service health is confirmed, results are stored, and a per-run report lists hosts, packages and failures.

Review and tune

After each cycle we review failures and adjust checks, windows and exceptions.

Frequently asked questions

Can Ansible patch Linux servers automatically?

Yes. Ansible uses the dnf, yum or apt modules to update packages, the reboot module to restart and wait for the host to return, and checks to confirm services are healthy. Combined with scheduling in AWX or CI, it applies updates in rings inside agreed maintenance windows.

How do you decide whether a reboot is needed?

On RHEL family systems the needs-restarting command with the -r flag reports whether a reboot is required. On Debian and Ubuntu the presence of /var/run/reboot-required signals it. The playbook checks these and only reboots when necessary, or when your policy requires reboots to pick up kernel changes.

Should we use Ansible or the cloud provider's patching service?

Cloud tools such as AWS Systems Manager Patch Manager or Azure Update Manager are good at scanning and scheduling within one cloud. Ansible fits when you span clouds and data centres, need custom pre and post checks, or want one process for everything. Some clients use both, with Ansible handling complex hosts.

How do you avoid patching all nodes of a cluster at once?

Use serial and inventory groups so only one node, or a safe fraction, is updated per batch, and add checks that the cluster is healthy before continuing. For databases and quorum systems we use service-specific tasks such as waiting for replication to catch up before the next node is touched.

What about emergency patches for critical vulnerabilities?

The same playbook runs outside the normal window with a narrower scope, such as one package, and an approval recorded in AWX or the pipeline. Because the process is already automated and tested, an out-of-band patch is quicker and safer than improvising on servers by hand.