Kubernetes Upgrade Runbook | Cloud DevOps Solutions

You can upgrade a Kubernetes cluster without downtime if you do four things in order: find removed APIs before you touch anything, upgrade the control plane one minor version at a time, replace nodes gradually behind well-tuned PodDisruptionBudgets, and verify at each step with a clear stop condition. The mechanics differ between EKS, AKS and GKE, but the runbook is the same. Most outages during upgrades come from skipped preparation, not from Kubernetes itself.

This guide avoids quoting specific version numbers or support windows, because they change several times a year. Always confirm the current policy in your provider’s documentation before you plan.

Why teams fall behind

Nobody decides to run an unsupported cluster. It happens gradually:

The cluster works, so upgrading feels like risk with no visible reward.
Each skipped release makes the next one bigger, because more APIs have been removed and more add-ons need to move together.
Applications were deployed years ago by people who have left, and nobody is sure what uses a deprecated API.

The cost of falling behind is real. Providers eventually stop supporting old versions, some will start charging extra for extended support, and security fixes stop arriving. Worse, a team that is two or three minor versions behind has to do two or three upgrades back to back, under pressure, which is exactly when mistakes happen.

Plan before you touch the cluster

Set a version policy

Write down a policy such as “production runs within one minor version of the newest version our provider supports, and never on a version approaching end of support.” The exact target is your choice. What matters is that it is written, owned and reviewed, so the upgrade is a scheduled task rather than a decision made during a crisis. The Kubernetes upgrade planner can help you lay out the sequence and the checks for your own environment.

Check for deprecated and removed APIs

Every minor release can remove APIs that were deprecated earlier. Manifests that use a removed API will fail to apply, and controllers that call one will break. Scan both what is running and what is in your repositories:

# Helm releases only (use your target version, for example k8s=v1.xx.0)
pluto detect-helm -o wide -t k8s=v1.xx.0

# Scan live cluster objects and Helm release manifests
pluto detect-all-in-cluster -o wide -t k8s=v1.xx.0

# Scan manifests stored in Git
pluto detect-files -d ./manifests -t k8s=v1.xx.0

# Alternative: kubent checks the live cluster for deprecated API usage
kubent -t <target-major.minor>

Pluto and kubent are both open source, and both read from the cluster rather than change it. Pass the target version explicitly with the flags shown, because without them the tools check against a default or the version the cluster runs now. Treat the output as a backlog to clear before the upgrade. Note that kubent’s rule set has lagged behind new Kubernetes releases in the past, so confirm it covers your target version, and prefer Pluto when it does not.

Check add-on and tooling compatibility

The control plane is only part of the picture. List everything that talks to the API server or runs on nodes: the CNI plugin, CoreDNS, kube-proxy, the CSI storage drivers, the ingress controller, cert-manager, the service mesh, the metrics stack, policy engines and autoscalers. For each, find the release that supports your target version and note whether it must be upgraded before or after the control plane. On EKS the managed add-ons give you a compatibility list per version, and AKS and GKE publish similar guidance. If you deploy add-ons through Helm, pin chart versions in Git so the upgrade is a reviewable diff.

Rehearse in staging first

A staging cluster only helps if it resembles production: same provider, same add-ons, same admission policies and similar workloads. Upgrade it first, run your normal smoke tests, and keep it on the new version for long enough to see a few deploys and a few scaling events. Time the node rollout, because that duration becomes your production change window estimate.

Control plane first, one minor version at a time

Kubernetes tolerates version skew between components only within a documented range, and the control plane must never be older than the nodes. So the order is fixed: control plane, then add-ons as required, then nodes.

Upgrade the control plane one minor version at a time. EKS and GKE require one minor version at a time, and AKS does too, except that clusters on long-term support (LTS) may skip to a higher LTS version. Even where a tool lets you chain versions, doing so removes your chance to verify between steps. If you are several versions behind, plan a series of separate upgrade windows, not one heroic weekend.

Aspect	EKS	AKS	GKE
Control plane upgrade	Triggered by you, runs in place	Triggered by you, optionally control plane only	Automatic on a release channel, or manual
Node upgrade	Managed node groups, self-managed groups, Karpenter	Per node pool, with max surge	Per node pool, surge or blue-green
Scheduling control	Your own pipeline and timing	Planned maintenance and auto-upgrade channels	Maintenance windows and exclusions
Downgrade	Rollback by one minor version within 7 days, with conditions	No supported control plane downgrade path documented	Patch downgrade; minor rollback only during a two-step upgrade soak period

Behaviour and flags change, so check the current documentation for each. The command shapes look like this:

# EKS: start the control plane upgrade (asynchronous), then track it
aws eks update-cluster-version --name prod --kubernetes-version <target>
aws eks describe-update --name prod --update-id <update-id>

# AKS: upgrade the control plane only, leaving node pools alone for now
# (not available when cluster auto-upgrade is enabled)
az aks upgrade -g rg-prod -n prod --kubernetes-version <target> --control-plane-only

# GKE: upgrade the control plane (master) only
gcloud container clusters upgrade prod --master --cluster-version <target> \
  --location <region-or-zone>

A control plane upgrade on a managed service is generally non-disruptive to running workloads, but not risk free. Your applications keep running, while controllers, webhooks and anything that calls the API may behave differently. Check that admission webhooks still respond, because a broken webhook with a fail-closed policy can block deployments across the cluster. Hold at this stage, verify, and only then proceed to nodes.

Node rollout strategies

Nodes are where capacity is actually replaced, so this is where downtime risk lives. Each strategy trades speed, cost and rollback ability differently.

Managed node groups and surge upgrades

EKS managed node groups, AKS node pools and GKE node pools all support rolling replacement. The key setting is how many nodes may be replaced at once. Surge (extra nodes created first) keeps capacity steady at the price of some temporary cost. Allowing unavailable nodes is cheaper but removes headroom.

# AKS example: surge one extra node at a time (the default; Microsoft recommends 33% for production)
az aks nodepool upgrade -g rg-prod --cluster-name prod -n apps \
  --kubernetes-version <target> --max-surge 1

On GKE the equivalent settings are surge and unavailable counts on the node pool. On EKS, the managed node group update configuration sets the maximum number of unavailable nodes (a count or percentage). Start conservatively. A rollout that takes longer is better than one that evicts too much at once.

Blue-green node pools

Create a new node pool on the target version, cordon the old pool, and drain it node by node, watching workloads settle on the new pool. If something is wrong, you uncordon the old pool and shift back. This costs extra capacity for the duration, but it is the only strategy that gives you a real rollback for the data plane. GKE also offers a built-in blue-green mode for node pool upgrades; on EKS and AKS the usual approach is a second node group or pool.

Karpenter drift

If you run Karpenter on EKS, nodes are replaced through drift: when the AMI selected by an EC2NodeClass changes, Karpenter marks old nodes as drifted and replaces them. Control the pace with disruption budgets:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "0 9 * * mon-fri"
        duration: 8h

This allows up to ten percent of nodes to be disrupted at a time and blocks voluntary disruption (drift, consolidation, emptiness) from 09:00 to 17:00 UTC on weekdays. Karpenter schedules are always UTC, and budgets do not stop expiration, spot interruption or manual node deletion. Field names have shifted between Karpenter releases, so verify against the version you run. Karpenter respects PodDisruptionBudgets when draining, unless you set terminationGracePeriod on the NodePool, which brings us to the next piece.

PodDisruptionBudgets and graceful termination

Draining a node evicts its pods through the Eviction API, and eviction honours PodDisruptionBudgets. Without a budget, a drain can take every replica of a service offline at once. With a budget that is too strict, the drain hangs forever.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout

Pair that with a deployment that can actually shut down cleanly:

Run at least two replicas, spread across nodes and zones with topology spread constraints.
Use readiness probes so new pods receive traffic only when ready.
Add a short preStop hook (a brief sleep is common) so load balancers deregister the pod before it stops accepting connections.
Set terminationGracePeriodSeconds long enough for in-flight requests and queue consumers to finish.

Audit for budgets that block upgrades: minAvailable equal to the replica count, or a single-replica workload with maxUnavailable: 0, will stall the drain. Fix these before the upgrade, not during it.

Stateful workloads

Databases, queues and anything with persistent volumes need extra care.

Take and test a backup or snapshot immediately before the upgrade. Do not assume a restore works; prove it.
Confirm your CSI driver version supports the target Kubernetes version and upgrade it first if needed.
Remember that block volumes are usually tied to one availability zone. Replacement nodes must exist in the same zone as the pod’s volume, or the pod will stay Pending.
For StatefulSets, rely on ordered, one-at-a-time updates, and consider the partition setting in the rolling update strategy to canary a single pod.
Where the database is replicated, check replica health and replication lag before each node drain, and drain one member at a time.

Rollback realities

Be honest about what you can undo. Rollback of the control plane is limited: EKS allows one minor version within 7 days, GKE only during a two-step upgrade soak window (plus patch downgrades), and AKS documents no downgrade path. Treat it as a one-way door unless you have confirmed the rollback conditions beforehand. Your dependable rollback options are:

Nodes: with blue-green pools, shift back to the old pool. With rolling upgrades, you can create new nodes on the older version only while it remains within the allowed skew from the control plane.
Workloads: roll back deployments and Helm releases to previous revisions.
Total failure: rebuild from infrastructure as code and restore state from backups into a new cluster.

That last point is the argument for treating clusters as reproducible. If you can recreate a cluster from code and restore data from backups, a failed upgrade is an inconvenience rather than a disaster. Because control plane rollback is so constrained, the planning and staging steps above are your safety net, not the rollback.

Verify and monitor during the upgrade

Define success before you start, and define the conditions that make you stop.

Before: all nodes Ready, no crash-looping pods, no pending pods, a green baseline for error rate and latency on key services, and a fresh backup.

During: watch the following, ideally on one dashboard.

Request error rate and latency per service, from your metrics stack.
Pods stuck Pending or Terminating, and PDB violations in drain logs.
API server error rates and admission webhook latency.

Stop and investigate if error rate rises beyond your agreed threshold, if drains stall, or if add-ons crash. Pausing between node batches costs minutes. Pushing on with a half-broken cluster can cost a night.

After: re-run the deprecation scanners, confirm every node reports the target version, and record what happened, how long it took and what surprised you.

Make it a repeatable cadence

The goal is to make upgrades dull. A workable cadence looks like this:

Review the provider’s release calendar each quarter and schedule the next upgrade in advance.
Run deprecation scans in CI so removed APIs are caught when a manifest is written, not when a cluster is upgraded.
Keep add-on versions in Git, and update them on a regular schedule, not only when the cluster moves.
Upgrade non-production first, soak, then production in a fixed window.
Keep this runbook in the repository, and improve it after every run.

When this advice does not apply

Fully managed modes such as GKE Autopilot handle much of the node side for you. You still own deprecated APIs and workload readiness, but the node strategies above are largely the provider’s job.
Automatic upgrade channels move your cluster on the provider’s schedule, so your work shifts to maintenance windows and workloads that tolerate disruption.
Small or disposable clusters can be recreated on the new version instead of upgraded in place.
Self-managed or on-premises clusters with tools like kubeadm have extra steps, such as etcd backup and manual control plane node upgrades, that this guide does not cover.
Single-replica or stateful systems without redundancy cannot be upgraded with zero downtime, however careful you are. You need to add redundancy first or accept a maintenance window.

Where we can help

If you would rather not rehearse this for the first time on production, we run Kubernetes upgrades as part of our Kubernetes and EKS production readiness review, which covers version policy, add-on compatibility and rollout design. You can sketch your own sequence with the free Kubernetes upgrade planner, and our technology pages for Amazon EKS, Azure AKS and Google GKE describe how we work on each platform.

A Practical Runbook for Upgrading Kubernetes Clusters Without Downtime