Why upgrades get postponed, and why that hurts
Teams put off Kubernetes upgrades because the last one was stressful. Meanwhile the cluster version ages out of standard support, add-ons fall behind, deprecated APIs pile up in manifests and Helm charts, and the eventual jump becomes bigger and riskier. The cost of delay compounds.
This page describes the upgrade runbook and execution work in our AWS EKS consulting service. We also run focused EKS upgrade projects, and the add-on side is covered in detail in add-on management.
What a good upgrade looks like
- It is boring: planned, rehearsed, completed in a known window.
- Nothing is surprised by an API removal, because manifests were scanned first.
- The order of operations is written down and followed.
- Application teams know when it happens and what to watch.
- Afterwards, the runbook is better than before.
The order of operations
EKS upgrades follow a fixed sequence:
- Pre-flight: check release notes, deprecated APIs, add-on compatibility, node and kubelet skew, and capacity.
- Control plane: upgrade through the EKS API. AWS replaces control plane instances while the endpoint stays available.
- Add-ons: move CoreDNS, kube-proxy, VPC CNI, EBS CSI and others to versions supported by the new release.
- Data plane: replace or update worker nodes so kubelets match the new version.
- Verify: run health checks and application tests, then confirm monitoring is clean.
Kubernetes tolerates a limited version difference between the control plane and nodes, which is why nodes can follow slightly later. Check the current skew policy for your version in the Kubernetes documentation instead of relying on memory.
Pre-flight: the checks that prevent incidents
Deprecated and removed APIs
Each Kubernetes release can remove API versions. A Helm chart or operator still using a removed one will fail to apply after the upgrade. We scan live objects and source repositories with tools such as Pluto or kubent, and use the EKS cluster insights feature, which reports upgrade readiness findings. Anything flagged is fixed before the upgrade, not after.
# scan Helm releases and manifests for deprecated APIs against a target version
pluto detect-helm -o wide --target-versions k8s=v1.31.0
pluto detect-files -d ./manifests --target-versions k8s=v1.31.0
# check which add-on versions are available for the target Kubernetes version
aws eks describe-addon-versions --kubernetes-version 1.31 \
--query 'addons[].{name:addonName,latest:addonVersions[0].addonVersion}'
Workload safety
Rolling nodes only stays quiet if workloads can be drained. We look for single-replica deployments, PodDisruptionBudgets that block eviction entirely, pods without controllers and local storage. A PDB with minAvailable equal to the replica count means a drain never completes. We fix these before the window.
Third-party compatibility
Ingress controllers, service meshes, CSI drivers, policy engines and observability agents each publish a compatibility matrix. We list them all and upgrade those first if the target release needs a newer version.
Node rollout strategies
| Strategy | How | Notes |
|---|---|---|
| Managed node group update | Rolling replacement of nodes with the new AMI | Controlled by update config, respects PDBs |
| Karpenter drift | New AMI triggers node replacement under disruption budgets | Needs sensible budgets and consolidation settings |
| New node group, drain old | Create parallel group, cordon and drain old nodes | More control, temporary extra cost |
| Blue-green cluster | Build a new cluster and move traffic | Highest isolation and highest cost |
For most clusters the managed node group rolling update or Karpenter drift is enough. Blue-green is worth considering for large version jumps, major CNI changes, or when business risk tolerance is very low. See blue-green deployment for the concept.
Rollback honestly
There is no downgrade for a control plane. A runbook that says “rollback: downgrade EKS” is wrong. Real options are limited to rolling back application releases, reverting add-on versions where supported, launching nodes on the previous AMI within skew limits, or failing traffic back to a previous cluster in a blue-green design. We write the actual options for your situation and test the ones that can be tested.
A worked example: a drain that never finishes
During a node rollout, one node sits in a draining state for far longer than the others. Checking the events shows an eviction being refused. The cause is a deployment with two replicas and a PodDisruptionBudget that requires two available, so no pod may ever be evicted. Nothing is broken, but the rollout cannot continue.
The fix is in the budget or the replica count. Raising the deployment to three replicas with a budget of two available lets one pod move at a time. We would find this in pre-flight by listing every budget whose allowed disruptions is zero, which is far cheaper than discovering it mid-window with stakeholders waiting.
Communication and timing
An upgrade is also a coordination task. The runbook names who approves, who executes, who watches dashboards and who talks to the business. We pick a window outside peak traffic, agree a stop point where we pause if checks fail, and send application teams a short note on what to watch. A clear stop condition matters more than speed.
Making upgrades routine
Running through Terraform keeps versions declared in code, so the upgrade is a reviewed pull request. See Terraform modules and state. Many teams then adopt a regular cadence, upgrading non-production soon after each release and production after a soak period. Automating pre-flight scans in CI turns surprises into failed builds, which pairs well with GitOps delivery.
Failure modes we plan for
- Surprise API removals in old charts. Caught by scanning.
- PDBs blocking drains and stalling the rollout.
- Add-on mismatch leaving DNS or networking degraded.
- IP exhaustion when surge nodes launch during rollout.
- Monitoring blind spots, so a regression is noticed by customers first. We keep dashboards open from Prometheus and Grafana during the window.
Delivery and handover
You receive the runbook (steps, owners, timings, checks, rollback), the pre-flight report, and a written record of the executed upgrade. We run the upgrade with your team watching or participating, depending on what you want to learn. For ongoing care see managed DevOps.
How to tell it worked
Workloads are healthy and at expected replica counts, error rates and latency match the pre-upgrade baseline, no add-on is degraded, and the pre-flight scan for the next version is already clean. To plan your next upgrade, book a consultation.
Where this fits
This page is the EKS runbook. For the wider question of keeping every cluster inside its support window, see Kubernetes version upgrades and the free Kubernetes upgrade planner.