Upgrades are a deadline, not a project
On a managed platform, Kubernetes versions expire. Providers publish when each version leaves standard support, and after that clusters may be force-upgraded, charged extended support fees, or lose security patches. Teams that treat upgrades as optional end up with a rushed multi-version jump at the worst possible time.
Kubernetes upgrade planning replaces that with a calendar, a checklist and a rehearsal. It is a core piece of our Kubernetes consulting work, and when we operate clusters under Managed DevOps it becomes a routine. If you want the managed version of this, see how we run Kubernetes cluster operations.
What good looks like
- A written upgrade calendar covering every cluster, with target versions and windows agreed months ahead.
- Deprecated API usage is found by tooling in CI, not by a failed deploy.
- Add-on versions are known, pinned and compatible with the target.
- Every upgrade is rehearsed on a cluster shaped like production.
- Rollback triggers are decided before the change starts.
- Upgrading is as boring as deploying.
What actually breaks
Three categories account for most failed upgrades:
Removed APIs. Kubernetes removes beta APIs after deprecation. A manifest or Helm chart using a removed API cannot be applied after the upgrade, and operators using it can break at runtime. Scanning for these is mechanical and should always be done first.
Incompatible add-ons. CNI plugins, CoreDNS, kube-proxy, CSI drivers, ingress controllers, autoscalers, service meshes and operators each have their own supported version matrix. This is the top cause of upgrade surprises.
Workloads that cannot move. Single-replica services, strict pod disruption budgets, pods with local storage and long-running jobs all resist node draining. See PDB design for how to avoid deadlocks.
Pre-flight in practice
# Scan Helm releases and manifests for removed APIs (target version is illustrative)
pluto detect-helm -o wide --target-versions k8s=v1.32.0
pluto detect-files -d ./manifests --target-versions k8s=v1.32.0
# Which APIs has the cluster actually served recently?
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
# Single replica workloads and nodes by version
kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.replicas==1) | "\(.metadata.namespace)/\(.metadata.name)"'
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion
Replace the target with the version you actually plan to reach, and read the upstream changelog and your cloud provider’s release notes for it. The apiserver_requested_deprecated_apis metric is valuable because it reveals live callers, including controllers and CI jobs that static scans cannot see.
Order of operations
- Control plane. One minor version at a time. On EKS, AKS and GKE this is a managed operation, usually minutes to tens of minutes, during which the API remains available but may be briefly slower.
- Add-ons. Update CoreDNS, kube-proxy, CNI and CSI drivers to versions supporting the new release. Cloud-managed add-ons may offer automatic compatible versions, which still need verifying.
- Node pools. Replace nodes with images at the new version. Prefer surge-style replacement: bring up new nodes, cordon and drain old ones, respecting disruption budgets. With Karpenter, drift detection replaces nodes when the AMI changes.
- Workloads. Confirm health and, if needed, update controllers and charts to use newer APIs.
Because tools like Terraform hold the cluster version as a variable, each step is a reviewed change and a plan you can read before applying. GitOps keeps application state consistent across the process.
Across all environments
We never upgrade production first. The path is dev, then staging, then production, with a deliberate pause between each to watch for problems that need real traffic to appear. The design of your environment layout matters here: if staging is a different shape from production, the rehearsal proves less.
Each rehearsal records timings, issues and fixes in a checklist, so the production run is a repeat of something already done twice. We schedule production in a window agreed with your team, with someone watching the dashboards from the monitoring stack throughout, focusing on error rate, latency and pod restarts.
Communicating an upgrade
An upgrade touches every team, so communication is part of the plan. We publish the window, the expected effect and a freeze on unrelated platform changes beforehand, then post short updates as each stage finishes. After the work we share a brief summary of what changed and any follow-up items, so the next upgrade starts with real notes rather than memory.
Rollback reality
You cannot downgrade a managed control plane. State this plainly to stakeholders, because it changes where risk control sits. Prevention (rehearsal, API checks, compatibility testing) is the main defence. After that:
- Node pools: can be reverted to the previous image or version quickly.
- Add-ons: versions can be rolled back via Helm or the managed add-on.
- Application releases: unaffected if deployed through GitOps and reverted in Git.
- Whole cluster: a blue-green approach, building a new cluster and shifting traffic, is the only true rollback for a control plane problem. It costs double capacity for a while and needs care with stateful data.
Trade-offs and when not to do it yourself
- Small stable clusters on automatic upgrade channels may accept the provider’s auto-upgrade for patch versions, while planning minor ones manually.
- Big version gaps are often better handled by a new cluster than by many sequential in-place steps.
- Stateful systems such as databases on Kubernetes deserve their own plan, including backups verified beforehand through your backup and recovery process.
- Auto-upgrading everything trades control for convenience. Choose deliberately.
Common failure modes
- Upgrading on a Friday afternoon because the window was available.
- Skipping add-on checks because the control plane upgrade succeeded.
- Drains blocked by a PDB with
maxUnavailable: 0. - No communication, so a team deploys in the middle of the upgrade.
- Forgetting node pools, leaving them far behind the control plane and out of supported skew.
Delivery and measurement
You receive an upgrade calendar, a repeatable runbook and checklist, pre-flight tooling in CI, the executed upgrades with a short report each, and updated documentation. It works when no cluster is within a few months of end of support without a plan, upgrades produce no user-facing errors, and your team could run the next one from the runbook. To plan yours, book a consultation.