Why cluster operations is where teams struggle
Getting a cluster running is a weekend. Keeping it healthy for years is a job. Managed control planes such as EKS, AKS and GKE remove etcd and API server care, but they do not remove the rest: version upgrades, node images, add-ons, autoscaling, networking and the long tail of security patches. Teams that treat this as a side task fall behind on versions, and an out-of-support cluster eventually forces a rushed multi-version jump.
Our Kubernetes operations work is part of the wider Managed DevOps service. If you need design or migration rather than steady operation, see Kubernetes consulting and AWS EKS consulting.
What good looks like
- Clusters sit on a supported Kubernetes version with an upgrade planned before the old one expires.
- Deprecated API usage is detected before an upgrade, not discovered by a failed deploy.
- Nodes are replaced regularly with patched images rather than lived in for a year.
- Workloads have requests, limits, probes and pod disruption budgets so maintenance is safe.
- Add-ons (CoreDNS, CNI, kube-proxy, ingress, metrics) are versioned and managed as code.
How upgrades run in practice
Pre-flight
Before touching production we check three things. First, API deprecations: a tool such as Pluto or kubent scans manifests and Helm releases for removed APIs. Second, add-on compatibility: each add-on must support the target version. Third, workload readiness: single-replica deployments and missing disruption budgets are flagged.
pluto detect-helm -o wide # scan Helm releases for removed APIs
pluto detect-files -d ./manifests --target-versions k8s=v1.31.0
kubectl get deploy -A -o json \
| jq -r '.items[] | select(.spec.replicas==1) | "\(.metadata.namespace)/\(.metadata.name)"'
Check the target version you actually plan to move to; the version above is illustrative.
Order of operations
Control plane first, then add-ons, then node pools. Managed services only allow one minor version step at a time on the control plane, so skipped versions turn into a sequence of upgrades. Nodes may lag the control plane within the supported skew, but we do not leave them there.
For node pools we prefer replacing nodes over patching in place. On EKS managed node groups, a rolling update with a surge setting brings up new nodes before draining old ones. On Karpenter-managed capacity, drift detection replaces nodes when the AMI changes. Either way, the sequence is cordon, drain, respect disruption budgets, and watch error rate.
Rollback reality
You cannot downgrade a managed control plane. That is why rehearsal and pre-flight checks matter more than a rollback plan. Node pools, by contrast, are easy to revert to the previous image. We state this plainly to clients, because the lack of a control plane downgrade changes how cautious the process must be.
Node pool sizing
Node pools should reflect workload shape, not guesswork. Our approach:
- Read actual usage from Prometheus against pod requests and limits.
- Fix requests first. Oversized requests inflate node count; undersized ones cause noisy neighbours.
- Decide on pool separation. Common reasons: GPU or high-memory workloads, spot-tolerant batch jobs, system add-ons that need stable capacity, and tenant isolation using taints and tolerations.
- Choose an autoscaler: Karpenter for flexible instance selection on AWS, or the Cluster Autoscaler for fixed node groups.
- Keep headroom for rolling updates and failure of one availability zone.
Sizing connects directly to capacity planning and to cost reviews, and our EKS cost optimisation guide goes deeper on the levers.
Security patching
Patching has three layers.
- Nodes: new AMIs or node images are rolled out on a schedule, with an expedited path for critical kernel or runtime CVEs.
- Cluster add-ons: ingress controllers, service meshes and operators publish their own CVEs and often lag.
- Workload images: base images are rebuilt and scanned in the pipeline, so a patched library reaches production through a normal deploy.
We tie this to the CVE and compliance work so findings have one owner.
A typical maintenance calendar
Predictability is half the value. A rhythm we commonly agree with clients looks like this, adjusted to your release cycle and traffic pattern:
- Weekly: review pending node image updates, certificate expiry, failed pods and pending pods, and any capacity warnings from monitoring.
- Monthly: roll patched node images through each pool, review add-on release notes, and check cloud account quotas against growth.
- Quarterly: assess whether a minor Kubernetes upgrade is due, rehearse it in staging, and schedule production in an agreed window.
- On demand: emergency patching when a critical CVE affects the node OS, container runtime or a cluster add-on you run.
None of these steps is exotic. The value is that they happen on time, in order, and with evidence that nothing regressed, rather than being postponed until a cloud provider emails you a retirement notice.
What we need from your team
Cluster care works best with a short list of agreements: which services are safe to restart, which have special draining needs (long-running jobs, stateful workloads such as databases on persistent volumes), and who approves a maintenance window. Stateful workloads deserve extra care: a node drain that is harmless for a stateless API can interrupt a job queue or trigger a long volume re-attach. We record these exceptions in the runbook for each workload.
Trade-offs and when not to do this
Fewer big clusters or many small ones? One shared cluster with namespaces is cheaper to run; separate clusters give stronger blast-radius isolation. For most startups, one production and one non-production cluster is enough.
Self-managed Kubernetes. Running your own control plane gives control but you inherit etcd, certificates and upgrades. We rarely recommend it unless a compliance or on-premises constraint requires it.
Do you need Kubernetes at all? If you run a few containers with modest traffic, a simpler platform such as ECS or a PaaS may carry much less operational weight. We will say so rather than sell you cluster care you do not need.
Common failure modes
- Skipped upgrades. Falling several versions behind turns routine work into a project.
- No disruption budgets. A drain evicts every replica of a service at once.
- Pinned add-ons. An ancient CNI blocks the Kubernetes upgrade.
- Over-trusting autoscaling. Autoscalers react to pending pods, not to unsound requests.
- Ignoring quotas. Cloud account limits halt node creation during an upgrade, so we check them first.
Delivery and handover
We take over through a health review and a written upgrade calendar. All cluster definitions live in Terraform and Helm, changes go through pull requests, and every procedure has a runbook.
How to tell it is working
Look for supported versions with months of margin, no deprecated API warnings, node age under a defined limit, upgrades completed with no user-visible errors, and steady or improving node utilisation. Book a consultation if you would like a cluster health review.