The unglamorous parts that break clusters
When an EKS upgrade goes wrong, the cause is rarely the control plane. It is usually an add-on: DNS resolution slows because CoreDNS is under-scaled, pods stop getting IPs because the VPC CNI is a version behind, or a stateful workload hangs because the EBS CSI driver lost its permissions. These components work quietly until the day they do not.
This page describes how we manage them within AWS EKS consulting. Add-on versions are a central input to the EKS version upgrade runbook, and for the dedicated upgrade service see EKS upgrades.
What good looks like
- Every core add-on is installed, versioned and configured by code.
- Versions are compatible with the cluster’s Kubernetes version, verified before each upgrade.
- Each add-on that calls AWS has its own scoped IAM role.
- Capacity settings, such as CoreDNS replicas and CNI IP warm pools, come from measurement.
- Someone owns the add-on update routine, and it is on the calendar.
The core four
CoreDNS
Provides cluster DNS. Every service call that uses a name goes through it, so latency or failure here looks like a whole-application outage. We scale replicas with load (the cluster proportional autoscaler or a node-local DNS cache are common choices), spread them across zones, and add a PodDisruptionBudget. DNS errors and latency are monitored, since a failing CoreDNS rarely announces itself.
kube-proxy
Programs service routing on every node. Its version should track the cluster version closely and it runs as a DaemonSet, so upgrades roll across nodes. Mode choices (iptables or IPVS) matter at very large service counts, but most clusters should stay on the default unless a measured issue says otherwise.
Amazon VPC CNI
Gives pods VPC IP addresses. This is the add-on with the most design consequence:
- IP exhaustion: each node reserves addresses ahead of need. Settings such as
WARM_IP_TARGET,MINIMUM_IP_TARGETand prefix delegation (ENABLE_PREFIX_DELEGATION) change how IPs are allocated. - Custom networking: puts pods in secondary CIDR subnets when the primary range is tight.
- Network policy: the CNI can enforce Kubernetes network policies when enabled, which ties to network policies and security groups.
Changing CNI settings affects new nodes and sometimes requires replacing existing ones, so we test them in a non-production cluster first.
Amazon EBS CSI driver
Provisions and attaches EBS volumes for persistent volumes. It needs an IAM role and a sensible default StorageClass. We prefer gp3 volumes, set volumeBindingMode: WaitForFirstConsumer so volumes are created in the pod’s zone, and enable encryption, using a customer managed key where required. See secrets and volume encryption with KMS.
Managed add-ons as code
EKS managed add-ons let AWS supply a tested version and let you set configuration through a JSON schema. In Terraform:
resource "aws_eks_addon" "vpc_cni" {
cluster_name = module.eks.cluster_name
addon_name = "vpc-cni"
addon_version = "v1.19.0-eksbuild.1"
service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
resolve_conflicts_on_update = "PRESERVE"
configuration_values = jsonencode({
env = {
ENABLE_PREFIX_DELEGATION = "true"
WARM_PREFIX_TARGET = "1"
}
})
}
The version string above is an example only. You can list the versions available for a given Kubernetes release with aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version 1.31. The resolve_conflicts_on_update setting decides whether your manual edits are kept or overwritten, so choose it consciously. Code structure is described in Terraform modules and state.
Managed add-on or self-managed
| Approach | Good for | Caveat |
|---|---|---|
| EKS managed add-on | CoreDNS, kube-proxy, VPC CNI, EBS CSI, other supported components | Not every add-on or setting is exposed |
| Helm chart through GitOps | Ingress, observability, controllers outside the managed list | You own compatibility and upgrades |
| Raw manifests | Almost never | Drift, no upgrade path |
Our default is managed add-ons for the four above and Argo CD or Flux for everything else. Having one owner per component prevents two tools fighting over the same object.
A worked example: pods stuck on IP addresses
A common call looks like this: new nodes join, but pods stay Pending and events mention a failure to assign an IP. The cause is usually one of three things. The subnets are small and nearly full, the instance type has a low per-node network interface limit, or warm IP settings are reserving addresses that no pod is using. We check subnet free addresses first, then the CNI logs on an affected node, then the add-on configuration.
The fix depends on the finding. Prefix delegation helps when the node limit is the constraint. A secondary CIDR with custom networking helps when the subnet is the constraint. Lowering warm targets helps when addresses are being held idle. Each option has a cost in complexity, which is why we change one thing at a time in a test cluster and watch the allocation metrics before touching production.
Where add-ons sit in the upgrade order
Add-ons are upgraded in step with the control plane, not after the fact. Our usual sequence is: confirm each add-on has a version supported for the target release, upgrade the control plane, then update kube-proxy, VPC CNI and CoreDNS, and finally the storage driver once workloads with volumes have been checked. Pin the previous add-on version in code so that rollback is a one-line change.
Common failure modes
- Skipping add-on upgrades until they are several versions behind and incompatible with a new control plane.
- Editing add-ons in place with
kubectl edit, and losing the change at the next update. - Role on the node, not the add-on, which violates least-privilege IAM.
- Under-scaled CoreDNS causing intermittent timeouts that look like application bugs.
- No default StorageClass or an old gp2 default left in place.
Delivery and handover
You receive Terraform for each add-on, IAM roles, tuned settings with reasons recorded, and a compatibility checklist your team can run before any upgrade. If you would like add-on care to be ongoing, our managed Kubernetes operations cover it.
How to tell it is working
Terraform plans show no add-on drift. Every add-on is at a version supported for the running Kubernetes release. DNS latency and error rates stay flat during upgrades, and no pods wait on IP addresses. To review your current setup, book a consultation.