Why Terraform and state design matter for EKS
An EKS cluster is not one resource. It is a control plane, an OIDC provider, node groups or Karpenter, IAM roles, security groups, add-ons and access mappings, all connected to a VPC. Built by hand, that estate drifts within weeks. Built with copy-pasted Terraform, it drifts more slowly but just as surely.
This page covers how we build clusters as code as part of our AWS EKS consulting service. For the wider discipline, see Terraform and infrastructure as code and our guide on moving from ClickOps to Terraform on AWS.
What good looks like
- Every environment is created from the same modules with different inputs.
- Module and provider versions are pinned, and upgrades are deliberate.
- State is remote, encrypted, versioned and locked.
- No one runs
terraform applyfor production from a laptop. - Someone who did not write the code can find, read and change any part of it.
Module strategy
The practical choice is between the widely used community module for EKS and your own. The community module covers control plane, managed node groups, access entries, add-ons and security groups, and it is actively maintained. A thin wrapper module in your repository can then enforce your defaults: private endpoint settings, logging, tags, approved instance families and add-on versions.
We avoid two extremes. A giant in-house module that reimplements everything becomes a product you must maintain. A pile of copy-pasted resource blocks becomes a set of slightly different clusters. The wrapper approach keeps the surface small.
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.0"
cluster_name = "platform-prod"
cluster_version = "1.31"
vpc_id = data.terraform_remote_state.network.outputs.vpc_id
subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
cluster_endpoint_public_access = false
enable_cluster_creator_admin_permissions = false
eks_managed_node_groups = {
system = {
instance_types = ["m6i.large"]
min_size = 2
max_size = 4
desired_size = 2
taints = {
critical = { key = "CriticalAddonsOnly", value = "true", effect = "NO_SCHEDULE" }
}
}
}
}
Check the module version, input names and supported Kubernetes versions against current documentation before use, because the module interface has changed between major releases.
State management that survives real teams
The backend
State lives in an S3 bucket with versioning and server-side encryption enabled, public access blocked, and a bucket policy that only allows the pipeline role and named administrators. Locking prevents two applies colliding; recent Terraform versions support S3 native locking, while older setups use a DynamoDB table.
terraform {
backend "s3" {
bucket = "example-org-tfstate"
key = "eks/platform-prod/cluster.tfstate"
region = "eu-west-1"
encrypt = true
use_lockfile = true
}
}
State can contain sensitive values in plain text, so treat read access to it as sensitive. Details on the general practice are in our Terraform state management guide for managed teams.
Layering
We separate state by change frequency and ownership:
- Network: VPC, subnets, endpoints. Rarely changes.
- Cluster: control plane, node groups, OIDC, access entries. Changes occasionally.
- Add-ons and platform: CoreDNS, CNI, controllers, ingress. Changes during upgrades.
- Workload infrastructure: IAM roles per service, queues, buckets. Changes often.
Layers share data through outputs or data sources. A single state for all of it makes plans slow and mistakes large, and a bad edit to workload IAM should never put the control plane in a plan.
A worked example: adding a node group safely
A common change request is a new node group for a different workload class. In a well-structured repository this is a small pull request: a new block in the cluster stack’s inputs, with labels and taints that match the scheduling rules the workload uses. The pipeline posts a plan that shows one node group being created and nothing else being touched. Reviewers check that no existing node group, security group or add-on appears in the plan.
If the plan shows unrelated changes, that is a signal worth stopping on. It usually means drift from a manual edit, an unpinned module version that moved, or a provider upgrade. We resolve the cause before applying, so the pull request does exactly what its title says.
Recovering from a bad state
State problems are rare but stressful, so the recovery steps are written before they are needed. A stuck lock is cleared only after confirming no apply is running. A damaged state file is restored from a previous bucket version. A resource removed from state by mistake is brought back with an import block, reviewed through a plan. We run through at least one of these scenarios with your team during handover so the first time is not during an incident.
Pipeline and review
Pull requests run terraform fmt -check, validate, tflint, a policy scan such as Checkov, and plan. The plan is posted to the pull request. After approval, the pipeline applies it using short-lived credentials via OIDC federation, not stored access keys. Atlantis and plain GitHub Actions both work. See CI/CD pipelines for how this fits your wider delivery process.
Trade-offs and when to do it differently
- Terragrunt reduces repetition across many environments but adds a tool layer. We use it when there are many similar stacks, not for a two-environment setup.
- Importing a hand-built cluster works, but some settings force replacement. We check the plan carefully before anything is applied.
- Helm and Kubernetes resources in Terraform are convenient for bootstrap but awkward for ongoing application delivery. We hand day-two deployments to GitOps with Argo CD or Flux.
Common failure modes
- Wide permissions on the state bucket, exposing secrets inside state. We restrict by role and log access.
- Cluster creator admin left in place, giving an invisible superuser. We use explicit access entries.
- Unpinned module versions, so a routine plan proposes unexpected changes. We pin and upgrade on purpose.
- Manual console fixes that cause drift. We run scheduled read-only plans to catch them.
Delivery and handover
You get the modules in your repository, the backend set up, a pipeline running, and a short operations guide covering state layout, how to add a node group, how to recover a corrupted or locked state, and how upgrades are applied. Your team should be able to change the cluster the day we finish.
How to tell it is working
A fresh environment should come up from code with no manual steps. Scheduled plans should usually show no changes. Pull request plans should finish in minutes. If any of those fail, the state design needs attention. To discuss yours, contact us.