Encrypting Kubernetes Secrets on EKS with AWS KMS

What a Kubernetes Secret actually is

A Kubernetes Secret is an object whose values are base64 encoded. That is formatting, not protection. Anyone who can read the object can decode it, and anyone who can read the underlying etcd data or a snapshot of it can see every database password in the cluster.

This page covers how we protect secrets on EKS as part of AWS EKS consulting. Encryption is one of the findings checked in a CIS EKS Benchmark review, and it works together with least-privilege IAM.

What good looks like

• Secrets in etcd are encrypted with a customer managed KMS key.
• Key policy is narrow: named administrators, the cluster’s service role and nothing else.
• Volumes are encrypted, and the same key discipline applies to them.
• Real secret values are held in AWS Secrets Manager, Parameter Store or a similar store, and never in Git or CI logs.
• Only the workloads and people who need a secret can read it.

Envelope encryption on EKS

With envelope encryption, Kubernetes generates a data encryption key for each secret write, encrypts the secret with it, and the data key is itself encrypted by your KMS key. The result is that reading raw etcd content without KMS access yields nothing useful, and every use of your key is recorded in CloudTrail.

In Terraform, enabling it on a cluster looks like this:

resource "aws_kms_key" "eks_secrets" {
  description             = "EKS secrets encryption for platform-prod"
  enable_key_rotation     = true
  deletion_window_in_days = 30
}

resource "aws_eks_cluster" "this" {
  name     = "platform-prod"
  role_arn = aws_iam_role.cluster.arn

  encryption_config {
    resources = ["secrets"]
    provider {
      key_arn = aws_kms_key.eks_secrets.arn
    }
  }

  vpc_config {
    subnet_ids = var.private_subnet_ids
  }
}

If you use the community EKS module, the equivalent is its encryption inputs. Check the module version you use, as input names changed between releases. The cluster’s role needs permission to use the key, and the key policy must allow it.

Things that catch teams out

• It is one way. Once enabled, envelope encryption cannot be disabled. Plan key ownership before switching it on.
• Key deletion is catastrophic. If the key is deleted after its waiting period, secrets in the cluster become unreadable. We set a long deletion window, restrict kms:ScheduleKeyDeletion, and alert on it.
• Rotation of the KMS key material is automatic once enabled, but this rotates the key backing, not the secret values themselves.
• Existing secrets are re-encrypted when written. A one-off rewrite of all secrets makes sure none stay in the old form: kubectl get secrets -A -o json | kubectl replace -f -, tested first in a non-production cluster because it touches every secret.

Encryption does not equal access control

Envelope encryption protects data at rest. The API server decrypts secrets for any authorised caller, so a user or service account with get on secrets can read them. We therefore review RBAC: who has cluster-wide secret read, which controllers do, and whether wildcard rules grant it by accident. Many clusters give far more access than anyone realises.

Keep real secrets out of Git

Secrets that live only in the cluster are hard to rotate and easy to lose. Three patterns work on EKS:

Our usual recommendation is External Secrets for most teams because rotation then happens in AWS and the cluster follows. If you rely on GitOps tooling, see our GitOps with Argo CD on EKS tutorial and the Sealed Secrets glossary entry. Access to the AWS side uses IRSA or Pod Identity, scoped to specific secret ARNs.

Volumes and other data at rest

Secrets are not the only sensitive data. We set the default EBS encryption for the account, define StorageClasses with encrypted: "true" and an explicit kmsKeyId where a customer managed key is required, and review S3 buckets and RDS instances used by workloads. The EBS CSI driver role needs access to the key, a detail covered in add-on management.

A worked example: a database password

Take a database password used by one service. In a weak setup it sits in a Helm values file, gets committed, and is copied into a Secret by hand. In the setup we build, the value lives in Secrets Manager. An ExternalSecret resource in the application’s namespace names the secret and the key, and the External Secrets Operator creates the Kubernetes Secret from it. The operator’s role can read only that secret ARN, and the application mounts the resulting Secret as usual.

Rotation then becomes an AWS task. The value changes in Secrets Manager, the operator refreshes the Kubernetes Secret on its interval, and the application picks it up on restart or reload. We confirm how your application handles a changed value, since some read credentials only at startup and need a rollout to refresh.

Recovery planning

Encryption adds a dependency: if the key is lost, the data is lost. We document who can administer the key, how to confirm the cluster role still has access, and what to do if the key policy is edited by mistake. The recovery steps are tested in a non-production cluster rather than assumed to work.

When this is not the priority

If your cluster has an openly reachable API server, shared admin credentials or no pod isolation, fix those first. KMS encryption is cheap and worthwhile, but it addresses a narrower threat than exposed endpoints or over-broad RBAC. We rank work by risk during assessment.

Delivery and handover

You get the key and policy in Terraform, envelope encryption enabled, a tested rotation and recovery procedure, secret management wired to your chosen store, and a checklist for onboarding a new secret. Your engineers learn how to add a secret without ever pasting it into a manifest.

How to tell it is working

Describe the cluster and confirm the encryption configuration is present. In CloudTrail, KMS Decrypt and GenerateDataKey calls from the cluster role should appear. No plaintext credentials should be found by a secret scanner across your repositories. To review your setup, contact us.

References

• Encrypt Kubernetes secrets with KMS on existing clusters, Amazon EKS documentation
• Encrypting Confidential Data at Rest, Kubernetes documentation
• Data encryption and secrets management, Amazon EKS Best Practices Guide