Secrets in GitOps: Sealed Secrets or External Secrets Operator

The contradiction at the heart of GitOps

GitOps says everything that runs in the cluster should be described in git. Security says passwords must never be in git. Both are right, and the way through is to store something in git that is safe to read: an encrypted blob, or a reference to a secret that lives elsewhere.

This page is how we resolve that for clients, as part of our GitOps, Helm, ArgoCD and Flux service. Background on the object itself is in our glossary entries for ConfigMaps and Secrets and Sealed Secrets.

What good looks like

• No secret value appears in git history, CI logs or chat.
• Secrets are rotated on a schedule without a release.
• Access to read a secret is audited and restricted.
• A rebuilt cluster recovers all its secrets from git plus the secret store, without manual steps.
• Developers can add a new secret without learning cryptography.

Two main approaches

Sealed Secrets

A controller in the cluster holds a private key. You encrypt a Secret locally with kubeseal using the cluster’s public certificate, producing a SealedSecret that is safe to commit. Only the controller can decrypt it.

kubectl create secret generic db-credentials \
  --from-literal=username=app \
  --from-literal=password="$(openssl rand -base64 24)" \
  --dry-run=client -o yaml \
  | kubeseal --format yaml > sealed-db-credentials.yaml

Commit sealed-db-credentials.yaml. The controller turns it into a normal Secret in the target namespace. Sealed Secrets is scoped by name and namespace by default, so a sealed value cannot be moved elsewhere.

Strengths: simple, no external dependency, works anywhere, fully offline. Weaknesses: the secret’s source of truth is the cluster key, not a vault. Rotation is manual, there is no central view of who uses what, and the key must be backed up.

External Secrets Operator

The operator reads a value from a secret manager and writes a Kubernetes Secret. Git contains only the reference:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password

Check which API version your installed release serves, since it has moved between versions. The store is configured once with workload identity, so no access keys are stored.

Strengths: one source of truth, native rotation and audit from AWS Secrets Manager, Azure Key Vault or HashiCorp Vault, and multiple clusters can share the same secret. Weaknesses: a runtime dependency on the vault, and an extra operator to run.

Choosing between them

Flux has built-in SOPS decryption for Kustomizations, which is a solid third option: files are encrypted in git with a KMS or age key, and the controller decrypts at apply time. ArgoCD does not decrypt natively, so teams there tend towards Sealed Secrets or External Secrets Operator.

Operating it well

Identity, not keys

Give the operator access through IAM Roles for Service Accounts on EKS, or the equivalent on AKS and GKE. Scope the policy to a path prefix, such as secrets under prod/payments/*, so a compromise cannot read everything.

Access inside the cluster

Encrypting in git is only half the story. Anyone who can get secrets in a namespace can read the values. We tighten this with Kubernetes RBAC and, for GitOps users, with the access model described here. Developers can deploy without ever reading production secrets.

Rotation and reloads

Changing a secret is not the same as an application noticing. Mounted secret volumes update eventually, but environment variables are only read at startup. We pair secrets with a reloader or a checksum annotation that triggers a rolling restart, and we test rotation end to end before relying on it.

Decisions and trade-offs

Vault dependency. External Secrets Operator makes your cluster rely on the vault’s availability at sync time. Existing Kubernetes Secrets keep working if the vault is unavailable, but new ones cannot be created. We account for that in recovery plans.

Where the key lives. For Sealed Secrets, the private key is your crown jewel. Back it up in a secure store and rehearse recovery.

When not to use either. If a workload can fetch secrets itself, directly from a cloud secret manager using workload identity, it may not need Kubernetes Secrets at all. That removes a layer, at the cost of changing the application.

A worked example: rotating a database password

Take a payments database password. With External Secrets, rotation happens in the cloud secret store, either by hand or by an automated rotation function. The operator refreshes the Kubernetes Secret on its interval, and the application picks up the new value after a reload or restart. Nothing in git changes, and the audit record lives in the secret store.

With Sealed Secrets, rotation means generating the new value, sealing it for the target cluster and namespace, and merging the encrypted file. That is more manual, but it works without any external service, which suits small estates and air-gapped clusters. Whichever you choose, we test the rotation end to end before go-live, including the awkward case where the old and new value must both work for a short overlap.

Common failure modes

• Plain secrets already in history. Rotate them; deleting the file does not remove the history.
• Lost controller key. Fixed with backup and a tested restore.
• Overbroad vault permissions. One compromised pod reads every secret. Use path-scoped policies.
• Secrets in Helm values. They end up in release history. Reference existing Secrets instead, as we do in our chart designs.
• Rotation never tested. The first real rotation becomes an outage.

Delivery and handover

We deliver the controller installed through GitOps, store definitions, example manifests for each secret type, a rotation procedure, a key backup process where relevant, and a migration of your existing secrets with the old credentials rotated. Your team creates and rotates a secret themselves before we finish.

How to tell it is working

A repository scan finds no secrets. A new cluster comes up with every secret restored. Rotation happens by changing one value. Access to secrets is narrow and logged. If you want to check your current setup, contact us.

References

• Secret Management, Argo CD documentation
• Introduction, External Secrets Operator documentation
• Secrets, Kubernetes documentation