The problem with how most teams promote
In many organisations, “promotion” means running a pipeline again with a different target. The build is repeated, a slightly different image comes out, and staging is no longer a faithful rehearsal for production. Environment configuration lives partly in the pipeline, partly in a wiki, partly in someone’s head. When production fails and staging did not, nobody can say exactly what differed.
GitOps gives a cleaner model. Each environment is a place in git, and promotion is a change to that place. This page is the part of our GitOps, Helm, ArgoCD and Flux service that defines how releases move. It depends on a sound repository structure.
What good looks like
- The same image digest or chart version runs in every environment on its way to production.
- Environment differences are small, visible and reviewed: replica counts, hostnames, resource sizes, feature flags.
- A pull request shows exactly what will change in a given environment.
- Production approval is a deliberate step with named owners.
- Reverting a bad release is a git revert, nothing more.
How it works in practice
Kustomize overlays
A base holds the shared definition. Overlays patch only what differs.
apps/payments/
base/
deployment.yaml
service.yaml
kustomization.yaml
overlays/
dev/kustomization.yaml
staging/kustomization.yaml
prod/kustomization.yaml
The production overlay pins the image and adjusts size:
# apps/payments/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: registry.example.com/payments/api
newTag: "1.18.3"
replicas:
- name: payments-api
count: 6
patches:
- path: resources-patch.yaml
Promotion then means changing newTag in the next environment’s file. You can confirm what will be applied with kustomize build apps/payments/overlays/prod before merging, and CI can run the same command on every pull request.
Helm values per environment
With a chart, each environment has a values file, and the controller combines the shared chart with the right one:
# ArgoCD Application (excerpt)
source:
repoURL: https://github.com/acme/fleet-config.git
path: charts/payments
helm:
valueFiles:
- values.yaml
- values-prod.yaml
In Flux the HelmRelease holds values inline or references a ConfigMap. Both patterns work, and the discipline is the same: shared defaults in one place, environment values minimal.
Automating the lower environments
A pipeline builds an image, pushes it, and then updates the dev overlay in the config repository, either by committing directly with a bot identity or by opening a pull request. Staging can follow automatically after dev checks pass. This is where the tags, digests and chart versions flow, linking to chart publishing and to your wider CI/CD pipeline.
Gating production
Production promotion is a pull request that changes only the production version reference. Branch protection requires review from the owning team, and CI posts the rendered diff. Some teams add a staging soak time: a check confirms the version has been healthy in staging for a set period before the pull request can merge. For higher risk changes, combine this with canary or blue-green delivery, so production promotion starts small and widens gradually.
Decisions and trade-offs
Automatic versus pull-request-based updates. Controller-driven image automation reduces tooling but writes to git from inside the cluster, which complicates audit and permissions. A pipeline that opens pull requests is more explicit. We lean towards the pipeline approach for production and accept either for lower environments.
Pin by tag or digest. Tags are readable; digests are immutable. We use tags in git for readability combined with registry tag immutability, or digests where supply chain assurance matters.
Per-environment clusters or namespaces. Separate clusters for production isolate failure and access but cost more. Namespaces on one cluster are cheaper but share a blast radius. Overlays work with either; the choice is covered in our multi-cluster work.
When this is overkill. A product with one production environment and a short-lived preview environment does not need a staged pipeline with soak gates. Start with two environments and add stages when releases begin to cause problems.
A worked promotion, step by step
Take a change to the payments-api image. The build pipeline publishes a new tag and opens a pull request that edits one line in the development overlay. It merges automatically once checks pass, and the controller deploys it within minutes. A scheduled or manual job then opens a second pull request that copies the same tag into staging. Staging runs its smoke tests and soaks for an agreed period. Only then is a third pull request raised for production, assigned to the code owners of that folder.
Each pull request contains a one-line diff, which is what makes review meaningful. Reviewers see exactly which version moves, can check the staging result linked in the description, and approve knowing that rollback is a revert of the same commit. Compare this with promotion by rebuilding: the artefact that reaches production is never the one that was tested.
Promoting config as well as images
Image tags are the easy part. Config changes, such as a new feature flag default or a changed resource limit, should travel the same road. We keep the base identical across environments and let overlays hold only deliberate differences, then review those differences regularly. A quick way to spot accidental divergence is to render both overlays and compare them:
kustomize build overlays/staging > /tmp/staging.yaml
kustomize build overlays/production > /tmp/production.yaml
diff /tmp/staging.yaml /tmp/production.yaml
Anything in that diff should be explainable: replica counts, hostnames, resource sizes, external endpoints. Anything else is drift waiting to cause a surprise on release day.
Common failure modes
- Rebuilding per environment. Breaks the promise that staging mirrors production. Fixed by building once.
- Overlays that copy whole files. They drift. Keep patches small.
- Secrets differing silently. Environment secrets come from a managed source such as Sealed Secrets or External Secrets Operator, with the same structure in every environment.
- Skipped staging under pressure. Hotfix paths are defined in advance, with fewer steps but still reviewed.
- Config changes and code changes in one release. Separating them makes failures easier to diagnose.
Delivery and handover
We deliver the base and overlay structure, the pipeline that updates lower environments, the pull request workflow for production, and CI checks that render and validate every environment. We also run a rehearsal in which your team promotes a real change and rolls it back.
How to tell it is working
Production releases are boring: a small pull request, a visible diff, a clean sync. Staging reliably predicts production. A rollback takes minutes and does not need a meeting. If you would like us to review your current release path, contact us.