When one cluster becomes ten
Running one cluster, you can get away with a hand-written Application per service. By the time you have staging and production, two regions, a dedicated cluster for a regulated customer and a handful of ephemeral preview clusters, that approach produces hundreds of files that differ by a single hostname. Someone updates nine of ten, and the tenth quietly stays on the old configuration for a month.
Generators solve this. You describe the pattern once and let the tool stamp it out. This page covers how we use ArgoCD ApplicationSets and Flux Kustomizations for multi-cluster deployments, as part of our GitOps, Helm, ArgoCD and Flux service. If you run on AWS, our EKS consulting often provides the clusters this sits on.
What good looks like
- Adding a cluster is a pull request that registers it and labels it, not a project.
- Adding a service to every cluster, or a subset, is a single change.
- Every cluster’s sync status is visible in one place.
- Changes roll out in a deliberate order with a chance to stop.
- The controller’s access to each cluster is as narrow as practical.
How it works in practice
ArgoCD ApplicationSet with the cluster generator
Registered clusters carry labels. The generator selects those that match, and the template fills in cluster-specific values:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: platform-addons
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
env: production
template:
metadata:
name: "addons-{{name}}"
spec:
project: platform
source:
repoURL: https://github.com/acme/fleet-config.git
targetRevision: main
path: platform/overlays/{{metadata.labels.region}}
destination:
server: "{{server}}"
namespace: platform
syncPolicy:
automated:
prune: true
selfHeal: true
Every production cluster now receives the platform add-ons, and the overlay folder is chosen from the region label.
Git and matrix generators
The git directory generator creates one Application per folder, so a new service folder is deployed the moment it is merged. The matrix generator crosses two sources, for instance every cluster with every app folder, with a selector to limit combinations. These are powerful and easy to overuse: a careless matrix can create hundreds of Applications and put load on the repo server, so we test and size accordingly.
Flux across a fleet
In Flux the pattern is a repository with a shared base and a folder per cluster. Each cluster’s Flux reconciles its own folder, and Kustomizations declare ordering:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 10m
path: ./apps/overlays/prod-eu
prune: true
sourceRef:
kind: GitRepository
name: fleet
dependsOn:
- name: platform
postBuild:
substitute:
CLUSTER_NAME: prod-eu
dependsOn makes applications wait for the platform layer, and postBuild substitution injects cluster-specific values. Bootstrapping a new cluster is the same bootstrap command pointing at a new path.
Onboarding clusters as code
A fleet is only as manageable as its onboarding. We make cluster creation, registration and labelling part of the same pipeline, usually tied to Terraform or Cluster API. For ArgoCD, cluster credentials are stored as labelled secrets and created from code. The new cluster appears, picks up its labels, and the ApplicationSets deploy add-ons and baseline applications on their own. The overall layout is explained in repository structure design.
Rollout order and blast radius
A generator makes it as easy to break ten clusters as one. Our defaults:
- A canary cluster in each environment tier receives changes first.
- Waves follow, defined by labels, with health checks between them.
- ApplicationSets can use a progressive sync strategy to update clusters in steps; where that is not suitable, we gate by changing the version reference per group.
- Alerts on failed syncs stop humans from promoting the next wave.
For application-level traffic shifting inside a cluster, see progressive delivery.
Decisions and trade-offs
Central controller or one per cluster. Central is easier to see and operate. Per-cluster is safer and works across network boundaries. Many teams use one for production and one for everything else.
Pull versus push for remote clusters. ArgoCD pushes from the hub using stored credentials; Flux per-cluster pulls from git and needs no inbound access. Air-gapped or locked-down networks usually favour pull.
When not to use generators. With three clusters and a stable set of apps, explicit manifests are easier to read and debug. Introduce generators when repetition causes real errors.
A worked example: adding a new region
A client opens an environment in a second region. Without generators, an engineer copies thirty Application manifests, edits server addresses and hopes nothing is missed. With an ApplicationSet driven by a cluster label, the process is shorter and safer. The new cluster is registered with labels such as env: production and region: eu-west. The generator sees it, the template stamps out every application that targets production clusters, and the platform components arrive first because of sync waves or a dependency ordering we set up in advance.
The review that matters happens before registration: does the cluster have the base add-ons, the right network access to the repository, and quotas that match the workloads? We turn that into a checklist, then into code, so onboarding becomes a pull request instead of a project.
Per-cluster differences
Clusters rarely run identical config. Ingress hostnames, replica counts and cloud account identifiers differ. The cluster generator can pass labels and annotations into the template, and a values file per cluster or per region can be selected with the same variables. This keeps the shared definition in one place and the differences in small, reviewable files.
Common failure modes
- Deleting an ApplicationSet deletes its Applications. We set deletion policies and protections so a mistaken change does not remove workloads.
- Template sprawl. One ApplicationSet doing too many jobs becomes unreadable. Split by concern.
- Credential sprawl. Cluster secrets proliferate. We rotate and scope them.
- Silent skew. One cluster stuck on an old revision goes unnoticed. Fleet dashboards show revision per cluster.
Delivery and handover
We deliver the generator definitions, cluster onboarding procedure, rollout waves, dashboards for fleet status and a runbook for adding, draining and removing a cluster. Your platform team adds a test cluster themselves during handover. For a full view of observability, our monitoring service can extend the same dashboards.
How to tell it is working
A new cluster is fully configured within minutes of registration. Cluster differences are explainable from labels and overlays. A bad change is caught in the canary wave. To discuss your fleet, get in touch.