How to Separate Dev, Staging and Production Across Kubernetes Clusters

The question behind the question

When teams ask how many clusters they need, they are really asking how much damage one mistake is allowed to do. A multi-cluster Kubernetes strategy answers that deliberately. Without one, clusters multiply by accident: a team creates one for a proof of concept, it becomes load bearing, and a year later nobody is sure what runs where.

This page covers how we advise on dev, staging and production separation as part of our Kubernetes consulting work, including when a single cluster is the better answer.

What goes wrong with the wrong layout

Everything in one cluster. A runaway staging job consumes memory and evicts production pods. A cluster upgrade needed by one team forces risk on all of them. A leaked credential in dev reaches production secrets because RBAC was never tightened.

A cluster per everything. Twelve clusters, each upgraded by whoever remembers, each with slightly different add-on versions. Security fixes land unevenly, monitoring has gaps, and cost climbs because every cluster has fixed overhead.

Both are failures of intent, not of technology.

Namespaces versus clusters

A namespace is a soft boundary. It scopes names, and lets you attach quotas, access rules and network policy. It does not give separate nodes, a separate control plane or isolation from cluster-wide objects.

If you cannot tolerate a bad day in one area affecting another, use a separate cluster. If you simply want order, use namespaces.

Layouts we recommend, and when

1. Two clusters: production and non-production. The default for small and mid-sized teams. Dev and staging share a cluster, separated by namespace and ideally dedicated node pools.
2. Three clusters: dev, staging, production. Worth it when you need to rehearse cluster-level changes in staging before production, or when dev is chaotic enough to threaten staging stability.
3. Production split further. By region for latency or resilience, by business unit, or by compliance scope. Each split needs a clear owner.
4. A sandbox cluster. Disposable, torn down on a schedule, for experiments that should never touch real environments.

We do not recommend a design before understanding team structure. A platform team serving ten product teams is a different problem from a team of six engineers.

Making environments actually comparable

Staging that differs from production gives false confidence. We enforce parity on the things that cause surprises:

• Same Kubernetes minor version and add-on versions (ingress, CNI, DNS, autoscaler).
• Same policies: network policy, admission rules and RBAC patterns.
• Same deployment mechanism, so the path to staging is the path to production.

Differences are allowed in size, replica counts, and data volume. A convenient pattern is a shared base with per-environment overlays:

# overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - target:
      kind: Deployment
      name: api
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 2

Overlays like this are the backbone of GitOps deployment with Argo CD or Flux, where each cluster tracks its own directory in Git.

Promotion, access and cost

Promotion. A change is merged, built once, and the same image digest moves from dev to staging to production. Environment specific values live in configuration, never in a rebuilt image. Approvals or automated checks gate the production step.

Access. Developers usually get broad rights in dev, read access in staging and tightly controlled access in production. Our RBAC design work covers how to express that without hand-edited bindings.

Cost. Non-production clusters can scale down outside working hours, and quotas stop one team from consuming the pool. Our EKS cost optimisation guide lists the levers that apply on AWS. Fixed per-cluster costs, such as control plane fees and monitoring, are real, so count them before adding clusters.

Worked example: a growing team deciding on a third cluster

Imagine a company with one production cluster and one shared non-production cluster. Dev and staging are separate namespaces, and the arrangement has worked for a year. Then the platform team plans a Kubernetes minor upgrade and realises they cannot rehearse it safely: any change to the shared non-production cluster affects developers and staging tests together.

That is a concrete reason for a third cluster, and it is a different reason from “staging should be separate”. The new cluster can be smaller and can follow the same Terraform module as production, so it works as the rehearsal ground for cluster-level change. Dev stays on the shared cluster. We write down the trigger that justified the split, so a later request for a fourth cluster is judged against the same standard: what risk does it remove, and who will operate it?

Retiring a cluster

Retirement deserves as much care as creation. We list what still runs there, move or delete it through Git, remove DNS records and cloud load balancers, revoke access, and only then destroy the infrastructure from code. Skipping these steps is how orphaned clusters survive for years.

Trade-offs and when not to split

• Small team, one product: two clusters is usually the ceiling. More is overhead.
• Multi-region: do it for a stated resilience or latency target, not by default. Active-active adds data consistency problems that Kubernetes does not solve.
• Strict compliance: separate clusters often simplify audits, because scope is easier to draw.
• Shared tooling: a separate management cluster for the GitOps controller can help or hurt, depending on how much you trust a single control point.

Common failure modes

• Staging running a different ingress or policy set, so network issues appear only in production.
• Hand-applied hotfixes that exist in one cluster only.
• No retirement process, leaving orphaned clusters that still cost money and still need patching.
• Production credentials reachable from non-production clusters.

Delivery and how to tell it works

We hand over a written topology with the reasoning, Terraform and GitOps structure for every cluster, and rules for adding or removing one. Signs it is working: staging failures predict production behaviour, an upgrade is rehearsed before it reaches production, and you can name the owner and purpose of every cluster. To talk through your layout, book a consultation or see how managed operations can keep several clusters consistent.

References

• Multi-cluster Services, Google Kubernetes Engine documentation
• Multi-region deployment models for Azure Kubernetes Service (AKS), Microsoft Learn
• Argo CD, Declarative GitOps CD for Kubernetes