Is Your Cluster Ready for Production? A Hands-On Readiness Audit

Why audit before something breaks

Most clusters reach production by accumulation. A deployment works in staging, someone raises a limit to stop an OOM kill, an access rule is loosened for a deadline, and nothing is written down. The cluster is “ready” because it has not failed yet.

A Kubernetes production readiness audit replaces that hope with evidence. It is the first thing we recommend inside our Kubernetes consulting work for teams who inherited a cluster, are about to take real traffic, or have had an incident they do not fully understand.

What a good audit actually looks like

Three properties separate a useful audit from a PDF of scanner output:

1. It looks at behaviour, not only configuration. Does the app survive losing a node? What happens when the ingress pods restart? We test with controlled disruptions where it is safe to do so.
2. It ranks findings. A list of 140 warnings is noise. We rank by how much damage a failure causes and how hard the fix is.
3. It ends in fixes. Remediation is included. Findings that sit in a backlog for months are the most common outcome of reports alone.

The areas we review

Reliability

• Replica counts above one for anything user facing, spread across zones with topology spread constraints.
• Liveness, readiness and startup probes that reflect real health, not just “port open”.
• Requests and limits, autoscaling and pod disruption budgets. We cover this in depth in resource requests, HPA, VPA and PDBs.
• Graceful shutdown: preStop hooks and terminationGracePeriodSeconds matched to how the app drains connections.

Security

• RBAC bindings, especially anything bound to cluster-admin, and service account token use.
• Pod security settings: non-root users, dropped capabilities, read-only root filesystems where practical.
• Network policies and exposure of the API server and dashboards.
• Secrets handling and image provenance. The CIS Kubernetes Benchmark is a useful reference for the control plane and node level checks.

Operability

• Metrics, alerts and logs that would actually tell you what broke. See cluster monitoring with Prometheus and Grafana.
• Runbooks for the failures you can predict.
• Backup of cluster state and persistent data, with a restore someone has really tried. Our backup and disaster recovery service covers this side.

Lifecycle

• Supported Kubernetes version, deprecated API usage and add-on compatibility.
• Whether upgrades are rehearsed or improvised.

How the inspection works in practice

We begin with read-only access and a handful of commands that reveal most of the picture quickly:

# Workloads with a single replica
kubectl get deploy -A -o json | jq -r \
  '.items[] | select(.spec.replicas==1) | "\(.metadata.namespace)/\(.metadata.name)"'

# Containers with no resource requests set
kubectl get pods -A -o json | jq -r \
  '.items[] | . as $p | .spec.containers[]
   | select(.resources.requests == null)
   | "\($p.metadata.namespace)/\($p.metadata.name)/\(.name)"'

# Subjects bound to cluster-admin
kubectl get clusterrolebindings -o json | jq -r \
  '.items[] | select(.roleRef.name=="cluster-admin")
   | .metadata.name + ": " + ([.subjects[]?.name] | join(","))'

Then we run a benchmark scanner such as kube-bench for node and control plane settings, and a workload scanner such as Kubescape or Polaris for manifests. These produce the long list. Our job is the next step: deciding which five items on that list could cause your next outage.

Trade-offs and when not to buy this

• Pre-launch with no workloads yet: an architecture review is a better use of money than an audit, because there is nothing live to inspect. See cluster architecture.
• Perfectly stable clusters with a strong platform team: you may only need a narrow review, such as security or upgrade posture.
• Strict regulatory audits: an independent audit for compliance has requirements our readiness work does not replace. We can prepare the evidence but not certify it.
• Managed control plane caveat: on EKS, AKS or GKE you cannot inspect or change some control plane settings. The audit covers what you control.

Common failure modes we find

• Requests unset, so the scheduler packs pods blindly and one noisy workload starves others.
• A single replica behind a Service with a PodDisruptionBudget of maxUnavailable: 0, which blocks every node drain and, with it, every upgrade.
• Broad cluster-admin grants made “temporarily”.
• Probes that call the same endpoint as the database check, so a database blip restarts every pod at once.
• Unscanned images and unpinned latest tags.
• Monitoring that tracks nodes but not application error rate.

Worked example: ranking what we find

Imagine an audit turns up three things: a customer-facing API running as a single replica, a CI service account bound to cluster-admin, and fifty workloads without resource requests. A flat severity list would treat them equally. We rank on two questions: how large is the blast radius if this fails, and how much effort does the fix take?

The single replica is a likely outage and a quick fix, so it goes first. The cluster-admin binding has a large blast radius but needs care, because narrowing a pipeline’s permissions means finding out what it really calls, so it is scheduled with a test plan. The missing requests are widespread but low urgency individually, so we fix the busiest namespaces first and add a policy that stops new workloads shipping without them. The result is a sequence your team can start on the same week, not a pile of equal-looking warnings.

What the report contains

Each finding states what we saw, why it matters in plain language, the evidence (the command or object involved), the proposed fix, an effort estimate and an owner. We deliberately avoid scoring everything as critical, because a report where every line is red teaches people to stop reading.

Remediation and handover

Fixes arrive as pull requests in your repository with a short explanation each, so your team reviews and learns as they go. For larger items, such as introducing network policy, we stage the rollout (audit mode first, enforce after). You receive the ranked findings, the closed items, the remaining backlog with effort estimates, and a checklist that can run in CI.

How to tell it worked

• The top-ranked findings are closed and verified by rerunning the checks.
• A node drain or rolling restart no longer causes errors visible to users.
• Your team can explain why each guardrail exists.

If you want a second pair of expert eyes on a cluster, book a consultation or take the DevOps maturity assessment first to see where gaps are likely.

References

• Security Checklist, Kubernetes documentation
• CIS Kubernetes Benchmarks, Center for Internet Security
• Best practices, Kubernetes documentation