What Kubernetes is and why teams adopt it
Kubernetes is an open-source container orchestrator. You describe the state you want (this image, three replicas, this much memory, exposed on this port) and a set of controllers keep reality matching that description. If a node dies, pods are rescheduled. If load grows, you can add replicas or nodes. If a rollout goes wrong, you roll back to the previous ReplicaSet.
Teams adopt it for consistency more than for raw power. Once everything is a Kubernetes object, deployment, scaling, secrets, configuration and access follow one model, which is what makes a ten-team organisation manageable. Our Kubernetes consulting work starts from that view: the goal is a boring, predictable platform that product engineers rarely have to think about.
It is also a commitment. You are taking on networking, identity, storage, upgrades and a large add-on ecosystem. That is the right trade for many companies and the wrong one for some, and we say so early.
How we design a production cluster
Topology and environments
Start with the smallest number of clusters that meets your isolation needs. For most startups and scale-ups that is one non-production and one production cluster, with namespaces, quotas and network policy separating teams. Add clusters when you need regional presence, regulatory separation or a smaller blast radius for upgrades. Our write-up on multi-cluster strategy for dev, staging and prod covers the decision in more detail.
We build clusters on a managed control plane and define them in Terraform, so a cluster is a reviewed module rather than a hand-tuned snowflake. The cloud-specific differences (identity, networking, upgrade mechanics) are covered on our pages for Amazon EKS, Azure AKS and Google GKE.
Workload settings that matter most
Three settings account for a large share of avoidable incidents:
- Requests and limits. Requests drive scheduling and autoscaling, so missing or inflated values either overpack nodes or waste money. We set them from observed usage rather than guesses. See our approach to requests, limits, HPA and PDB tuning.
- Probes. Readiness probes decide whether traffic is sent to a pod, liveness probes decide whether it is restarted. A badly chosen liveness probe can turn a slow dependency into a restart storm.
- PodDisruptionBudgets. Without them, a node drain during an upgrade can remove every replica of a service at once.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-api
namespace: shop
spec:
minAvailable: 2
selector:
matchLabels:
app: checkout-api
Security and access
Treat the cluster as a multi-tenant system even if all tenants are your own teams.
- RBAC should grant namespaced roles to groups from your identity provider, not cluster-admin to individuals. Review it regularly; least-privilege RBAC is one of the first things we fix on inherited clusters.
- Network policies turn the default allow-all pod network into an explicit allow-list. Start with a default deny per namespace and open only what services need (implementation guide).
- Pod Security Standards stop privileged containers, host mounts and root users from reaching production unless deliberately allowed.
- Secrets should come from a cloud secret manager or sealed, encrypted manifests, and etcd encryption should be enabled on your provider.
- Images should be scanned in CI and signed or at least pinned by digest, from a registry you control.
Cost and capacity
Kubernetes cost problems are mostly capacity problems. Oversized requests leave nodes half empty; undersized ones cause throttling and evictions. We combine right-sized requests, the Horizontal Pod Autoscaler for replicas, and a node autoscaler (Karpenter on AWS, the cluster autoscaler or node auto-provisioning elsewhere) so nodes follow pod demand. Spot or preemptible capacity suits stateless, interruption-tolerant workloads; keep stateful and latency-critical services on on-demand nodes.
Add cost visibility by namespace and label early. Teams that can see their own spend tend to fix it themselves.
Day-2 operations
Upgrades
Kubernetes ships minor releases regularly, and each is supported for a limited window. Plan upgrades as a recurring calendar item: scan for removed APIs, upgrade the control plane, then add-ons, then nodes, one minor version at a time. Our upgrade planning and execution service produces a runbook and runs it with you.
Observability
Metrics, logs and traces answer different questions. We typically deploy Prometheus and Grafana for metrics and alerting, a log pipeline (Loki or a managed equivalent) and OpenTelemetry for traces. Alert on symptoms users feel (error rate, latency, saturation), not every pod restart. See monitoring Kubernetes with Prometheus and Grafana.
Backup and recovery
Stateless workloads are rebuilt from Git. Persistent volumes, databases and cluster-scoped objects such as custom resources need explicit backup, and restores must be rehearsed, not assumed.
Common failure modes
- No resource requests: noisy neighbours and unpredictable evictions.
- Skipped upgrades: a forced multi-version jump under time pressure.
- Cluster-admin everywhere: one leaked credential becomes a full compromise.
- Ingress as a single point of failure: one replica, no disruption budget, no alerts.
- Unbounded add-ons: twenty Helm charts nobody owns, each with its own upgrade path.
A production readiness audit checks for each of these against your real cluster.
When not to use Kubernetes
If you run a few services, have one small team and do not need custom scheduling, a managed container platform such as ECS on Fargate or Cloud Run will cost you less attention. If your workload is mostly event-driven functions, serverless may suit better. If you do move to Kubernetes later, start with one managed cluster and one deployment pipeline, and grow from there. When you are ready to talk specifics, book a consultation and we will give you a straight answer on fit.
Official documentation: kubernetes.io