Why this is the highest-return Kubernetes tuning
Almost every unstable cluster we see has the same root: pods that never told Kubernetes what they need. Without requests, the scheduler packs workloads blindly. Without sensible limits, one leak takes a node down. Without disruption budgets, a routine upgrade becomes an outage. Getting kubernetes resource requests and limits right is cheap, and it improves stability, cost and upgrade safety at once.
This is a core part of our Kubernetes consulting work, and it is one of the first things we check in a production readiness audit.
Requests and limits, in plain terms
A request is a reservation. The scheduler only places a pod on a node with enough unreserved capacity. A limit is a ceiling enforced at runtime, and it behaves differently per resource:
- Memory over the limit: the container is OOM killed and restarted.
- CPU over the limit: the container is throttled, which shows up as latency, not a restart.
This asymmetry drives our policy. Memory is not compressible, so we set memory requests equal or close to limits and leave headroom above observed peak. CPU is compressible, so requests reflect typical use, and we think carefully before adding hard CPU limits to latency sensitive services.
Requests also determine the quality of service class. Pods where requests equal limits for every container are Guaranteed and are evicted last. Pods with no requests are BestEffort and are evicted first, which is rarely what you want for production.
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 512Mi
Here memory is pinned and CPU is left unlimited, a common pattern for web services on shared nodes.
How we pick the numbers
Guessing is the failure mode. We query history instead:
# 95th percentile CPU use per container over two weeks
quantile_over_time(0.95,
rate(container_cpu_usage_seconds_total{container!=""}[5m])[14d:5m])
# Peak working set memory per container
max_over_time(container_memory_working_set_bytes{container!=""}[14d])
We include deploy windows and any known peak events. Requests come from a high percentile of CPU and from peak memory plus margin. The result is reviewed with the owning team, because an engineer knows about the month-end batch job that a two-week window missed. The Prometheus and Grafana setup makes this review repeatable.
Horizontal Pod Autoscaler
The HPA adds or removes replicas based on a metric, most often CPU utilisation as a percentage of requests. That last part matters: with wrong requests, the percentage means nothing.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 65
behavior:
scaleDown:
stabilizationWindowSeconds: 300
Points we tune: a minReplicas that survives a zone loss, a target low enough to absorb a spike while new pods start, and a scale-down stabilisation window so replicas do not flap. For queue consumers or event driven work, scaling on queue depth with KEDA often beats CPU.
Vertical Pod Autoscaler
The VPA observes usage and recommends, or sets, requests. Its automatic mode restarts pods to apply changes, which many production teams avoid. We usually run it in recommendation mode, feed its suggestions into pull requests, and keep humans in the loop. Do not point HPA and VPA at the same CPU metric on the same workload.
Pod disruption budgets
A pod disruption budget limits how many pods of a workload can be voluntarily evicted at once, for example during a node drain or cluster upgrade.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api
spec:
maxUnavailable: 1
selector:
matchLabels:
app: api
It protects availability but does nothing for crashes or node failures. And it has a sharp edge: a workload with one replica and minAvailable: 1 can never be evicted, so upgrades stall. We pair each PDB with a replica count that makes it satisfiable, then test a real drain. This directly affects upgrade planning.
Worked example: sizing one service end to end
Take an API that runs three replicas and was deployed with no resources set. We start by watching it through a normal week and one release. Suppose memory settles in a narrow band with a small spike at startup, while CPU rises and falls with traffic. The memory request and limit are set to the same value, a margin above the observed peak. The CPU request goes at a high percentile of typical use, and we leave out a hard CPU limit.
Next we check the HPA. With the CPU request now meaningful, a utilisation target does what it says, and minimum replicas are set so that losing one zone still leaves the service able to carry load. Finally the PDB is written to allow one pod to be evicted at a time, and we drain a node in a quiet window to watch it hold. Each step is a small, reviewable pull request, so a mistake is easy to revert.
Signals that sizing has drifted
Sizes are not set once and forgotten. Traffic grows, code changes and runtimes update. We watch for restarts with an OOM reason, sustained throttling, pods whose usage sits far below their request, and HPAs that sit at their maximum for long periods. Any of these is a prompt to revisit that workload.
Trade-offs and when not to automate
- Tiny, steady workloads: a fixed replica count and a modest request may beat an HPA you must now tune.
- Spiky traffic with slow startup: HPA reacts after load arrives. Pre-scaling on a schedule, or faster startup, matters more than clever targets.
- Node layer: pod autoscaling needs capacity. Pair it with Karpenter or the Cluster Autoscaler, otherwise scaling pods just produces pending pods.
- Stateful workloads: scaling databases is not an HPA problem, and PDBs for them need special care.
Common failure modes
- Requests copied from a blog post and never revisited.
- Memory limits below the language runtime’s real needs, causing repeated restarts under load.
- HPA max set so high it exhausts a database connection pool before it helps.
- PDBs on every workload with identical values, regardless of replicas.
- Throttling mistaken for application slowness. Check
container_cpu_cfs_throttled_periods_totalbefore tuning code.
Delivery and measurement
We deliver tuned manifests as pull requests, a dashboard that shows requested versus used resources per workload, and a short guide for setting values on new services. You will know it works when OOM kills and throttling alerts fall, node utilisation rises without hurting latency, and a node drain completes without user visible errors. Cost impact follows from better packing, which our EKS cost calculator can help you estimate. For a review of your own settings, talk to us.