What goes wrong without monitoring you can trust
Teams usually find out about cluster problems in one of three bad ways: a customer reports it, a pod has been crash looping for a week, or a node ran out of disk during a deploy. Often there is monitoring, but it is a default Grafana dashboard nobody opens and forty alerts that everyone mutes.
Good Kubernetes monitoring with Prometheus and Grafana is less about installing tools than about deciding what deserves a human’s attention. We set it up as part of our Kubernetes consulting work, and we run it continuously for clients on Managed DevOps. If you want the wider picture, our monitoring service and the Prometheus and Grafana guide go further.
What good looks like
- Alerts page someone only for user impact or imminent impact, and each links to a runbook.
- Everything else is a dashboard or a ticket, not a page.
- A new engineer can answer “is the cluster healthy?” in under a minute.
- Capacity trends are visible before a node group fills up.
- The monitoring stack itself is monitored, so silence is never mistaken for health.
- Dashboards, rules and scrape configs are in Git and reviewed like code.
The stack we deploy
For most clusters the sensible base is the kube-prometheus-stack Helm chart. It installs the Prometheus Operator, a Prometheus, Alertmanager, Grafana, kube-state-metrics and node-exporter, along with default dashboards and rules. Applications are discovered through ServiceMonitor and PodMonitor objects, so teams add monitoring by shipping a small manifest with their service.
Key values we always set deliberately:
# values.yaml (excerpt)
prometheus:
prometheusSpec:
retention: 15d
replicas: 2
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
alertmanager:
alertmanagerSpec:
replicas: 3
The two NilUsesHelmValues settings make Prometheus pick up ServiceMonitors from any release instead of only this chart. Sizes and retention here are examples, and real values come from your metric volume. Use persistent volumes with a storage class that suits write-heavy workloads.
Alerts that deserve to exist
We design alerts from the user outwards, in line with SLIs and SLOs:
- Symptoms: error ratio, latency percentile, availability from the edge.
- Saturation that predicts failure: disk filling, memory approaching limits, pods pending for a long time.
- Platform health: node not ready, control plane errors, certificate expiry, failed GitOps syncs.
An example rule with a duration so brief blips do not page:
groups:
- name: workload-health
rules:
- alert: DeploymentReplicasUnavailable
expr: |
kube_deployment_status_replicas_unavailable > 0
and kube_deployment_spec_replicas > 1
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.deployment }} has unavailable replicas"
runbook_url: "https://docs.example.com/runbooks/replicas-unavailable"
Routing in Alertmanager separates severities: critical goes to the on-call tool, warning goes to a chat channel or ticket queue, and informational is not sent anywhere. Our Prometheus alerting rules examples cover many more patterns.
Dashboards in layers
One giant dashboard is unreadable. We build layers that mirror how people investigate:
- Overview: one screen for cluster health, capacity and top problems.
- Nodes and capacity: CPU, memory, disk and pod density by node pool.
- Namespace and workload: restarts, requests versus usage, HPA behaviour.
- Service: request rate, errors, latency, dependencies. These are built with the owning team.
- Platform components: ingress, DNS, GitOps controller, certificate manager.
Requests versus actual usage panels are particularly valuable, because they feed the sizing work in requests, limits and autoscaling.
Worked example: from a vague complaint to one good alert
A common request is “alert us when the site is slow”. We turn that into something testable in four moves. First, agree what slow means for users, for example a latency percentile on the main request path measured at the ingress. Second, agree how long it must persist before a human is woken, so a thirty second blip does not page anyone. Third, decide who receives it, with a named team and a runbook link in the alert annotations. Fourth, replay the rule against the last few weeks of data to see how often it would have fired.
If it would have paged ten times last month for issues nobody acted on, the threshold or duration is wrong, not the engineers. That replay step is the cheapest way we know to avoid launching a noisy alert.
Keeping the monitoring itself healthy
Prometheus can fall over quietly: a full disk, an out-of-memory kill from a cardinality spike, or a failed scrape of everything at once. We add a small set of meta alerts: Prometheus down or restarting, rule evaluation failures, scrape targets missing, and Alertmanager unable to send notifications. We also route a constantly firing “watchdog” alert to an external dead man’s switch, so you are told when the whole alert path goes silent.
Decisions and trade-offs
Self-managed or managed Prometheus. Managed offerings remove storage and scaling concerns, priced per sample. Self-managed is cheaper at small scale but you own upgrades and capacity. We help you choose by metric volume and team size.
Long-term storage. Thanos or Mimir add months of history and global queries, and add moving parts. If you rarely look back more than two weeks, skip them.
Cardinality. Labels with unbounded values (user IDs, request paths) can overwhelm Prometheus. We set guardrails and review the heaviest metrics.
Logs and traces. Metrics tell you that something is wrong; logs and traces tell you why. See our logging and OpenTelemetry service for those.
When not to build it yourself. If nobody will own the stack, a fully managed monitoring product may serve you better than a self-hosted one that rots.
Common failure modes
- Defaults left untouched, so dozens of alerts fire for components you do not run (for example scheduler or controller manager alerts on managed control planes where metrics are unavailable).
- Prometheus on ephemeral storage, losing history on every reschedule.
- No alert for Prometheus itself failing.
- Alerts without runbooks, so the on-call engineer starts from nothing.
- Dashboards that nobody maintains and quietly show wrong data after a rename.
Delivery and how to tell it works
You receive the stack as code, tuned values, layered dashboards, an alert catalogue with owners and runbook links, and a review session after a few weeks of live data to remove noise. It works when pages are rare and meaningful, incidents start from an alert rather than a customer message, and your team extends the monitoring themselves. Book a consultation if you want a review of your current setup.