Kubernetes Monitoring with Prometheus and Grafana

Running Kubernetes without monitoring is flying blind. With kube-prometheus-stack, you get Prometheus, Grafana, Alertmanager, and a complete set of pre-built dashboards and alerting rules in a single Helm chart. Here is how to set it up correctly and extend it for your workloads.

Install kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install with persistent storage and custom retention
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --values - <<EOF
prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 50Gi
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 2
        memory: 8Gi

grafana:
  persistence:
    enabled: true
    size: 10Gi
  adminPassword: change-me-in-production

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 5Gi
EOF

Access Grafana:

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80
# Login: admin / change-me-in-production

What you get out of the box

kube-prometheus-stack auto-discovers and monitors:

Node metrics (CPU, memory, disk, network) via node-exporter
Kubernetes control plane (API server, scheduler, etcd)
All Deployments, StatefulSets, DaemonSets
Pod resource requests vs actual usage
PersistentVolume usage
CoreDNS, kube-proxy

Pre-built Grafana dashboards (imported automatically):

Kubernetes / Cluster Overview
Kubernetes / Nodes
Kubernetes / Pods
Kubernetes / Persistent Volumes
Node Exporter / Full

Monitor your own services with ServiceMonitor

kube-prometheus-stack uses the Prometheus Operator, which discovers scrape targets via ServiceMonitor and PodMonitor CRDs.

# Expose metrics from your app
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app
  labels:
    release: kube-prometheus-stack  # must match Prometheus selector
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics        # named port in your Service
      path: /metrics
      interval: 15s
      scrapeTimeout: 10s

Your app’s Service must expose a named port:

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  ports:
    - name: http
      port: 8080
    - name: metrics
      port: 9090  # Prometheus endpoint

Essential PromQL queries

# CPU usage by pod (5-minute average)
rate(container_cpu_usage_seconds_total{namespace="production"}[5m])

# Memory usage vs requests
container_memory_working_set_bytes / on(pod) kube_pod_container_resource_requests{resource="memory"}

# HTTP error rate (needs your app to expose http_requests_total)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service)

# P99 request latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
)

# Pod restart count (last hour)
increase(kube_pod_container_status_restarts_total[1h]) > 0

# Node CPU utilisation
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (node)

# PVC usage percentage
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100

Recording rules to reduce query cost

Complex queries run on every scrape. Recording rules pre-compute them.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-recording-rules
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: my-app.recording
      interval: 1m
      rules:
        - record: job:http_requests_total:rate5m
          expr: sum(rate(http_requests_total[5m])) by (job, status_code)

        - record: job:http_request_duration_p99:rate5m
          expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

Alertmanager routing

# alertmanager-config.yaml (create as Secret)
global:
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK'

route:
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: slack-warnings
  routes:
    - matchers:
        - severity = critical
      receiver: pagerduty
      continue: true
    - matchers:
        - severity = warning
      receiver: slack-warnings

receivers:
  - name: pagerduty
    pagerduty_configs:
      - routing_key: YOUR_PAGERDUTY_KEY
        description: '{{ .CommonAnnotations.description }}'
        details:
          firing: '{{ .Alerts.Firing | len }} alerts'

  - name: slack-warnings
    slack_configs:
      - channel: '#alerts-warnings'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

inhibit_rules:
  - source_matchers: [severity = critical]
    target_matchers: [severity = warning]
    equal: [alertname, namespace]

Persistent storage for Prometheus

Prometheus stores data on disk. Without persistent storage, a pod restart loses all history.

For EKS, use gp3:

# Create StorageClass for gp3 (lower cost than gp2)
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

Remote write to Thanos or Mimir (long-term storage)

For retention beyond 15-30 days or multi-cluster aggregation, remote write to Thanos or Grafana Mimir.

# In kube-prometheus-stack values
prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://mimir.example.com/api/v1/push
        basicAuth:
          username:
            name: mimir-secret
            key: username
          password:
            name: mimir-secret
            key: password
        writeRelabelConfigs:
          - sourceLabels: [__name__]
            regex: 'job:.*'  # only send recording rules
            action: keep

Cardinality management

High cardinality (millions of unique label combinations) kills Prometheus performance. Common causes: user IDs, request IDs, or URLs as labels.

# Find high-cardinality metrics
topk(20, count by (__name__)({__name__=~".+"}))

# Find series count per job
count by (job) ({__name__=~".+"})

# Drop high-cardinality labels at scrape time
prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: my-app
        metric_relabel_configs:
          - source_labels: [user_id]
            action: labeldrop  # drop user_id from all metrics
          - source_labels: [__name__]
            regex: 'go_.*'
            action: drop  # drop Go runtime metrics if not needed

Key takeaways

kube-prometheus-stack is the fastest way to a production-ready monitoring stack, install it on day one.
Use ServiceMonitor to scrape your own services; avoid editing additionalScrapeConfigs directly.
Pre-compute expensive queries with recording rules to keep dashboard load times fast.
Configure Alertmanager with inhibition rules so a single outage does not generate hundreds of noise alerts.
Use gp3 storage for Prometheus on EKS, it is cheaper than gp2 and faster.
Set up remote write to Thanos/Mimir if you need retention beyond 30 days or cross-cluster metrics.

Kubernetes Monitoring with Prometheus and Grafana: A Production Guide