Monitoring EKS with Prometheus and Grafana: Dashboards and Alerts That Help

Monitoring that answers questions

Many EKS clusters have a Grafana instance with thirty dashboards and nobody who trusts any of them. Alerts either never fire or fire all night. During an incident the first ten minutes go on finding the right graph, not fixing the fault.

This page describes the monitoring we put in place as part of AWS EKS consulting. For the wider topic see our guide to Kubernetes monitoring with Prometheus and Grafana and the dedicated Grafana and Prometheus service.

What good looks like

• One overview dashboard tells you within seconds whether the cluster is healthy.
• Drill-down dashboards follow a path: cluster, then namespace, then workload, then pod.
• Alerts mean someone should act, and each links to a runbook.
• Dashboards and rules live in Git and are provisioned, not clicked together.
• The monitoring system watches itself, including a dead-man’s switch alert.

The four layers

1. Control plane: API server request latency, error rates and etcd-related signals, as exposed through EKS metrics.
2. Nodes: CPU, memory, disk, network and pressure conditions via node-exporter.
3. Kubernetes objects: pod phases, restarts, pending pods, deployments out of sync, job failures, volume capacity via kube-state-metrics and cAdvisor.
4. Applications: request rate, errors and duration for your services, plus queue depth or other business-relevant signals.

Most guides stop at layer three. Layer four is what lets you say whether customers are affected. If your services lack metrics, we help add them using client libraries or OpenTelemetry.

Choosing the stack

For one or two clusters, kube-prometheus-stack with a persistent volume and sensible retention is often enough. For many clusters, high series counts or a team with no spare capacity for running Prometheus, the managed option is easier to justify. Pricing differs by usage, so we model it from your metric volume rather than quote a rule of thumb.

Installing and tuning

Installation is usually a Helm release managed through GitOps or Terraform. A trimmed values example:

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 80GB
    serviceMonitorSelectorNilUsesHelmValues: false
    resources:
      requests:
        cpu: "1"
        memory: 4Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 100Gi
alertmanager:
  alertmanagerSpec:
    replicas: 2
grafana:
  sidecar:
    dashboards:
      enabled: true

Sizes depend entirely on your series count, so treat these as placeholders to be replaced with measured values. The dashboards sidecar loads dashboards from ConfigMaps, which means they are version controlled alongside everything else.

Cardinality

The most common cause of an unhappy Prometheus is high-cardinality labels: user IDs, request paths with IDs in them, or per-pod labels multiplied by many metrics. We check the top series by metric name early, drop or relabel the worst offenders and set limits.

Dashboards we provision

• Cluster overview: node count, pod capacity, pending pods, API server errors.
• Nodes: utilisation, pressure, disk and network, by node pool.
• Namespace and workload: requests versus usage, restarts, throttling, replica health.
• Ingress and DNS: request rate, 4xx and 5xx rates, CoreDNS latency.
• Autoscaling: Karpenter or Cluster Autoscaler activity, linked to scaling configuration.
• Cost and efficiency: requested versus used resources, tying into cost analysis.

Alerts people can act on

We write alerts around symptoms and imminent failure: error ratio above objective, pods crash looping, nodes not ready, volumes close to full, certificate expiry, and the autoscaler unable to add capacity. Each rule has a for duration to avoid flapping, a severity label and an annotation linking a runbook. Routing in Alertmanager sends critical alerts to on-call and the rest to a channel. A worked rule set is in our post on Prometheus alerting rules.

- alert: PodCrashLooping
  expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.namespace }}/{{ $labels.pod }} is restarting repeatedly"
    runbook_url: "https://runbooks.example.internal/pod-crashloop"

A worked example: a pending pod incident

Imagine a deploy where new pods sit in Pending. A good dashboard path gets you to the cause quickly. The cluster overview shows pending pods rising. The autoscaling dashboard shows whether new nodes are being requested. The node dashboard shows whether the nodes that exist have allocatable CPU or memory left, and the namespace view shows whether the new pods requested far more than the old ones did.

Each of those panels maps to a likely cause: no capacity limit reached in the scaler, an instance type unavailable, an IP shortage, or a request that no node can satisfy. Because the dashboards are arranged along that path, the on-call engineer follows it instead of searching. We rehearse this kind of path during handover so the layout is judged by how fast it explains a real fault.

Watching the watcher

Monitoring that fails silently is worse than none. We add an always-firing heartbeat alert that routes to an external check, so a dead Alertmanager or broken route is noticed. We also alert on scrape failures, rule evaluation errors and the Prometheus disk filling, and we keep a periodic restore test for Grafana’s dashboards, which should always be rebuildable from Git.

Failure modes we plan for

• Alert fatigue from noisy defaults. We disable or tune rules that nobody will act on.
• Prometheus out of memory from cardinality. We set limits and track series growth.
• Single point of failure: monitoring on the same node pool it monitors. We separate and replicate.
• Missing logs and traces: metrics tell you what, not why. We connect to logging and tracing where needed.

Delivery and handover

You get the stack running, dashboards and rules in Git, a short runbook for each alert, and a session on reading the dashboards in an incident. If you want us to run it ongoing, that sits under managed DevOps; otherwise your team owns it from day one.

How to tell it is working

During a game-day test, break something on purpose, such as scaling a deployment to an impossible size. The right alert should fire within minutes, the dashboard should show why, and the runbook should say what to do. To plan this for your cluster, contact us.

References

• Overview, Prometheus documentation
• Alerting, Prometheus documentation
• Dashboards, Grafana documentation