Skip to main content
Cloud DevOps Solutions
Get a quote Talk to an engineer
Observability 2026-11-12 17 min read

Prometheus Dashboard: Setup, PromQL Queries, and Grafana Best Practices

Saddique Khan

DevOps / SRE Engineer & CEO, Cloud DevOps Solutions

Prometheus collects metrics. Grafana turns those metrics into dashboards people actually look at. Together they are the most widely deployed open-source monitoring stack in the Kubernetes ecosystem, and for good reason: Prometheus handles the hard part (reliable metric collection at scale) while Grafana handles the human part (making that data visible, searchable, and actionable).

But installing both tools is the easy part. The hard part is building dashboards that help engineers find problems faster, not dashboards that look impressive in a demo and get ignored in production.

This guide walks through the full setup, from installation to production-ready dashboards, with the PromQL queries and panel configurations that matter most.

What Prometheus monitors and how

Prometheus is a pull-based monitoring system. It scrapes HTTP endpoints (called targets) at regular intervals, stores the metrics as time series data, and provides a query language (PromQL) for analysing that data.

Metric types

Prometheus collects four types of metrics:

  • Counter: a value that only goes up (e.g. total HTTP requests, total errors). You almost always use rate() or increase() with counters.
  • Gauge: a value that goes up and down (e.g. current CPU usage, memory usage, temperature). Use directly or with avg_over_time().
  • Histogram: samples observations and counts them in configurable buckets (e.g. request latency distribution). Use histogram_quantile() to compute percentiles.
  • Summary: similar to histogram but calculates quantiles on the client side. Less flexible than histograms for aggregation.

Understanding these types is essential for writing correct PromQL queries. Using rate() on a gauge or forgetting it on a counter is the most common source of wrong dashboard numbers.

Metric sources for Kubernetes

In a Kubernetes cluster, metrics come from several exporters:

SourceWhat it providesTypical job name
kubelet (cAdvisor)Container CPU, memory, disk, networkkubelet
kube-state-metricsDeployment replicas, pod status, node conditionskube-state-metrics
node-exporterHost-level CPU, memory, disk, networknode-exporter
Application instrumentationCustom business metricsApplication-specific
Prometheus itselfScrape duration, target healthprometheus

Installing Prometheus and Grafana

The most common installation method for Kubernetes is the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics.

Helm installation

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=your-secure-password \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

This deploys the full stack with sensible defaults: Prometheus scraping all Kubernetes targets, Grafana with pre-built dashboards, and Alertmanager for routing notifications.

Key Helm values to customise

prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: "45GB"
    resources:
      requests:
        cpu: "500m"
        memory: "2Gi"
      limits:
        memory: "4Gi"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 50Gi
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

grafana:
  persistence:
    enabled: true
    size: 10Gi
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: custom
          folder: Custom
          type: file
          options:
            path: /var/lib/grafana/dashboards/custom

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 5Gi

Two critical settings that trip people up: serviceMonitorSelectorNilUsesHelmValues: false and podMonitorSelectorNilUsesHelmValues: false. Without these, Prometheus only discovers ServiceMonitors with the Helm release’s labels, silently ignoring monitors created by other teams or tools.

Accessing Grafana

After installation, access Grafana via port-forward for initial setup:

kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring

For production, expose Grafana through an Ingress with TLS. The default username is admin with the password you set during installation.

Essential PromQL queries for dashboards

PromQL is the query language that powers every Prometheus dashboard. These are the queries you will use most often.

CPU usage

Node CPU utilisation (percentage):

100 - (avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
) * 100)

Pod CPU usage (cores):

sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

CPU usage vs request (shows over/under provisioning):

sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/
sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"})

A value above 1.0 means the pod is using more CPU than it requested (borrowing from the node). A value below 0.3 means the pod is significantly over-requesting.

Memory usage

Node memory utilisation:

100 * (1 - (
  node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
))

Pod memory working set (what the OOM killer watches):

sum by (namespace, pod) (
  container_memory_working_set_bytes{container!=""}
)

Memory usage vs limit (OOMKill risk):

sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"})

Values approaching 1.0 mean the pod is close to its memory limit and at risk of being OOMKilled.

Disk usage

Node disk utilisation:

100 * (1 - (
  node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"}
  / node_filesystem_free_bytes{mountpoint="/", fstype!="tmpfs"}
))

PersistentVolume usage (useful for database monitoring):

100 * (1 - (
  kubelet_volume_stats_available_bytes
  / kubelet_volume_stats_capacity_bytes
))

Network

Pod network traffic (bytes per second):

sum by (namespace, pod) (
  rate(container_network_receive_bytes_total[5m])
)

Network errors:

sum by (instance) (
  rate(node_network_receive_errs_total[5m])
  + rate(node_network_transmit_errs_total[5m])
)

HTTP request metrics (application level)

If your application exposes standard HTTP metrics:

Request rate (requests per second):

sum by (service) (
  rate(http_requests_total[5m])
)

Error rate (5xx percentage):

sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
* 100

P99 latency:

histogram_quantile(0.99,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

These three queries (rate, errors, latency) form the foundation of the RED method (Rate, Errors, Duration), the standard approach for monitoring request-driven services.

Building effective Grafana dashboards

Dashboard structure

A well-organised dashboard follows a top-down pattern:

  1. Overview row: high-level health indicators (up/down status, total request rate, error rate, p99 latency)
  2. Resource row: CPU, memory, disk, network gauges
  3. Detail rows: per-service or per-pod breakdowns
  4. Alerting row: recent alerts and their status

Panel types and when to use them

Panel typeBest forExample
StatSingle important numberTotal requests/sec, error count
GaugeValues with known rangesCPU %, memory %, disk %
Time seriesTrends over timeRequest rate, latency, throughput
TableComparing many itemsTop 10 pods by CPU, recent errors
HeatmapDistribution patternsRequest latency distribution
LogsCorrelating eventsApplication logs next to metrics

Dashboard variables (template variables)

Template variables make dashboards reusable. Instead of hardcoding a namespace, use a variable that lets the viewer filter:

Namespace variable:

  • Type: Query
  • Query: label_values(kube_pod_info, namespace)
  • Multi-value: enabled
  • Include “All” option: yes

Pod variable (dependent on namespace):

  • Type: Query
  • Query: label_values(kube_pod_info{namespace=~"$namespace"}, pod)
  • Multi-value: enabled

Then use {namespace=~"$namespace"} in every panel query. This single pattern makes one dashboard serve every team.

Colour thresholds

Set meaningful thresholds, not arbitrary ones:

  • CPU utilisation: green below 60%, yellow 60-80%, red above 80%
  • Memory utilisation: green below 70%, yellow 70-85%, red above 85%
  • Error rate: green at 0%, yellow above 0.1%, red above 1%
  • Disk usage: green below 70%, yellow 70-85%, red above 85%
  • P99 latency: depends on SLO. If your SLO is 500ms, green below 400ms, yellow 400-500ms, red above 500ms

Four dashboards every team needs

1. Cluster overview dashboard

Shows the health of the entire Kubernetes cluster at a glance:

Row 1: Cluster health

  • Total nodes (stat panel)
  • Ready vs not-ready nodes (stat with threshold)
  • Total pods running / total capacity (stat)
  • Cluster CPU utilisation (gauge)
  • Cluster memory utilisation (gauge)

Row 2: Node resources

  • CPU by node (time series, stacked)
  • Memory by node (time series, stacked)
  • Disk by node (table with colour thresholds)
  • Network I/O by node (time series)

Row 3: Pod status

  • Pods by phase (stat panels: Running, Pending, Failed, Unknown)
  • Pod restart count by namespace (table, sorted descending)
  • OOMKilled pods in last 24h (table)

2. Namespace / team dashboard

Scoped to a single namespace using template variables:

Row 1: Namespace summary

  • Total pods (stat)
  • CPU usage vs requests (gauge, shows efficiency)
  • Memory usage vs requests (gauge)
  • Active alerts (stat with link to Alertmanager)

Row 2: Deployments

  • Deployment replicas: desired vs available (table)
  • Pod restart count by deployment (time series)
  • HPA current vs min vs max replicas (time series)

Row 3: Resource usage

  • CPU usage by pod (time series)
  • Memory usage by pod (time series)
  • Network receive/transmit by pod (time series)

3. Service / application dashboard

Focuses on application-level metrics using the RED method:

Row 1: Golden signals

  • Request rate (time series)
  • Error rate percentage (stat with threshold)
  • P50 / P95 / P99 latency (time series with multiple queries)

Row 2: Endpoints

  • Request rate by endpoint (table)
  • Error rate by endpoint (table with colour)
  • Latency by endpoint (heatmap)

Row 3: Dependencies

  • Outbound request rate to external services (time series)
  • Outbound error rate (time series)
  • Database connection pool usage (gauge)
  • Cache hit rate (stat)

4. Alerting dashboard

Aggregates alert status across the cluster:

  • Active alerts by severity (stat panels: critical, warning, info)
  • Alert history timeline (annotations on a time series panel)
  • Top firing alerts in last 7 days (table)
  • Silenced alerts (table)

Configuring Prometheus alerting rules

Dashboards show the current state. Alerts notify you when something needs attention. Both use the same PromQL queries.

Alert rule structure

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: application-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: application.rules
      rules:
        - alert: HighErrorRate
          expr: |
            sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m]))
            > 0.01
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate on {{ $labels.service }}"
            description: "{{ $labels.service }} has a 5xx error rate above 1% (current: {{ $value | humanizePercentage }})"

        - alert: HighLatency
          expr: |
            histogram_quantile(0.99,
              sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
            ) > 0.5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High P99 latency on {{ $labels.service }}"
            description: "{{ $labels.service }} P99 latency is above 500ms (current: {{ $value | humanizeDuration }})"

        - alert: PodOOMKilled
          expr: |
            kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} OOMKilled"
            description: "Container {{ $labels.container }} in pod {{ $labels.pod }} was OOMKilled"

        - alert: PersistentVolumeFillingUp
          expr: |
            kubelet_volume_stats_available_bytes
            / kubelet_volume_stats_capacity_bytes
            < 0.15
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "PV {{ $labels.persistentvolumeclaim }} is filling up"
            description: "PV {{ $labels.persistentvolumeclaim }} in {{ $labels.namespace }} has less than 15% space remaining"

Alertmanager routing

Configure Alertmanager to route alerts to the right channels:

route:
  receiver: default-slack
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: slack-warnings
      repeat_interval: 4h

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: '<your-pagerduty-key>'
  - name: slack-warnings
    slack_configs:
      - channel: '#alerts-warning'
        send_resolved: true
  - name: default-slack
    slack_configs:
      - channel: '#alerts-info'
        send_resolved: true

Alert best practices

  • Use for duration to avoid alerting on transient spikes. A 5-minute for means the condition must be true for 5 continuous minutes before firing.
  • Keep alert count manageable. More than 50 active alerts and the team ignores all of them. If everything is critical, nothing is.
  • Include context in annotations. The engineer receiving the alert at 3 AM needs to know what is wrong, what service is affected, and what the current value is, without opening a dashboard.
  • Set severity correctly. Critical means “wake someone up.” Warning means “look at this during business hours.” Info means “for your awareness.”

Production best practices

Retention and storage

Prometheus stores data locally. For production:

  • Retention by time: 15-30 days for high-resolution data
  • Retention by size: cap storage to prevent disk exhaustion (retentionSize)
  • Long-term storage: for data beyond 30 days, use Thanos or Cortex. These add a remote storage layer backed by object storage (S3, GCS, Azure Blob) and enable cross-cluster queries.

High availability

Run two Prometheus replicas scraping the same targets. Grafana queries both and deduplicates automatically when configured with prometheusType: Thanos or using a Thanos Querier frontend.

Recording rules for performance

Queries that aggregate across thousands of time series are expensive. Pre-compute them with recording rules:

groups:
  - name: recording.rules
    rules:
      - record: namespace:container_cpu_usage:sum_rate5m
        expr: |
          sum by (namespace) (
            rate(container_cpu_usage_seconds_total{container!=""}[5m])
          )
      - record: namespace:container_memory_working_set:sum
        expr: |
          sum by (namespace) (
            container_memory_working_set_bytes{container!=""}
          )

Use recording rules in your dashboards instead of raw queries. This makes dashboards load faster and reduces the load on Prometheus during peak usage (when everyone is looking at dashboards because something is broken).

ServiceMonitor for custom applications

To monitor your own applications, create a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-service
  namespace: production
spec:
  selector:
    matchLabels:
      app: checkout-service
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics
  namespaceSelector:
    matchNames:
      - production

This tells Prometheus to scrape the /metrics endpoint of every pod behind the checkout-service service every 15 seconds.

Dashboard management as code

Store dashboards in version control, not in the Grafana UI. Options:

  1. Grafana provisioning: export dashboards as JSON and store them in a ConfigMap or persistent volume that Grafana loads on startup (configured via the dashboardProviders Helm value shown earlier).

  2. Grafonnet (Jsonnet library): generate dashboard JSON programmatically. More maintainable for large numbers of similar dashboards.

  3. Grafana Terraform provider: manage dashboards, data sources, and folders as Terraform resources. Integrates with existing IaC workflows.

Dashboard-as-code prevents the “someone changed the production dashboard and nobody knows what it looked like before” problem.

Scrape interval tuning

The default scrape interval is 15 seconds for most targets. Adjust based on the metric:

  • Infrastructure metrics (node-exporter, kubelet): 15-30 seconds
  • Application metrics: 15 seconds for latency-sensitive services, 30-60 seconds for batch jobs
  • kube-state-metrics: 30 seconds (infrequent changes)

Shorter intervals mean higher storage costs and more load on both Prometheus and the targets. A 10-second interval doubles storage compared to 20 seconds.

Common mistakes to avoid

1. Dashboard sprawl

Teams create dashboards for every occasion and abandon them. Within months you have 200 dashboards, 190 of which are broken or irrelevant. Establish ownership: every dashboard has an owner, and dashboards without active owners are archived quarterly.

2. Missing rate() on counters

A panel showing http_requests_total as a time series produces a monotonically increasing line that tells you nothing useful. You almost always want rate(http_requests_total[5m]) to see the per-second request rate.

3. Ignoring label cardinality

A label with high cardinality (e.g. user_id with millions of unique values) creates millions of time series. This slows Prometheus, increases storage, and makes queries time out. Keep labels to dimensions you actually filter or aggregate by: service, endpoint, status_code, namespace, pod.

4. No dashboard variables

Hardcoded namespace and pod names in queries make dashboards useless for other teams. Always use template variables so one dashboard serves everyone.

5. Alerting on symptoms, not causes

An alert on “CPU above 80%” fires constantly and teaches the team to ignore it. Alert on the customer impact instead: error rate above threshold, latency above SLO, pods not ready. Engineers can then investigate the cause (which might be CPU, memory, or something else entirely).

Integrating with the broader stack

Prometheus and Grafana are often the centre of a larger observability stack:

  • Loki for log aggregation, queryable from the same Grafana dashboards alongside metrics. See our logging with ELK, Loki, and OpenTelemetry guide for setup details.
  • Tempo or Jaeger for distributed tracing, linked from Grafana panels via trace IDs.
  • OpenTelemetry as the unified instrumentation layer, exporting metrics to Prometheus and traces to Tempo/Jaeger.
  • PagerDuty, Slack, or Microsoft Teams as alert notification channels via Alertmanager.

For teams using Kubernetes cost monitoring alongside Prometheus, Kubecost integrates with Prometheus to provide cost allocation per namespace and deployment. And for EKS-specific monitoring, our EKS cost optimisation guide covers how to combine Prometheus metrics with AWS billing data.

Our monitoring and observability service helps teams design, deploy, and maintain Prometheus and Grafana at scale. Whether you are setting up monitoring for the first time or migrating from a commercial tool, we bring the patterns that work in production.

Key takeaways

  • Prometheus collects metrics via pull-based scraping; Grafana visualises them. The kube-prometheus-stack Helm chart deploys both with sensible defaults.
  • Understand the four metric types (counter, gauge, histogram, summary) to write correct PromQL. Using rate() on counters is the most common requirement.
  • Build four core dashboards: cluster overview, namespace/team, service/application (RED method), and alerting status.
  • Template variables make dashboards reusable across teams and namespaces. Never hardcode label values.
  • Alerting rules should fire on customer impact (error rate, latency) not raw resource metrics (CPU percentage).
  • Recording rules pre-compute expensive aggregations, making dashboards faster under load.
  • Store dashboards as code (provisioning, Grafonnet, or Terraform) to prevent configuration drift.
  • High cardinality labels are the most common cause of Prometheus performance problems. Audit labels before they reach production.

Need help setting up Prometheus and Grafana for your infrastructure? Book a free consultation and we will design a monitoring stack tailored to your environment.

#prometheus #grafana #kubernetes #monitoring #observability #alerting

Related service

Need hands-on help? Our certified engineers deliver this as a managed service.

Monitoring with Grafana & Prometheus
Found this useful? Share it with your team or link to it from your documentation.

Need help implementing this?

Our certified engineers can handle the setup, migration, or ongoing operations for you.