Prometheus Dashboard: Setup, PromQL Queries, and Grafana Best Practices
DevOps / SRE Engineer & CEO, Cloud DevOps Solutions
Prometheus collects metrics. Grafana turns those metrics into dashboards people actually look at. Together they are the most widely deployed open-source monitoring stack in the Kubernetes ecosystem, and for good reason: Prometheus handles the hard part (reliable metric collection at scale) while Grafana handles the human part (making that data visible, searchable, and actionable).
But installing both tools is the easy part. The hard part is building dashboards that help engineers find problems faster, not dashboards that look impressive in a demo and get ignored in production.
This guide walks through the full setup, from installation to production-ready dashboards, with the PromQL queries and panel configurations that matter most.
What Prometheus monitors and how
Prometheus is a pull-based monitoring system. It scrapes HTTP endpoints (called targets) at regular intervals, stores the metrics as time series data, and provides a query language (PromQL) for analysing that data.
Metric types
Prometheus collects four types of metrics:
- Counter: a value that only goes up (e.g. total HTTP requests, total errors). You almost always use
rate()orincrease()with counters. - Gauge: a value that goes up and down (e.g. current CPU usage, memory usage, temperature). Use directly or with
avg_over_time(). - Histogram: samples observations and counts them in configurable buckets (e.g. request latency distribution). Use
histogram_quantile()to compute percentiles. - Summary: similar to histogram but calculates quantiles on the client side. Less flexible than histograms for aggregation.
Understanding these types is essential for writing correct PromQL queries. Using rate() on a gauge or forgetting it on a counter is the most common source of wrong dashboard numbers.
Metric sources for Kubernetes
In a Kubernetes cluster, metrics come from several exporters:
| Source | What it provides | Typical job name |
|---|---|---|
| kubelet (cAdvisor) | Container CPU, memory, disk, network | kubelet |
| kube-state-metrics | Deployment replicas, pod status, node conditions | kube-state-metrics |
| node-exporter | Host-level CPU, memory, disk, network | node-exporter |
| Application instrumentation | Custom business metrics | Application-specific |
| Prometheus itself | Scrape duration, target health | prometheus |
Installing Prometheus and Grafana
The most common installation method for Kubernetes is the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics.
Helm installation
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set grafana.adminPassword=your-secure-password \
--set prometheus.prometheusSpec.retention=30d \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi
This deploys the full stack with sensible defaults: Prometheus scraping all Kubernetes targets, Grafana with pre-built dashboards, and Alertmanager for routing notifications.
Key Helm values to customise
prometheus:
prometheusSpec:
retention: 30d
retentionSize: "45GB"
resources:
requests:
cpu: "500m"
memory: "2Gi"
limits:
memory: "4Gi"
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gp3
resources:
requests:
storage: 50Gi
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
grafana:
persistence:
enabled: true
size: 10Gi
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: custom
folder: Custom
type: file
options:
path: /var/lib/grafana/dashboards/custom
alertmanager:
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 5Gi
Two critical settings that trip people up: serviceMonitorSelectorNilUsesHelmValues: false and podMonitorSelectorNilUsesHelmValues: false. Without these, Prometheus only discovers ServiceMonitors with the Helm release’s labels, silently ignoring monitors created by other teams or tools.
Accessing Grafana
After installation, access Grafana via port-forward for initial setup:
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring
For production, expose Grafana through an Ingress with TLS. The default username is admin with the password you set during installation.
Essential PromQL queries for dashboards
PromQL is the query language that powers every Prometheus dashboard. These are the queries you will use most often.
CPU usage
Node CPU utilisation (percentage):
100 - (avg by (instance) (
rate(node_cpu_seconds_total{mode="idle"}[5m])
) * 100)
Pod CPU usage (cores):
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{container!=""}[5m])
)
CPU usage vs request (shows over/under provisioning):
sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/
sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"})
A value above 1.0 means the pod is using more CPU than it requested (borrowing from the node). A value below 0.3 means the pod is significantly over-requesting.
Memory usage
Node memory utilisation:
100 * (1 - (
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
))
Pod memory working set (what the OOM killer watches):
sum by (namespace, pod) (
container_memory_working_set_bytes{container!=""}
)
Memory usage vs limit (OOMKill risk):
sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"})
Values approaching 1.0 mean the pod is close to its memory limit and at risk of being OOMKilled.
Disk usage
Node disk utilisation:
100 * (1 - (
node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"}
/ node_filesystem_free_bytes{mountpoint="/", fstype!="tmpfs"}
))
PersistentVolume usage (useful for database monitoring):
100 * (1 - (
kubelet_volume_stats_available_bytes
/ kubelet_volume_stats_capacity_bytes
))
Network
Pod network traffic (bytes per second):
sum by (namespace, pod) (
rate(container_network_receive_bytes_total[5m])
)
Network errors:
sum by (instance) (
rate(node_network_receive_errs_total[5m])
+ rate(node_network_transmit_errs_total[5m])
)
HTTP request metrics (application level)
If your application exposes standard HTTP metrics:
Request rate (requests per second):
sum by (service) (
rate(http_requests_total[5m])
)
Error rate (5xx percentage):
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
* 100
P99 latency:
histogram_quantile(0.99,
sum by (le, service) (
rate(http_request_duration_seconds_bucket[5m])
)
)
These three queries (rate, errors, latency) form the foundation of the RED method (Rate, Errors, Duration), the standard approach for monitoring request-driven services.
Building effective Grafana dashboards
Dashboard structure
A well-organised dashboard follows a top-down pattern:
- Overview row: high-level health indicators (up/down status, total request rate, error rate, p99 latency)
- Resource row: CPU, memory, disk, network gauges
- Detail rows: per-service or per-pod breakdowns
- Alerting row: recent alerts and their status
Panel types and when to use them
| Panel type | Best for | Example |
|---|---|---|
| Stat | Single important number | Total requests/sec, error count |
| Gauge | Values with known ranges | CPU %, memory %, disk % |
| Time series | Trends over time | Request rate, latency, throughput |
| Table | Comparing many items | Top 10 pods by CPU, recent errors |
| Heatmap | Distribution patterns | Request latency distribution |
| Logs | Correlating events | Application logs next to metrics |
Dashboard variables (template variables)
Template variables make dashboards reusable. Instead of hardcoding a namespace, use a variable that lets the viewer filter:
Namespace variable:
- Type: Query
- Query:
label_values(kube_pod_info, namespace) - Multi-value: enabled
- Include “All” option: yes
Pod variable (dependent on namespace):
- Type: Query
- Query:
label_values(kube_pod_info{namespace=~"$namespace"}, pod) - Multi-value: enabled
Then use {namespace=~"$namespace"} in every panel query. This single pattern makes one dashboard serve every team.
Colour thresholds
Set meaningful thresholds, not arbitrary ones:
- CPU utilisation: green below 60%, yellow 60-80%, red above 80%
- Memory utilisation: green below 70%, yellow 70-85%, red above 85%
- Error rate: green at 0%, yellow above 0.1%, red above 1%
- Disk usage: green below 70%, yellow 70-85%, red above 85%
- P99 latency: depends on SLO. If your SLO is 500ms, green below 400ms, yellow 400-500ms, red above 500ms
Four dashboards every team needs
1. Cluster overview dashboard
Shows the health of the entire Kubernetes cluster at a glance:
Row 1: Cluster health
- Total nodes (stat panel)
- Ready vs not-ready nodes (stat with threshold)
- Total pods running / total capacity (stat)
- Cluster CPU utilisation (gauge)
- Cluster memory utilisation (gauge)
Row 2: Node resources
- CPU by node (time series, stacked)
- Memory by node (time series, stacked)
- Disk by node (table with colour thresholds)
- Network I/O by node (time series)
Row 3: Pod status
- Pods by phase (stat panels: Running, Pending, Failed, Unknown)
- Pod restart count by namespace (table, sorted descending)
- OOMKilled pods in last 24h (table)
2. Namespace / team dashboard
Scoped to a single namespace using template variables:
Row 1: Namespace summary
- Total pods (stat)
- CPU usage vs requests (gauge, shows efficiency)
- Memory usage vs requests (gauge)
- Active alerts (stat with link to Alertmanager)
Row 2: Deployments
- Deployment replicas: desired vs available (table)
- Pod restart count by deployment (time series)
- HPA current vs min vs max replicas (time series)
Row 3: Resource usage
- CPU usage by pod (time series)
- Memory usage by pod (time series)
- Network receive/transmit by pod (time series)
3. Service / application dashboard
Focuses on application-level metrics using the RED method:
Row 1: Golden signals
- Request rate (time series)
- Error rate percentage (stat with threshold)
- P50 / P95 / P99 latency (time series with multiple queries)
Row 2: Endpoints
- Request rate by endpoint (table)
- Error rate by endpoint (table with colour)
- Latency by endpoint (heatmap)
Row 3: Dependencies
- Outbound request rate to external services (time series)
- Outbound error rate (time series)
- Database connection pool usage (gauge)
- Cache hit rate (stat)
4. Alerting dashboard
Aggregates alert status across the cluster:
- Active alerts by severity (stat panels: critical, warning, info)
- Alert history timeline (annotations on a time series panel)
- Top firing alerts in last 7 days (table)
- Silenced alerts (table)
Configuring Prometheus alerting rules
Dashboards show the current state. Alerts notify you when something needs attention. Both use the same PromQL queries.
Alert rule structure
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: application-alerts
namespace: monitoring
labels:
release: monitoring
spec:
groups:
- name: application.rules
rules:
- alert: HighErrorRate
expr: |
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))
> 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate on {{ $labels.service }}"
description: "{{ $labels.service }} has a 5xx error rate above 1% (current: {{ $value | humanizePercentage }})"
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "High P99 latency on {{ $labels.service }}"
description: "{{ $labels.service }} P99 latency is above 500ms (current: {{ $value | humanizeDuration }})"
- alert: PodOOMKilled
expr: |
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
for: 0m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} OOMKilled"
description: "Container {{ $labels.container }} in pod {{ $labels.pod }} was OOMKilled"
- alert: PersistentVolumeFillingUp
expr: |
kubelet_volume_stats_available_bytes
/ kubelet_volume_stats_capacity_bytes
< 0.15
for: 10m
labels:
severity: warning
annotations:
summary: "PV {{ $labels.persistentvolumeclaim }} is filling up"
description: "PV {{ $labels.persistentvolumeclaim }} in {{ $labels.namespace }} has less than 15% space remaining"
Alertmanager routing
Configure Alertmanager to route alerts to the right channels:
route:
receiver: default-slack
group_by: ['alertname', 'namespace']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: pagerduty-critical
repeat_interval: 1h
- match:
severity: warning
receiver: slack-warnings
repeat_interval: 4h
receivers:
- name: pagerduty-critical
pagerduty_configs:
- service_key: '<your-pagerduty-key>'
- name: slack-warnings
slack_configs:
- channel: '#alerts-warning'
send_resolved: true
- name: default-slack
slack_configs:
- channel: '#alerts-info'
send_resolved: true
Alert best practices
- Use
forduration to avoid alerting on transient spikes. A 5-minuteformeans the condition must be true for 5 continuous minutes before firing. - Keep alert count manageable. More than 50 active alerts and the team ignores all of them. If everything is critical, nothing is.
- Include context in annotations. The engineer receiving the alert at 3 AM needs to know what is wrong, what service is affected, and what the current value is, without opening a dashboard.
- Set severity correctly. Critical means “wake someone up.” Warning means “look at this during business hours.” Info means “for your awareness.”
Production best practices
Retention and storage
Prometheus stores data locally. For production:
- Retention by time: 15-30 days for high-resolution data
- Retention by size: cap storage to prevent disk exhaustion (
retentionSize) - Long-term storage: for data beyond 30 days, use Thanos or Cortex. These add a remote storage layer backed by object storage (S3, GCS, Azure Blob) and enable cross-cluster queries.
High availability
Run two Prometheus replicas scraping the same targets. Grafana queries both and deduplicates automatically when configured with prometheusType: Thanos or using a Thanos Querier frontend.
Recording rules for performance
Queries that aggregate across thousands of time series are expensive. Pre-compute them with recording rules:
groups:
- name: recording.rules
rules:
- record: namespace:container_cpu_usage:sum_rate5m
expr: |
sum by (namespace) (
rate(container_cpu_usage_seconds_total{container!=""}[5m])
)
- record: namespace:container_memory_working_set:sum
expr: |
sum by (namespace) (
container_memory_working_set_bytes{container!=""}
)
Use recording rules in your dashboards instead of raw queries. This makes dashboards load faster and reduces the load on Prometheus during peak usage (when everyone is looking at dashboards because something is broken).
ServiceMonitor for custom applications
To monitor your own applications, create a ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: checkout-service
namespace: production
spec:
selector:
matchLabels:
app: checkout-service
endpoints:
- port: metrics
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- production
This tells Prometheus to scrape the /metrics endpoint of every pod behind the checkout-service service every 15 seconds.
Dashboard management as code
Store dashboards in version control, not in the Grafana UI. Options:
-
Grafana provisioning: export dashboards as JSON and store them in a ConfigMap or persistent volume that Grafana loads on startup (configured via the
dashboardProvidersHelm value shown earlier). -
Grafonnet (Jsonnet library): generate dashboard JSON programmatically. More maintainable for large numbers of similar dashboards.
-
Grafana Terraform provider: manage dashboards, data sources, and folders as Terraform resources. Integrates with existing IaC workflows.
Dashboard-as-code prevents the “someone changed the production dashboard and nobody knows what it looked like before” problem.
Scrape interval tuning
The default scrape interval is 15 seconds for most targets. Adjust based on the metric:
- Infrastructure metrics (node-exporter, kubelet): 15-30 seconds
- Application metrics: 15 seconds for latency-sensitive services, 30-60 seconds for batch jobs
- kube-state-metrics: 30 seconds (infrequent changes)
Shorter intervals mean higher storage costs and more load on both Prometheus and the targets. A 10-second interval doubles storage compared to 20 seconds.
Common mistakes to avoid
1. Dashboard sprawl
Teams create dashboards for every occasion and abandon them. Within months you have 200 dashboards, 190 of which are broken or irrelevant. Establish ownership: every dashboard has an owner, and dashboards without active owners are archived quarterly.
2. Missing rate() on counters
A panel showing http_requests_total as a time series produces a monotonically increasing line that tells you nothing useful. You almost always want rate(http_requests_total[5m]) to see the per-second request rate.
3. Ignoring label cardinality
A label with high cardinality (e.g. user_id with millions of unique values) creates millions of time series. This slows Prometheus, increases storage, and makes queries time out. Keep labels to dimensions you actually filter or aggregate by: service, endpoint, status_code, namespace, pod.
4. No dashboard variables
Hardcoded namespace and pod names in queries make dashboards useless for other teams. Always use template variables so one dashboard serves everyone.
5. Alerting on symptoms, not causes
An alert on “CPU above 80%” fires constantly and teaches the team to ignore it. Alert on the customer impact instead: error rate above threshold, latency above SLO, pods not ready. Engineers can then investigate the cause (which might be CPU, memory, or something else entirely).
Integrating with the broader stack
Prometheus and Grafana are often the centre of a larger observability stack:
- Loki for log aggregation, queryable from the same Grafana dashboards alongside metrics. See our logging with ELK, Loki, and OpenTelemetry guide for setup details.
- Tempo or Jaeger for distributed tracing, linked from Grafana panels via trace IDs.
- OpenTelemetry as the unified instrumentation layer, exporting metrics to Prometheus and traces to Tempo/Jaeger.
- PagerDuty, Slack, or Microsoft Teams as alert notification channels via Alertmanager.
For teams using Kubernetes cost monitoring alongside Prometheus, Kubecost integrates with Prometheus to provide cost allocation per namespace and deployment. And for EKS-specific monitoring, our EKS cost optimisation guide covers how to combine Prometheus metrics with AWS billing data.
Our monitoring and observability service helps teams design, deploy, and maintain Prometheus and Grafana at scale. Whether you are setting up monitoring for the first time or migrating from a commercial tool, we bring the patterns that work in production.
Key takeaways
- Prometheus collects metrics via pull-based scraping; Grafana visualises them. The kube-prometheus-stack Helm chart deploys both with sensible defaults.
- Understand the four metric types (counter, gauge, histogram, summary) to write correct PromQL. Using
rate()on counters is the most common requirement. - Build four core dashboards: cluster overview, namespace/team, service/application (RED method), and alerting status.
- Template variables make dashboards reusable across teams and namespaces. Never hardcode label values.
- Alerting rules should fire on customer impact (error rate, latency) not raw resource metrics (CPU percentage).
- Recording rules pre-compute expensive aggregations, making dashboards faster under load.
- Store dashboards as code (provisioning, Grafonnet, or Terraform) to prevent configuration drift.
- High cardinality labels are the most common cause of Prometheus performance problems. Audit labels before they reach production.
Need help setting up Prometheus and Grafana for your infrastructure? Book a free consultation and we will design a monitoring stack tailored to your environment.
Related service
Need hands-on help? Our certified engineers deliver this as a managed service.
Monitoring with Grafana & Prometheus