The problem with most dashboard collections
Open the Grafana of an average growing company and you find sixty dashboards. Some were imported from the community and never trimmed, some were built for one incident two years ago, and a few are titled “test”. During an outage, the engineer on call opens three of them, finds each showing something slightly different, and asks in chat which one to trust.
Dashboards are an interface, and bad interfaces cost minutes during incidents. Our approach in the monitoring and observability service is to build a small, layered set that a new engineer can navigate on their first shift. If you only want help with general dashboard design, see the existing Grafana dashboards page; this one covers the specific set we ship.
What good looks like
- Layered. An overview dashboard shows service health. Clicking a service opens its dashboard. From there, links lead to the component views for the database, the runtime or the node.
- Question-led. Each dashboard title and row says what it answers: “Is checkout healthy?” rather than “Checkout metrics”.
- Consistent. Same colours for the same meaning, same time range defaults, same variable names (
cluster,namespace,service). - Honest about units and limits. Panels show thresholds and capacity lines so a number has context.
- Linked to action. Each dashboard links to its alerts and its runbook.
The set we deliver
Kubernetes
Cluster overview, node pool capacity, namespace and workload resource use, pod lifecycle (restarts, OOM kills, pending pods), persistent volume usage and control plane health where your platform exposes it. These are driven by kube-state-metrics and cAdvisor, as covered in the Kubernetes monitoring page. The panels that earn their place compare requests, usage and limits, which feeds right-sizing work and connects to resource requests and limits.
Hosts
Node exporter dashboards for CPU, memory, disk, network and file descriptor saturation. We follow the USE method here: utilisation, saturation, errors for each resource.
JVM
For Java services: heap and non-heap use, GC pause time and frequency, thread states, connection pool usage and class loading. Most JVM latency incidents trace back to GC pressure, pool exhaustion or thread starvation, so those panels sit next to request latency.
PostgreSQL
Connections against the maximum, transactions per second, cache hit ratio, lock waits, replication lag, checkpoint behaviour and slow query indicators when pg_stat_statements is available. On managed Postgres, some of these come from the cloud provider rather than the exporter.
Application
For each service: request rate, error ratio, latency percentiles (RED), plus saturation signals such as queue depth and worker utilisation. These depend on instrumentation in your code.
How it works in practice
Queries that stay honest
A latency panel should use histograms and aggregate correctly. A frequent mistake is averaging percentiles across instances. The right pattern aggregates buckets first:
histogram_quantile(
0.95,
sum by (le, service) (
rate(http_server_request_duration_seconds_bucket{service="$service"}[5m])
)
)
For error ratio, divide error rate by total rate, and guard against divide by zero in quiet periods:
sum(rate(http_server_requests_total{service="$service", status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_server_requests_total{service="$service"}[5m])), 1)
Dashboards as code
We keep dashboards in your repository. For Kubernetes, the Grafana sidecar loads any ConfigMap with a dashboard label, so a service team ships its dashboard in the same chart as the Deployment:
apiVersion: v1
kind: ConfigMap
metadata:
name: checkout-dashboard
namespace: monitoring
labels:
grafana_dashboard: "1"
data:
checkout.json: |
{ "title": "Checkout service", "uid": "checkout", "schemaVersion": 39, "panels": [] }
The JSON above is trimmed for illustration. For larger sets, Grafonnet or the Grafana Terraform provider generate dashboards from templates, which keeps panels consistent across services. See also how this sits with GitOps.
Decisions and trade-offs
Community dashboards or custom. Community dashboards save days and encode lessons from many users, but they carry panels for metrics you do not collect and naming that differs from your labels. We import, then edit, and record where each came from.
One dashboard with many variables, or many dashboards. A single templated dashboard with $namespace and $service variables scales better than a copy per service. The exception is a critical service that deserves its own tailored view.
Dashboards versus alerts. Dashboards are for investigation. If a condition needs a human immediately, it should be an alert, not a panel someone has to watch. We design them together.
When you do not need this. A single small service on one server can be well served by an exporter dashboard and a couple of panels. Do not build a hierarchy you will not use.
Common failure modes
- Panel sprawl. Thirty panels on one screen, none of them actionable.
- Broken after relabelling. A label change in a ServiceMonitor silently empties every panel that used it.
- Misleading averages. Mean latency hides tail pain.
- Stale time zones and units. Bytes shown as bits, UTC against local time.
- Slow dashboards. Heavy queries over long ranges; we use recording rules for expensive expressions.
Reviewing dashboards after launch
Dashboards decay unless someone owns them. After handover we suggest a short review each quarter, led by whoever was on call most often. Ask three questions of every dashboard: did anyone open it during an incident, did it answer the question in its title, and is anything on it misleading since the last architecture change. Panels that fail all three are removed, not hidden in a folder. A smaller set that people trust beats a large archive that nobody reads.
Pay particular attention to variables and thresholds. A namespace selector that no longer lists new namespaces, or a capacity line drawn for last year’s instance size, will mislead an engineer under pressure without any visible error. Keeping dashboards in Git, as described above, makes these reviews ordinary pull requests with a clear history of why a panel changed.
Delivery and handover
You receive the dashboards in Git with variables, folder structure, links to alerts and runbooks, and a short guide on adding a dashboard for a new service. We walk your developers and on-call through the layers during a working session, using a recent incident as the example.
How to tell it is working
During the next incident, note whether the engineer reached the right panel without asking where to look. Track the number of dashboards opened per incident, panels with no data, and how long a new joiner takes to answer “is this service healthy?” To discuss a dashboard set for your stack, contact us.