What Grafana is and why teams adopt it
Grafana does not store your telemetry. It connects to the systems that do, runs queries against them and presents the results as panels, dashboards and alerts. That separation is its main strength: you can keep metrics in Prometheus, logs in Loki or Elasticsearch, traces in Tempo and business data in PostgreSQL, and still look at all of it together.
Teams adopt Grafana because it is flexible, widely understood and free to self-host under its open source licence. It also gives non-specialists something to look at. A well-built dashboard lets a product manager see error rates during a launch without reading PromQL. Our Grafana installation and data source guide shows the basic setup, and the Grafana and Prometheus monitoring service covers the wider engagement.
How we run Grafana in production
Deployment
Grafana is a stateless web application plus a database for users, folders and settings. For anything beyond a trial we use PostgreSQL or MySQL instead of the default embedded SQLite, run at least two replicas behind a load balancer, and keep the configuration in Helm values or Terraform. On Kubernetes the official Helm chart works well, with data sources and dashboards provisioned at startup.
A provisioned data source looks like this:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus-operated.monitoring:9090
isDefault: true
- name: Loki
type: loki
access: proxy
url: http://loki-gateway.logging
Dashboards as code
Hand-built dashboards decay. Someone clones one for a test, tweaks a query, and six months later nobody knows which copy is correct. We keep dashboard definitions in Git, deploy them through provisioning, and discourage saving edits to production. Engineers can still experiment in a scratch folder and export the result when it is worth keeping.
Variables do much of the work. A single service dashboard with cluster, namespace and workload variables replaces dozens of near-identical copies. Our Kubernetes, JVM and PostgreSQL dashboard patterns show the structure we start from.
Design for the on-call engineer
We lay dashboards out top to bottom: user-facing symptoms first (traffic, errors, latency), then saturation and dependencies, then detail. Panels link to related logs and traces with the time range carried across. A dashboard that needs explanation has failed its job, so each one gets a short description and an owner.
Alerting and on-call
Grafana’s unified alerting can evaluate rules against any data source and route notifications through contact points. It is convenient for rules over logs or SQL, where Prometheus cannot help. For metric alerts we often keep rules in Prometheus and Alertmanager, since that keeps them next to the data and testable with standard tooling, and use Grafana alerting for the rest. Whichever you choose, we apply the same discipline: alerts describe a user-visible problem, carry a runbook link and have an owner. See runbooks in alert annotations and Grafana OnCall schedule integration.
Security and access
Dashboards expose a lot about your systems and sometimes about customers. We configure SSO with OIDC or SAML, map identity provider groups to roles, switch off anonymous and sign-up access, and enforce TLS. Data sources are the real sensitive layer, because a viewer with query access to a database data source can read whatever that credential can read. We use read-only credentials, restrict data source permissions by team, and use service accounts with narrow scopes for automation rather than shared admin keys.
Cost and capacity
Self-hosted Grafana is cheap to run. The expensive part is the queries it generates. Dashboards that refresh every few seconds across wide time ranges can hammer Prometheus or Elasticsearch. We set sensible default refresh intervals, use recording rules for heavy expressions, and review slow panels. If you use Grafana Cloud or Amazon Managed Grafana, pricing is usage or seat based, so we check which volumes you actually send before you commit.
Day-2 operations and failure modes
Typical issues are an overloaded database, plugins that break on upgrade, stale dashboards pointing at renamed metrics and alert rules that error silently. We pin plugin versions, test upgrades in staging, back up the Grafana database, and alert on rule evaluation failures. A quarterly review removes dashboards nobody has opened, which is the simplest way to keep trust high.
Dashboard sprawl and governance
The commonest Grafana problem is not technical. After a year, an unmanaged instance holds hundreds of dashboards, many copied, many broken, and engineers stop trusting any of them. We prevent that with light governance: a folder structure per team, a short naming convention, tags for service type, and a rule that anything in production folders comes from Git.
We also keep a small set of golden dashboards per workload type, such as an HTTP service, a queue consumer or a database, so a new service gets a good view by changing a variable instead of building from scratch. Usage statistics help during reviews, since a dashboard nobody has opened in months can usually be archived without a debate.
When not to choose Grafana
If your telemetry is exclusively in Elasticsearch and your team lives in log search, Kibana as part of the ELK Stack may serve you better. If you want one vendor to collect, store and visualise everything with no infrastructure, Datadog bundles that, with a different cost model. Grafana is strongest as the front end for Prometheus and Grafana Loki. If you would like help untangling an existing dashboard estate, get in touch.
Official documentation: grafana.com