Why a careful Grafana install matters
Grafana is where your engineers actually look during an incident, so its reliability and access model matter more than most teams assume. We often meet instances installed in an afternoon: SQLite on a pod without persistence, a shared admin login pinned in a channel topic, data sources clicked in by hand and a dozen dashboards that nobody can reproduce. It works until the pod restarts and everything vanishes, or until a leaver still has the password.
Installing Grafana properly is the second building block of the monitoring and observability service, after Prometheus itself. For a plain-language definition, see our Grafana glossary entry.
What good looks like
- Provisioned, not clicked. Data sources, dashboards, folders and alert contact points are defined as files.
- Real authentication. SSO through your identity provider, with group to role mapping. Local admin used only as a break-glass account.
- A real database. PostgreSQL or MySQL, backed up, so replicas can run behind a load balancer.
- TLS and sensible exposure. Behind your ingress or an internal load balancer, not open to the internet unless you have decided it should be.
- Connected signals. Metrics, logs and traces link to each other.
- Upgradable. Version pinned, plugins pinned, and upgrades rehearsed.
Choosing where Grafana runs
| Option | Fits when | Watch for |
|---|---|---|
| Helm on your Kubernetes cluster | You already run the cluster and want data in your network | You own upgrades, backups and availability |
| Package or container on a VM | No cluster, or a small estate | Manual drift unless managed with Ansible or Terraform |
| Grafana Cloud | Small team, little appetite to operate monitoring | Ongoing usage cost, data leaving your network |
None of these is universally right. A five person startup with no platform engineer is often better served by a hosted Grafana, and we will say so. A regulated company that must keep telemetry in its own accounts will self-host.
How it works in practice
Installing with Helm
For a standalone install we use the official chart with an external database and persistence, and keep sensitive values out of the file by referencing a Kubernetes secret:
replicas: 2
persistence:
enabled: false # state lives in PostgreSQL
envFromSecret: grafana-secrets
grafana.ini:
server:
root_url: https://grafana.example.com
database:
type: postgres
host: grafana-db.internal:5432
name: grafana
user: grafana
ssl_mode: require
auth.generic_oauth:
enabled: true
name: Company SSO
allow_sign_up: true
scopes: openid profile email groups
role_attribute_path: "contains(groups[*], 'platform') && 'Admin' || 'Viewer'"
The database password and OAuth client secret come in through environment variables from the referenced secret, ideally synced from your cloud secret manager. Nothing sensitive is committed. Our GitOps with Helm, Argo CD and Flux work covers how that sync is usually wired.
Provisioning data sources
A data source file under provisioning/datasources/ looks like this:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
access: proxy
url: http://prometheus-operated.monitoring:9090
isDefault: true
jsonData:
timeInterval: 30s
exemplarTraceIdDestinations:
- name: trace_id
datasourceUid: tempo
- name: Loki
type: loki
uid: loki
access: proxy
url: http://loki-gateway.monitoring
jsonData:
derivedFields:
- name: TraceID
matcherRegex: 'trace_id=(\w+)'
url: '$${__value.raw}'
datasourceUid: tempo
- name: Tempo
type: tempo
uid: tempo
access: proxy
url: http://tempo.monitoring:3100
jsonData:
tracesToLogsV2:
datasourceUid: loki
Fixed uid values are the detail that keeps dashboards portable. Dashboards reference a data source by uid, so the same dashboard JSON works in staging and production without edits.
Linking metrics, logs and traces
This is the part that changes how an incident feels. With exemplars on a latency panel, a spike links to a specific trace. From the trace, a click opens the log lines for that request. Instead of guessing which service is slow and then grepping three tools, the engineer follows a trail. It depends on applications emitting trace IDs in logs and exemplars, which we cover in distributed tracing with OpenTelemetry and Tempo.
Access, teams and folders
We create folders by team or domain, assign permissions per folder, and keep the number of Admins small. Viewers and Editors map to groups in your identity provider so leavers lose access automatically. Service accounts with tokens (not personal API keys) are used for automation, with an expiry.
Decisions and trade-offs
Dashboards in the UI or in Git. Allowing UI edits is friendly, but edits are lost if the provisioned dashboard is overwritten. We provision shared dashboards as read-only and give people a personal folder or a “save a copy” workflow for experiments. The detail of that approach is in our dashboard work.
Grafana alerting or Prometheus alerting. Both exist. We usually keep metric alert rules in Prometheus and Alertmanager, where they live beside the code, and use Grafana alerting where you need to alert across mixed data sources. Running both without a plan creates duplicate pages.
Plugins. Each plugin is code running in your monitoring system. We install few, pin versions and avoid unsigned plugins.
Common failure modes
- Data lost on restart from SQLite on ephemeral storage.
- Wide-open anonymous access left on after a demo.
- Wrong data source URL scope, for example pointing at a single Prometheus replica instead of a stable service.
- Query timeouts from dashboards that scan too much; we set sensible timeouts and query limits.
- Upgrade surprises when plugins break on a major version. We test upgrades against a copy.
Delivery and handover
You get the Helm values, provisioning files, SSO configuration, a backup job for the Grafana database, and a one page upgrade procedure. Everything sits in your repository. We run through a login and permission test with your team so ownership is clear.
How to tell it is working
Check that a rebuilt Grafana comes up with identical data sources and dashboards, that a new joiner gets the right access without an admin, that cross-links from metrics to logs and traces work on real incidents, and that database backups restore. To plan a Grafana rollout, get in touch.