Why most log dashboards go unused
A common pattern: someone builds twenty panels after a rough week, the team looks at them twice, and from then on everyone goes back to typing queries into the search bar. The dashboards fail for predictable reasons. They show what was easy to chart rather than what people ask. They depend on fields that differ across services. They cannot be trusted because nobody knows who maintains them.
Useful dashboards are small and tied to decisions. They are part of the wider logging, ELK, Loki and OpenTelemetry service, and they pair with metrics dashboards covered in our Grafana and Prometheus work.
What good looks like
- A top-level view showing error rate by service and environment for the last hours and days.
- A per-service page with volume by level, top error messages, slowest or failing endpoints and recent deploy markers.
- Every chart clickable into the matching log lines.
- Consistent variables: the same
service,envandversionselectors everywhere. - Dashboards in Git with an owner, not hand-edited in the UI.
The four views we build first
1. Fleet overview. Error rate and log volume per service, sorted by change. It answers “is anything unusual right now?”
2. Service triage. One service, with levels over time, top ten error messages grouped by pattern, and a table of the latest errors with trace links. This is the page an on-call engineer opens first.
3. Error explorer. Free filtering by status, endpoint, customer tier or version. Built on fields, not on substring search.
4. Release comparison. Error rate and top new messages in the hour after a deploy against the hour before. It turns “did the release break anything?” into a quick check, and connects with CI/CD deploy annotations.
How it works in practice
Fields before charts
Everything above assumes fields exist. If level, service, env, version, http.status and a request or trace identifier are present on every line, dashboards are a few queries each. If they are missing, you spend weeks writing parsers. Our structured logging guidelines are the foundation.
Error rate in Grafana with Loki
LogQL treats logs as a stream you can count over time. This query gives the share of error lines for a service, assuming JSON logs with a level field:
sum(rate({namespace="checkout", app="api"} | json | level="error" [5m]))
/
sum(rate({namespace="checkout", app="api"} [5m]))
For a table of the most frequent errors, group on a normalised message field:
topk(10, sum by (msg) (count_over_time({namespace="checkout"} | json | level="error" [1h])))
Normalise messages at the source, keeping IDs and values out of the message text, or every error looks unique and the table is useless.
Error rate in Kibana
In Kibana, Lens or a saved Discover view gives the same result. Plot a date histogram of documents filtered with KQL such as log.level : "error" and service.name : "checkout", split by service. Keep a data view per index pattern, and use runtime fields sparingly because they add query cost. Kibana’s strength is the combination of search, aggregations and saved filters over rich fields.
Linking to evidence
The goal is a short path from signal to line. In Grafana, data links from a panel open Explore with the same labels and time range, and a derived field can turn a trace ID in a log line into a link to the trace. That cross-navigation is covered on our trace and log correlation page. In Kibana, panels drill down into Discover with filters preserved.
Dashboards as code
Grafana dashboards export as JSON and can be provisioned from files or managed with Terraform. Kibana saved objects can be exported and imported through the API. We keep them in the same repository workflow as other infrastructure, with pull request review. Provisioning also means that if a stack is rebuilt, the dashboards come back.
Decisions and trade-offs
Logs as a metric source. Counting log lines to derive rates is convenient, but it is slower and costlier than real metrics. For SLOs and paging alerts, emit counters from the application or collector and keep logs for explanation. The SLI, SLO and SLA glossary entry explains why that distinction matters.
One dashboard per team or shared. Shared base dashboards with variables keep consistency. Team-specific pages should be additions, not copies.
Auto-refresh and time ranges. A dashboard that auto-refreshes heavy log queries every few seconds loads the backend. Sensible defaults, such as 1 hour with a 1 minute refresh, protect it.
When not to build many dashboards. For a small team with a handful of services, two or three dashboards and well-saved searches are enough.
Common failure modes
- Vanity charts that look impressive and drive no action. We drop them in review.
- Cardinality explosions from grouping on user IDs or full URLs.
- Inconsistent time zones between panels. Everything is set to UTC with local display only where needed.
- Heavy queries that time out during incidents, exactly when they are needed. We test dashboards at incident-level load.
- No owner. Every dashboard carries an owner tag and a review date.
Keeping dashboards trustworthy over time
Dashboards decay quietly. A field gets renamed, a service moves namespace, and a panel that used to show a line now shows nothing, which looks the same as “no errors”. We guard against that in three ways.
First, every panel that counts errors has a companion panel for total log volume from the same selector. If volume drops to zero, the empty error chart is understood as missing data, not good news. Second, dashboards live in version control and are provisioned, so a renamed label shows up as a pull request that someone has to review rather than an unnoticed edit in a browser. Third, we agree an owner for each dashboard and review the list periodically, deleting views nobody opened during an incident or a release.
A short list of dashboards that people trust beats a large folder where nobody knows which one is current.
Delivery and handover
You receive the dashboards as code, a short guide explaining each view and the query behind it, and a session with your on-call engineers using a past incident as the exercise. If you prefer us to keep them current, that sits within Managed DevOps.
How to tell it is working
On-call engineers reach the failing log lines within a minute or two, release checks take minutes rather than a meeting, and the dashboard list is short and used. To review your current views, book a consultation.