The two ways log alerts go wrong
The first failure is silence. A service starts throwing exceptions at 2am, logs record every one, and nobody is told because nobody wrote a rule. The second failure is noise. A rule matches the word “error”, fires hundreds of times a day, and soon the channel is muted. Both are common, often in the same company.
Good log alerting is a design exercise: decide which conditions need a human, express them as rates and patterns, and send them to a person who can act. It builds on the dashboards and queries you already use, and belongs to our wider logging, ELK, Loki and OpenTelemetry service. For the metrics side, see our guide to Prometheus alerting rules.
What good looks like
- Alerts describe user or system impact in plain words, with the service and environment named.
- Each alert has an owner, a severity and a runbook link.
- Thresholds use ratios and windows, with a minimum volume to avoid flapping at low traffic.
- Absence alerts exist for the critical sources.
- Alert rules are in Git, reviewed, and periodically pruned.
Types of log alert that earn their place
Error ratio. Share of requests logging an error over a window, per service. Better than a raw count because it scales with traffic.
Specific fatal pattern. A known message that always means trouble, such as out-of-memory, a failed migration, or a certificate expiry warning from a dependency.
Absence. No log lines from a service, namespace or node in a defined window.
Security and audit events. Repeated failed logins, privileged role changes or access from an unexpected region, covered in more detail on the audit log collection page.
Volume anomaly. A sudden jump in log volume often signals a loop or a debug flag left on, and protects the budget as well as the system.
How it works in practice
Loki: ruler and Alertmanager
The Loki ruler evaluates LogQL rule files and forwards alerts to Alertmanager, in the format of Prometheus rules:
groups:
- name: checkout-logs
rules:
- alert: CheckoutErrorRatioHigh
expr: |
sum(rate({namespace="checkout"} | json | level="error" [5m]))
/
sum(rate({namespace="checkout"} [5m])) > 0.05
for: 10m
labels:
severity: page
team: payments
annotations:
summary: "Checkout error ratio above 5% for 10 minutes"
runbook_url: "https://wiki.example.com/runbooks/checkout-errors"
- alert: CheckoutLogsMissing
expr: |
sum(count_over_time({namespace="checkout"}[10m])) == 0
for: 5m
labels:
severity: ticket
The 5% figure is a placeholder to be replaced by a value based on your baseline. Note that an absence rule built on == 0 only fires when the query returns a series, so we test absence rules with absent_over_time or equivalent in your version and confirm behaviour by stopping a test service. The routing then follows ordinary Alertmanager configuration, as in our Grafana and Prometheus monitoring work.
Grafana alerting
Grafana-managed alerts can query Loki, Elasticsearch, Prometheus and others in the same rule, with notification policies that route by label. This is convenient when you want one place for everything. Rules can be provisioned from files or Terraform, so they are not lost with the UI.
Kibana rules and Watcher
In Kibana, the alerting framework offers rule types such as log threshold and Elasticsearch query rules, with connectors for Slack, email, webhooks and on-call tools. Watcher, the older API, remains in use in many clusters and runs a scheduled search, a condition and actions. Which features are available depends on your licence tier and version, and some connectors are paid features. We confirm that before design. For new work we usually prefer Kibana rules, and we migrate existing watches where there is a reason.
Routing and runbooks
An alert that reaches the wrong person is almost as bad as none. Labels such as team, service and severity drive routing. Page severity goes to on-call, ticket severity creates a work item, and informational goes to a channel. Each alert links to a runbook with the first checks, the dashboards to open, the trace and log queries to run, and escalation contacts. The same principle appears in our SRE and incident response service.
Decisions and trade-offs
Log alert or metric alert. If you can emit a counter, do so. Metric rules evaluate cheaply and consistently. Log queries over large ranges are heavier and slower.
Short window or long window. Short windows detect quickly but flap. Combine a short rate with a for duration and a minimum volume.
Alert in the log platform or central alerting. One alerting system with one routing tree is easier to manage than three. We usually centralise on Alertmanager or Grafana alerting.
When not to alert. If nobody would act at 3am, it is not a page. Make it a ticket or a dashboard item.
Common failure modes
- Alerting on single lines, which creates floods during incidents.
- No minimum volume, so a low-traffic night causes a high ratio from two requests.
- Silent rule failures when a query errors or times out. We alert on rule evaluation failures.
- Duplicates because the same condition fires from logs and metrics. Deduplication and inhibition rules help.
- Stale rules pointing at renamed services. Monthly review catches them.
Testing an alert before it pages anyone
An alert rule is code, and it should be tested like code. Before enabling a rule we run its query over several weeks of historical logs and count how many times it would have fired. If the answer is “dozens”, the threshold or the window is wrong. If the answer is “never”, we check that the rule would have fired during a known past incident, otherwise it may be watching the wrong label.
For Loki rules, the ruler can be pointed at a staging Alertmanager route first, so the notification path is exercised without waking anyone. We also write down, for each rule, what the responder should check first and what a false positive usually looks like. That note goes in the runbook link on the alert.
New rules start in a notify-only channel for a settling period. Promotion to paging happens only when the rule has proved useful, which keeps the pager credible.
Delivery and handover
You receive the rule set in your repository, notification routing, runbook templates, an alert catalogue explaining each rule’s intent, and a tuning review after the first few weeks. If you want someone to carry the pager, see Managed DevOps.
How to tell it is working
Pages are infrequent and almost all lead to action. Incidents are detected by alerts rather than customers. The number of muted or ignored alerts is zero, and you can explain every rule to a new engineer in a sentence. To review your current alerts, get in touch.