Why monitoring without a responder fails
Most startups have dashboards. Fewer have someone who looks at them at 3am. A Grafana board that nobody opens is a record of what went wrong, not protection. The usual failure pattern is predictable: alerts go to a shared Slack channel, the channel gets muted after a noisy week, and the first real outage is reported by a customer.
Our 24/7 infrastructure monitoring closes that gap by pairing the tooling with a person who is accountable for it. It is one part of our wider Managed DevOps service, and it feeds directly into incident response.
What good monitoring looks like
Good monitoring answers three questions fast: is the user affected, where is the fault, and what changed. We design for those, in that order.
- Symptom first. Page on error rate, latency and availability against an objective. Causes such as disk, CPU and memory pressure are signals for dashboards and tickets unless they predict an imminent outage.
- Every alert is actionable. If the on-call engineer cannot do something about it, it is not a page.
- Every alert has a runbook link. See how we write them in our runbooks page.
- Coverage is explicit. We keep a list of what is monitored and what is deliberately not, so gaps are decisions rather than surprises.
How it works in practice
The metrics layer
Prometheus scrapes exporters and application endpoints on an interval. On Kubernetes we deploy it through the community kube-prometheus-stack Helm chart, which brings Prometheus, Alertmanager, Grafana, node_exporter and kube-state-metrics in one release. On plain VMs we run node_exporter and a standalone Prometheus. External checks use the Blackbox exporter to probe HTTP endpoints, TLS expiry and DNS from outside your network.
For longer retention or multiple clusters we add remote storage such as Thanos or Mimir. We do not recommend it by default: a single Prometheus with sensible retention is easier to run and covers most scale-ups.
The alert layer
An alert rule written for a symptom looks like this:
groups:
- name: checkout-slo
rules:
- alert: CheckoutHighErrorRate
expr: |
sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m])) > 0.02
for: 10m
labels:
severity: page
team: payments
annotations:
summary: "Checkout 5xx ratio above 2% for 10 minutes"
runbook_url: "https://runbooks.example.com/checkout-5xx"
The for clause stops a brief blip from paging anyone. The severity label drives routing. The runbook URL means the engineer who is woken up does not have to guess. More patterns are in our guide to Prometheus alerting rules.
The paging layer
Alertmanager groups, deduplicates and routes. Rules with severity: page go to PagerDuty or OpsGenie; severity: ticket goes to your tracker; informational alerts stay on dashboards. Escalation policies make sure an unacknowledged page moves to a second person rather than disappearing.
Dashboards
Grafana dashboards are provisioned from Git, not clicked together in the UI, so they survive a rebuild. We build one overview per service (the golden signals), one per platform component, and one for on-call triage that shows recent deploys next to the graphs. For the Kubernetes side, see our monitoring walkthrough.
Decisions and trade-offs
Self-hosted Prometheus versus a hosted platform. Self-hosted is cheaper at scale and portable, but someone must run it. That is part of what you are buying from us. If your team is five engineers with no platform skills and a small fleet, a hosted service such as Grafana Cloud may be the better first step, and we can manage that instead.
Metrics only, or logs and traces too? Metrics tell you something is wrong. Logs and traces tell you why. We usually add Loki for logs early because it is cheap and shares Grafana. Tracing is worth adding once you have several services calling each other. Our logging and OpenTelemetry service covers that depth.
When not to buy this. If you have a single low-traffic internal tool, uptime checks and a free tier of a hosted monitor are enough. Round-the-clock coverage matters when downtime costs revenue or breaks a contract.
What we monitor, layer by layer
Coverage is easier to reason about when you split it into layers, because each layer fails differently and needs different signals.
| Layer | Typical signals | Usual alert type |
|---|---|---|
| Edge and DNS | Probe success, TLS expiry, DNS resolution | Page on user-visible failure, ticket on expiry warnings |
| Load balancer | 5xx ratio, target health, request latency | Page against an objective |
| Application | Error rate, latency percentiles, queue depth | Page on SLO burn |
| Platform (Kubernetes, VMs) | Node readiness, pod restarts, pending pods | Ticket, page if capacity is exhausted |
| Data stores | Replication lag, connection saturation, disk growth | Page on lag or exhaustion, ticket on trend |
| Pipeline and deploys | Failed deploys, rollout stalls | Notify the owning team |
The point of the table is that a page should almost always come from the top three rows. The lower rows explain why the top rows broke, and they are the first place we look during an incident.
Deploy markers
One of the cheapest improvements is annotating Grafana graphs with deploy events. When latency steps up at the exact minute a release went out, the diagnosis takes seconds instead of an hour of guessing. We wire this from your pipeline, which is why monitoring and CI/CD ownership work best under the same team.
Common failure modes
- Alerting on causes. A CPU alert at 85% fires nightly during a batch job and trains everyone to ignore pages. We move these to dashboards.
- Single point of failure in the monitor. If Prometheus runs on the cluster it watches, a cluster outage silences the alarm. We add an external heartbeat (a watchdog alert that must always fire) so silence itself is detected.
- High cardinality. Labels such as user ID or request path blow up memory. We review metric labels before they reach production.
- Stale routing. People leave, schedules rot. We review on-call schedules and contact paths monthly.
Delivery and handover
Week one is inventory and baseline dashboards. Weeks two to four are rule writing and paging tests, where we deliberately trigger alerts to prove the route works end to end. After that we run a standing cadence: weekly review of what fired, and a monthly summary covered in reporting.
Everything is stored in your repositories. You can see our monitoring and observability service for the standalone project version.
How to tell it is working
- Mean time to acknowledge a page stays low and does not drift upward.
- The ratio of pages that needed action to pages that did not keeps improving.
- Customers rarely tell you about an outage before your alerts do.
- Alert counts per week fall as root causes get fixed, not because rules were muted.
If you want an honest read on your current setup, our DevOps maturity assessment is a quick starting point, or you can book a consultation.