Prometheus Alerting Rules That Actually Matter (with Examples)

Most teams inherit a pile of alerting rules that fire constantly or never. Here are the rules we actually enable on every cluster, written for signal, not noise.

Alert severity levels

Define three levels and use them consistently:

# PrometheusRule CRD skeleton
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: production-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # must match your Prometheus selector
spec:
  groups:
    - name: pod.rules
      rules: [] # filled in below

Pod health rules

- name: pod.rules
  rules:
    # Pod crash loop
    - alert: PodCrashLooping
      expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.pod }} is crash-looping"
        description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted {{ $value | humanize }} times in 15 minutes."
        runbook_url: "https://clouddevopsol.com/runbooks/pod-crash-loop"

    # Pod not ready
    - alert: PodNotReady
      expr: |
        sum by (namespace, pod) (
          max by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown"})
          * on (namespace, pod) group_left(owner_kind)
          max by (namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})
        ) > 0
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} stuck in {{ $labels.phase }}"
        description: "Pod {{ $labels.pod }} has been non-running for 15 minutes."

    # Deployment unavailable
    - alert: DeploymentUnavailable
      expr: |
        kube_deployment_status_replicas_available
          / kube_deployment_spec_replicas < 0.5
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Deployment {{ $labels.deployment }} is less than 50% available"
        description: "{{ $value | humanizePercentage }} of expected replicas are available."

    # OOMKilled
    - alert: ContainerOOMKilled
      expr: increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) > 0
      labels:
        severity: warning
      annotations:
        summary: "Container {{ $labels.container }} OOMKilled"
        description: "Container {{ $labels.container }} in pod {{ $labels.pod }} was OOMKilled. Check memory limits."

Resource saturation rules

- name: resource.rules
  rules:
    # Node CPU high
    - alert: NodeCPUHighUsage
      expr: |
        (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (node)) > 0.85
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.node }} CPU > 85%"
        description: "CPU usage is {{ $value | humanizePercentage }} for 10 minutes."

    # Node memory high
    - alert: NodeMemoryHighUsage
      expr: |
        1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.90
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node {{ $labels.instance }} memory > 90%"

    # Node disk high
    - alert: NodeDiskHighUsage
      expr: |
        (node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}
          - node_filesystem_free_bytes{fstype!~"tmpfs|overlay"})
          / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} > 0.80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.instance }} disk {{ $labels.mountpoint }} > 80%"

    # PVC usage high
    - alert: PVCUsageHigh
      expr: |
        kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "PVC {{ $labels.persistentvolumeclaim }} is {{ $value | humanizePercentage }} full"

    # PVC usage critical
    - alert: PVCUsageCritical
      expr: |
        kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.95
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "PVC {{ $labels.persistentvolumeclaim }} is {{ $value | humanizePercentage }} full, CRITICAL"

Latency SLO rules

- name: slo.rules
  rules:
    # P99 latency SLO breach (replace with your metric name)
    - alert: HighP99Latency
      expr: |
        histogram_quantile(0.99,
          sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
        ) > 1.0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Service {{ $labels.service }} P99 latency > 1s"
        description: "P99 is {{ $value | humanizeDuration }}. SLO is 1s."

    # Error rate SLO breach
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) > 0.01
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Service {{ $labels.service }} error rate > 1%"
        description: "Error rate is {{ $value | humanizePercentage }}."

Certificate expiry rules

- name: certificate.rules
  rules:
    # cert-manager certificate expiring soon
    - alert: CertificateExpiringSoon
      expr: |
        certmanager_certificate_expiration_timestamp_seconds
          - time() < 14 * 24 * 3600
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "Certificate {{ $labels.name }} expires in < 14 days"
        description: "Certificate {{ $labels.name }} in {{ $labels.namespace }} expires on {{ $value | humanizeTimestamp }}."

    - alert: CertificateExpiringCritical
      expr: |
        certmanager_certificate_expiration_timestamp_seconds
          - time() < 3 * 24 * 3600
      for: 1h
      labels:
        severity: critical
      annotations:
        summary: "Certificate {{ $labels.name }} expires in < 3 days, URGENT"

Inhibition rules in Alertmanager

Use inhibition to silence lower-severity alerts when a higher one is firing. This prevents alert storms.

# In alertmanager config
inhibit_rules:
  # If a node is down, silence pod alerts on that node
  - source_matchers:
      - alertname = NodeDown
    target_matchers:
      - alertname =~ "Pod.*|Container.*"
    equal: [node]

  # If a deployment is unavailable, silence pod-level alerts for it
  - source_matchers:
      - alertname = DeploymentUnavailable
    target_matchers:
      - alertname = PodNotReady
    equal: [namespace, deployment]

  # Critical silences warning of same name
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: [alertname, namespace]

Runbook URLs in annotations

Every alert should link to a runbook. Teams that skip this spend hours re-diagnosing the same issue.

annotations:
  runbook_url: "https://clouddevopsol.com/runbooks/{{ $labels.alertname | lower }}"
  summary: "Short description for the notification"
  description: "Detailed description with {{ $value }} interpolated"

Silence strategies

Silence during planned maintenance without disabling rules:

# Create a silence via amtool (Alertmanager CLI)
amtool silence add \
  --alertmanager.url=http://alertmanager:9093 \
  --author="ops-team" \
  --comment="Planned maintenance window" \
  --duration=2h \
  alertname=~".*" namespace="staging"

Key takeaways

• Three severity levels (critical, warning, info) and stick to them, resist adding more.
• Every alert must have a runbook_url annotation or your on-call team will waste time re-diagnosing.
• Use for: 5m on most alerts, firing immediately causes false positives from transient blips.
• PVC saturation alerts at 85% warning and 95% critical give you time to act before a full disk causes data loss.
• Inhibition rules are essential in busy clusters; without them, one node failure generates hundreds of pod alerts.
• Test all alerts by deliberately breaking things in staging before relying on them in production.