Deploying Prometheus with the Operator and Helm on Kubernetes

Why the deployment model matters

Most Prometheus problems we are called in to fix are not exotic. A server running on an undersized node that restarts every few days. A configuration file edited by hand on one pod and lost at the next reschedule. A disk that filled at 3am because someone added a label with a user ID in it. The software is solid, but the way it is installed decides whether you can trust it during an incident.

Prometheus is the metrics engine of the monitoring and observability service we deliver, so it gets built first and built carefully. Everything else, including dashboards, alerts and SLOs, reads from it. For background on the project itself, see our Prometheus glossary entry.

What a good deployment looks like

We hold every Prometheus install to the same criteria:

• Declarative. Install and configuration live in Git and are applied by Helm, Argo CD or Terraform. No one edits a running pod.
• Sized from evidence. Memory, disk and retention come from a measured estimate of active series, not a copied example.
• Resilient enough for the job. Persistent storage, resource requests and limits, and a known behaviour on restart.
• Observable itself. Prometheus alerts when its own scrapes fail, rules error or the disk is nearly full.
• Upgradable. Chart and CRD upgrades are rehearsed, because CRDs are not upgraded by Helm in the same way as other resources.

Operator or plain Prometheus

The operator earns its place on Kubernetes because scrape configuration follows the workloads. A team ships a Deployment, a Service and a ServiceMonitor in the same chart, and monitoring appears with the service. Without that, someone has to remember to update a central config file, and eventually nobody does. For the Kubernetes-specific exporters and dashboards that sit on top, see our Kubernetes monitoring page.

How it works in practice

Installing from a values file

The common route is the community kube-prometheus-stack chart. We keep a values file per environment in your repository and install with pinned chart versions:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --version <pinned-chart-version> \
  -f values/production.yaml

A trimmed production values file usually covers storage, retention, resources and selector behaviour:

prometheus:
  prometheusSpec:
    retention: 15d
    replicas: 2
    resources:
      requests: { cpu: "500m", memory: "2Gi" }
      limits: { memory: "4Gi" }
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 100Gi

The values above are an illustration of structure, not a recommendation for your cluster. The setting serviceMonitorSelectorNilUsesHelmValues: false matters more than it looks: by default the operator only picks up ServiceMonitors that carry the Helm release label, so teams add monitors and wonder why nothing is scraped.

Declaring what to scrape

Application teams describe their metrics endpoint with a ServiceMonitor that sits beside the Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: production
spec:
  selector:
    matchLabels:
      app: checkout
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Alert and recording rules follow the same pattern as PrometheusRule resources, which we cover in the Alertmanager routing page and the alerting rules guide.

Decisions and trade-offs

Retention versus remote storage. Local retention of a couple of weeks is cheap and fast. Keeping a year of data on one server is a poor idea. If you need long-range trends, we add Thanos or Mimir rather than growing a single disk.

One Prometheus or several. A single instance is simpler to reason about. Splitting by team or environment limits blast radius and cardinality accidents, at the cost of more things to operate. We usually start with one per cluster and split when evidence says so.

Prometheus in agent mode. For edge clusters or when metrics ship to a central store, agent mode forwards samples without local querying. It is useful, but you lose local alert evaluation, so we only use it with a clear reason.

When not to self-host. If you have no one to look after it and your metric volume is small, a managed offering can be cheaper than the engineering time. We will say so. Self-hosting wins on cost at scale and on control, but it is a system you own.

Failure modes we design out

• Cardinality explosions. Labels with unbounded values multiply series. We review labels during onboarding and use sample_limit on scrape configs as a guard.
• Out-of-memory restarts. Memory limits set below real usage cause a restart loop and a gap in data. We size from measured series.
• Orphaned monitors. A renamed Service leaves a ServiceMonitor scraping nothing. A rule alerting on up == 0 for expected jobs catches it.
• CRD drift. Chart upgrades that skip CRD updates break the operator. We apply CRDs deliberately and test the upgrade in a non-production cluster first.
• No alert on Prometheus itself. If the monitor dies silently, nothing tells you. We add a dead-man’s-switch alert that must always fire, routed to an external check.

Delivery and handover

You receive the Helm values, the Git repository layout, ServiceMonitor templates for your common service types, and a short operations guide covering upgrades, scaling and restoring from a volume snapshot. If your team uses GitOps, we wire the chart into Argo CD or Flux so changes arrive through pull requests. We stay available for the first weeks of production traffic, because that is when the real series count and retention needs show up.

How to tell it is working

Look at scrape success rate across targets, rule evaluation duration against interval, the head series count trend, disk usage against the volume, and the time it takes a new service to appear in Prometheus after deploy. If a team can add a new application and see its metrics in the same day without asking us, the deployment is doing its job. To scope a deployment for your cluster, book a consultation.

References

• Introduction, Prometheus Operator
• kube-prometheus-stack, prometheus-community Helm charts
• Storage, Prometheus documentation