What Prometheus is and why teams adopt it
Prometheus is a metrics system built around a simple idea: every service exposes its current numbers over HTTP, and the server scrapes them on a schedule. Each sample is stored as a time series identified by a metric name and a set of labels, and PromQL lets you aggregate and alert on those series. There is no agent to configure per host and no schema to design up front.
Teams adopt Prometheus because it fits dynamic infrastructure. Service discovery finds new pods and instances without manual registration, labels let you slice by namespace or version, and the alerting model is expressive enough to encode real service level objectives. It is also free to run and widely supported, so most databases, proxies and frameworks already expose compatible metrics.
If you want help rolling it out, our Grafana and Prometheus monitoring service covers the full engagement. This page explains how we run it in production and when we would suggest something else.
How we run Prometheus in production
Deployment shape
On Kubernetes we deploy through the Prometheus Operator, usually via a Helm chart, so scrape targets and alert rules are Kubernetes resources reviewed in pull requests. Our walkthrough of the Prometheus Operator Helm deployment shows the layout. Teams define a ServiceMonitor next to their application, and monitoring follows the code instead of living in a separate repo nobody owns.
A minimal example:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: checkout-api
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: checkout-api
endpoints:
- port: metrics
interval: 30s
Labels and cardinality
Every unique label combination creates a new series, and memory scales with active series. We agree a label convention early: environment, cluster, service and team are fine, while user IDs, email addresses, raw URLs and trace IDs are not. Histograms deserve care too, because each bucket multiplies the series count. When a metric is already too expensive, relabelling rules can drop labels at scrape time.
Instrumentation
Good alerts need good metrics. We follow the request, error and duration approach for services and utilisation, saturation and errors for resources. Our guide to application metrics instrumentation shows how we add counters and histograms to existing code without turning every function into a metric.
Alerting that people trust
Most monitoring failures are alerting failures. Pages that fire every night teach engineers to ignore them. We write alerts against symptoms (error ratio, latency percentile, burn rate against an SLO) and keep resource alerts such as CPU for tickets or dashboards. Each alert carries a severity, an owner and a link to a runbook.
Alertmanager handles grouping, inhibition and routing. A critical alert might go to an on-call rotation while a warning goes to a team channel during working hours. Details are in our notes on Alertmanager severity routing and silences, and our article with Prometheus alerting rule examples gives concrete expressions to start from.
Security and access
Prometheus has no built-in authentication or authorisation on its own endpoints, so access is controlled around it. We keep the server and its API behind an authenticating proxy or internal network only, restrict who can reach exporters, and use network policies in clusters. Scrape credentials live in Kubernetes secrets, not in rule files. Metrics themselves can leak information such as customer identifiers or internal hostnames, so we review labels as part of security hygiene.
Cost and capacity
Self-hosted Prometheus costs compute, disk and engineering time rather than a per-metric fee. Memory and disk grow with series count, scrape interval and retention, so those three settings are the levers. A longer scrape interval for low-value targets, dropped unused metrics and a sensible local retention window keep a server small.
Storage needs a decision. Two weeks of local retention is often enough for debugging and alerting. If compliance or capacity planning needs more, we add remote write into Thanos or Mimir, accepting the extra components that come with it. We do not quote savings, because they depend on how noisy your metrics are today.
Day-2 operations and failure modes
Common problems are predictable. Disks fill because retention was never sized. A new label explodes cardinality after a deploy. Alert rules reference a metric that was renamed, so the alert silently never fires. Scrapes fail because a network policy changed.
We guard against these by monitoring Prometheus itself (scrape failures, rule evaluation time, series count, WAL and disk usage), setting sample and series limits, testing rules with promtool in CI, and adding a deadman alert that fires if the alerting pipeline goes quiet. Upgrades follow the release notes and are rehearsed in a non-production cluster, with persistent volumes snapshotted first.
Scaling beyond one server
A single Prometheus handles a surprising amount, but eventually you hit limits on memory or on needing a view across clusters. Options are functional sharding (separate servers for infrastructure and applications), federation of summary metrics, or remote write to a scalable backend. We prefer the simplest option that meets the need, and we always test failover before relying on it.
When not to choose Prometheus
Prometheus is a poor fit for event or log data, high-cardinality analytics, or exact billing figures, since scraping samples at intervals can miss short spikes. If your team has no appetite to operate monitoring infrastructure, a hosted platform such as Datadog removes that burden in exchange for usage-based cost and some vendor lock-in. Many teams land in the middle: Prometheus for metrics, Grafana for dashboards, OpenTelemetry for traces, and a managed backend for storage.
For a view of how this fits your wider platform, see our observability technologies or book a consultation and we will review your current setup.
Official documentation: prometheus.io