Long-Term Prometheus Metric Storage with Thanos or Mimir

When one Prometheus stops being enough

A single Prometheus is a remarkably capable system. It stores recent data locally, answers queries quickly, and is simple to reason about. Its limits show up in predictable places. Retention is bounded by one disk. A restart or node loss leaves a gap. Each cluster has its own island of data, so comparing environments means opening several Grafana data sources. And a 90 day SLO window will not fit if local retention is 15 days.

Thanos and Mimir address these limits in different ways. We treat them as an option inside the monitoring and observability service, applied when the need is real. The base layer is described in our Prometheus deployment page.

What good looks like

• A stated reason. Retention target, number of clusters, availability goal.
• Cheap long-term data on object storage.
• A single query endpoint for Grafana across clusters and time ranges.
• Fast queries through caching, downsampling or compaction.
• Operated like production. It is a distributed system with its own alerts and runbooks.
• A simple exit. Prometheus remains the collector, so the store is replaceable.

Thanos or Mimir

Neither is universally better. If you have a handful of Prometheus servers and want a global view and long retention, Thanos is the lighter step. If you are ingesting from many sources, need tenant isolation or expect very large series counts, Mimir is the stronger fit. Mimir also shares tooling with Loki and Tempo, which helps if you standardise on the Grafana stack.

How it works in practice

Thanos with the Prometheus Operator

The Prometheus custom resource has built-in support for a Thanos sidecar, and you give it a bucket configuration through a secret:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  namespace: monitoring
spec:
  replicas: 2
  retention: 6h
  thanos:
    objectStorageConfig:
      name: thanos-objstore
      key: objstore.yml

With the sidecar uploading blocks to storage, local retention can be short, since history is served from the bucket. The object store file names the provider and bucket:

type: S3
config:
  bucket: example-metrics-prod
  endpoint: s3.eu-west-1.amazonaws.com

Credentials come from workload identity or IAM roles rather than static keys where your platform allows. Components we then deploy are Thanos Query (global view and deduplication of replicas), Store Gateway (reads historical blocks), and Compactor (compacts and downsamples), plus optional Ruler and Query Frontend.

Mimir with remote write

Prometheus forwards samples to Mimir, and Grafana queries Mimir as a Prometheus-compatible data source:

remoteWrite:
  - url: https://mimir.example.com/api/v1/push
    headers:
      X-Scope-OrgID: platform-prod
    queueConfig:
      maxSamplesPerSend: 5000

The X-Scope-OrgID header selects the tenant. Mimir’s Helm chart deploys distributors, ingesters, queriers, a compactor and a store gateway, backed by object storage. We size ingesters carefully because they hold recent data in memory.

Costs and complexity to expect

Long-term storage adds components, network traffic and a new failure domain. Object storage is inexpensive; the compute behind it is the real bill. Queries over long ranges can be slow and costly without caching and sensible dashboards, so we pair this work with recording rules for SLOs to avoid scanning raw data repeatedly. We show you a cost model from your own series count before any build.

When not to adopt

• You run one cluster and need less than a month of history.
• Your team has no capacity to operate another distributed system.
• A managed service would meet the need for less total effort. A hosted metrics offering, or a cloud provider’s managed Prometheus, can be the pragmatic answer, and we will say so.
• The real problem is cardinality. Adding storage to a cardinality problem is paying to keep a mess.

Failure modes

• Replicas not deduplicated, giving doubled values.
• Clock skew and external labels missing, so blocks collide.
• Compactor neglected, leaving storage and queries bloated.
• Cold-cache queries that time out on first load.
• Bucket permissions or lifecycle rules deleting data early.

Migrating without losing history

Moving to long-term storage should not mean starting from nothing. With Thanos, the sidecar uploads existing Prometheus blocks to object storage, so earlier data becomes queryable once the store gateway is in place. With Mimir, history is usually backfilled separately or accepted as a clean start, and we agree which with you in advance.

We stage the change. Prometheus keeps scraping and alerting locally throughout, so nothing in your alerting depends on the new system until it has proved itself. Dashboards are first pointed at the new query layer alongside the old data source, compared for a period, and only then switched over. If results differ, we find out why before anyone relies on them. Keeping the original path intact until the end makes rollback a simple change of data source rather than an emergency.

Delivery and handover

You get a documented architecture, Helm values and bucket configuration in Git, dashboards and alerts for the store itself, a cost estimate with the assumptions visible, and a runbook for restoring, upgrading and expanding. If you already use GitOps, we integrate with Argo CD or Flux.

How to tell it is working

Verify that queries across the full retention return data, that Grafana shows a single view across clusters with no duplicate series, that compaction keeps pace, and that a Prometheus restart leaves no gap in global results. To check whether you need this at all, talk to an engineer.

References

• Getting started, Thanos
• Get started with Grafana Mimir, Grafana documentation
• Storage, Prometheus documentation
• Remote write specification, Prometheus documentation