The slow way logging bills grow
Log storage rarely breaks on a single day. It grows quietly. Daily indices pile up, nobody remembers why a particular index is kept, disks creep toward their limits, and eventually a watermark triggers and the cluster stops accepting writes during a busy period. The usual response is an emergency delete and a vow to automate it later.
Retention work is that automation, done deliberately. It is a core part of our logging, ELK, Loki and OpenTelemetry service and works hand in hand with log routing by storage tier.
What good looks like
- Retention is decided per log class and signed off by the people who rely on it.
- Data moves through hot, warm, cold and delete phases automatically.
- Shards are a healthy size and the shard count per node is bounded.
- Long-term history sits in cheap object storage and has been restored at least once in testing.
- Disk and shard metrics are alerted well before a limit is hit.
Start with requirements, not tooling
We ask four questions per log class:
- How far back do engineers actually search during incidents and post-mortems?
- What do auditors, customers or regulators require, and for how long?
- Is there a legal obligation to delete after a period, such as personal data limits?
- How fast must old data come back when someone asks for it?
Answers differ. Application debug logs may need two weeks. Authentication events may need a year. A privacy requirement can push the other way, requiring deletion on a timer, which links to our work on log scrubbing for GDPR and HIPAA. Retention is therefore a ceiling as well as a floor.
How it works in Elasticsearch
Data streams and rollover
Time-series logs belong in data streams. A data stream writes to a backing index and rolls over when conditions are met, so you never need date-named indices managed by scripts. An ILM policy attached through an index template controls the lifecycle.
PUT _ilm/policy/app-logs
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" }
}
},
"warm": {
"min_age": "7d",
"actions": {
"forcemerge": { "max_num_segments": 1 },
"set_priority": { "priority": 50 }
}
},
"cold": {
"min_age": "30d",
"actions": { "set_priority": { "priority": 0 } }
},
"delete": {
"min_age": "90d",
"actions": { "delete": {} }
}
}
}
}
The ages and sizes are examples. Rollover size is chosen from a target shard size appropriate to your hardware and query behaviour. Tier movement relies on node roles and data tier preferences being set correctly, which we configure with the cluster, as in our platform deployment page.
Snapshots and searchable snapshots
Snapshot lifecycle management writes regular snapshots to object storage. They serve two purposes: disaster recovery and cheap long-term retention. Where your licence and version allow it, searchable snapshots can make cold or frozen data queryable directly from object storage at lower cost, trading some query speed. Check the feature and licence tier for the version you run before designing around it.
Cleaning up old data
For clusters that already have daily indices, we create a migration plan rather than a big bang. Existing indices can be attached to policies, or reindexed into data streams during quiet hours with throttling. Each step is validated with document counts.
How it works in Loki
Loki is simpler. Chunks live in object storage and the compactor applies retention. A global period is set in limits_config, and per-stream overrides use selectors:
limits_config:
retention_period: 336h
retention_stream:
- selector: '{namespace="kube-system"}'
priority: 1
period: 72h
- selector: '{log_class="audit"}'
priority: 2
period: 8760h
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: s3
Check the exact keys against your Loki version because configuration has evolved. Then align the bucket lifecycle rules so they do not delete objects sooner than Loki expects. Mismatch between the two layers is a classic source of “missing logs”.
Decisions and trade-offs
Keep everything or sample. Keeping all data is simplest and sometimes legally needed. For non-critical logs, shorter retention is the largest cost lever you have.
Warm and cold tiers: worth it? They save money when volume is large and old data is rarely queried. For small clusters the extra node roles add complexity for little benefit, and one tier with a delete phase is enough.
Self-managed snapshots versus managed backup. Snapshots are easy to configure and easy to forget to test. We always test a restore, as with the practices on our backup and disaster recovery page.
When not to over-engineer. A team shipping a few gigabytes a day with 14 days of retention needs one policy and an alert on disk usage, not a five-phase lifecycle.
Common failure modes
- Policy attached but never applied because the template did not match new indices. We verify with the ILM explain API.
- Rollover stuck because of a missing alias or a read-only block caused by disk watermarks.
- Over-sharded indices from daily indices on low-volume services. Data streams with size-based rollover avoid this.
- Deletes that violate holds. Compliance streams are tagged and excluded from cost-driven deletion.
- Unrestorable snapshots, found only during an emergency. Scheduled restore tests catch this.
Reviewing retention as the system changes
A retention matrix is correct on the day it is signed off and drifts afterwards. New services appear, a regulator changes expectations, or a product starts logging more per request. We recommend a short review at a regular interval, using three questions. Which indices or streams grew fastest? Did anyone actually query data from the older tiers? Did any restore test fail or take longer than planned?
The answers drive concrete changes: moving a stream to a shorter hot phase, splitting a noisy application into its own policy so it does not inflate the shared one, or lengthening retention for a source that auditors keep asking about. Because the policies are code, each change is a reviewed pull request with a stated reason.
The goal is that nobody has to guess why a particular number of days was chosen.
Delivery and handover
You receive the retention matrix signed off by your stakeholders, the policies and templates as code, alerts for disk, shards and ILM errors, and a runbook for changing retention and restoring data. We run a restore drill with your team so the procedure is familiar.
How to tell it is working
Disk usage follows a predictable curve, shard counts stay in bounds, ILM errors are zero, and a restore from last month’s snapshot succeeds when tested. If your cluster is creeping toward its limits, contact us.