Why one big pipe becomes an expensive problem
The default log setup sends everything everywhere. Health checks, debug output, library chatter, security events and payment errors all land in the same index at the same price. Volume grows with traffic and with every new service, while the share of lines anyone ever reads shrinks. The usual symptom is a finance question: why has the logging bill become one of our largest line items?
Routing fixes this by recognising that logs have different value. It is one of the highest-leverage parts of our logging, ELK, Loki and OpenTelemetry work, and it also supports cloud spend work such as our EKS cost optimisation guide.
What good looks like
- A short list of named tiers, each with a purpose, a retention period and an owner.
- Rules written by namespace, service and severity, kept in version control.
- Audit and security logs separated from application logs, with stricter access.
- Every drop or sampling rule documented, with a reason and a review date.
- A cost report per tier and per team, so the people who generate volume see it.
A practical tier model
Most clients need four tiers, not ten.
| Tier | What goes there | Typical destination | Retention idea |
|---|---|---|---|
| Hot | Errors, warnings and key business events from production | Elasticsearch hot nodes or Loki with fast cache | Days to weeks |
| Standard | Normal info logs from production services | Loki, or Elasticsearch warm nodes | Weeks |
| Archive | Full copy of everything, plus compliance logs | S3, GCS or Azure Blob with lifecycle rules | Months to years as required |
| Drop or sample | Health checks, debug noise, known chatter | Discarded or sampled at a fixed ratio | None |
The retention column is deliberately qualitative. The right numbers depend on your incident history, regulation and budget, and are set on the retention and ILM page.
How routing works in practice
Tagging first, routing second
Routing is only as good as the metadata. The Kubernetes filter in the collector attaches namespace, pod, container and labels. Applications that emit structured JSON add level and service fields, which is why structured logging guidelines come before clever routing rules. If severity is buried in free text, you end up writing fragile regular expressions.
Rules in Fluent Bit
Fluent Bit’s rewrite_tag filter can re-tag records by field value, and later outputs match on the new tag. This example sends error-level records from the payments namespace to a dedicated destination:
[FILTER]
Name rewrite_tag
Match kube.*
Rule $kubernetes['namespace_name'] ^(payments)$ payments.$level true
Emitter_Name re_emitted_payments
[FILTER]
Name grep
Match kube.*
Exclude log GET /healthz
[OUTPUT]
Name es
Match payments.error
Host elasticsearch-hot.logging.svc
Index payments-errors
[OUTPUT]
Name s3
Match kube.*
bucket example-log-archive
region eu-west-1
s3_key_format /raw/$TAG/%Y/%m/%d/%H-%M-%S-$UUID.gz
compression gzip
Field names depend on how your parser exposes level. The grep filter shown drops health check noise before any output sees it. Test rules with a sample file before pointing them at production.
Rules in the OpenTelemetry Collector
If you run the OpenTelemetry Collector for logs, the routing connector or filter processor can make similar decisions based on resource attributes such as k8s.namespace.name, with severity available on the log record. The idea is identical, only the syntax differs. We follow whichever collector you standardise on.
Archive without paying search prices
Sending a complete compressed copy to object storage costs a small fraction of indexing it. It also gives you a recovery path: if a rule was too aggressive, the original line still exists. For Loki, the main store is already object storage, so the archive tier is often simply a longer retention setting for selected streams.
Decisions and trade-offs
Drop versus sample. Dropping health checks is almost always right. Dropping all debug logs loses context during a rare bug. Sampling a fixed fraction keeps a trace of behaviour with less volume.
Edge versus central routing. Edge routing cuts network and CPU cost but spreads rules across every node. A central tier gives one place to change rules but adds hops. We start at the edge for simple filters and add a central tier only when rules grow complex.
Per-team indices versus shared indices. Separate indices give clear access control and cost attribution, but too many create shard overhead in Elasticsearch. In Loki, tenant IDs or labels serve the same purpose with less overhead.
When not to bother. If total volume is small and the bill is low, routing complexity costs more engineer time than it saves. A single tier with sensible retention is fine until cost or noise becomes a real problem.
Common failure modes
- Silent loss. A mistyped match pattern sends logs nowhere. We compare record counts in and out per destination.
- Rules that rot. A service is renamed and its rule stops matching. Monthly review and a catch-all route prevent lost data.
- Debug off when you need it. We provide a documented, time-limited way to raise verbosity for one service.
- Compliance logs mixed in. Audit data must not be dropped by a cost rule. Those streams are protected and routed first.
- Regex severity detection. Guessing level from message text misclassifies lines. Structured fields avoid it.
Delivery and handover
You receive the tier definitions, routing configuration in your repository, a cost-per-tier dashboard and a runbook for adding a new service. Your teams learn how to label their logs, and the platform team knows how to change a rule safely. For monitoring spend over time, see how this connects to Managed DevOps reviews.
How to tell it is working
Volume into the expensive tier falls while the usefulness of searches stays the same or improves. Error investigations still find what they need. The archive holds complete data, and routing counters show no unexplained gaps. If your logging bill is rising faster than your traffic, talk to us.