Logs without the bill that hurts
Logs are the cheapest signal to produce and the most expensive to keep. A growing platform can easily write more log data than business data, and teams discover this when a search vendor’s invoice or an Elasticsearch cluster’s resource use becomes a line item of its own. Meanwhile, during an incident, the logs they need are either not retained or take minutes to search.
Loki was built on a different trade-off: index very little, keep the raw data cheaply, and rely on labels shared with Prometheus to find the right slice quickly. We deploy it as part of our monitoring and observability service, and where your needs are heavier on search and analytics we also deliver ELK-based logging. Background on the concept is in the log aggregation glossary entry.
What good looks like
- Few, bounded labels.
cluster,namespace,app,container,level. Not request IDs or user names. - Structured logs. JSON or logfmt so fields can be parsed at query time.
- Trace IDs in log lines, so you can jump between logs and traces.
- Retention that matches value. Debug logs expire quickly; audit logs are kept as policy requires.
- Cheap, durable storage. Chunks and index in object storage.
- Queries that start from labels.
Agent choice
| Agent | Strengths | Consider when |
|---|---|---|
| Promtail | Simple, Loki-native, Kubernetes label discovery | Loki is your only destination |
| Fluent Bit | Light, fast, many inputs and outputs, strong filters | Logs also go to other systems, or you need heavy processing |
| Grafana Alloy | Unified collector for logs, metrics and traces | New builds, once you are comfortable with its configuration |
| OpenTelemetry Collector | Vendor neutral, multi-signal | You are standardising on OpenTelemetry |
Check the current status of each before committing, as Promtail has been moving towards Alloy in Grafana’s own guidance. We do not treat any of these as permanent, and keep the agent config small so swapping is feasible.
How it works in practice
Fluent Bit shipping to Loki
A Fluent Bit output that sets a minimal label set and drops noisy namespaces:
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
[FILTER]
Name grep
Match kube.*
Exclude $kubernetes['namespace_name'] ^(kube-system|monitoring)$
[OUTPUT]
Name loki
Match kube.*
Host loki-gateway.monitoring
Port 80
Labels job=fluentbit, cluster=prod-eu
Label_keys $kubernetes['namespace_name'], $kubernetes['labels']['app']
Auto_Kubernetes_Labels Off
Whether you drop namespaces at all is your decision; we suggest excluding only what you are sure you will never debug.
Loki storage and retention
Loki’s schema config and storage point at an object store. A shortened illustration of the shape:
schema_config:
configs:
- from: "2024-04-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
compactor:
retention_enabled: true
limits_config:
retention_period: 336h
ingestion_rate_mb: 8
max_streams_per_user: 50000
The numbers are examples of where the settings live, not recommendations. Retention of 336 hours is 14 days. We tune ingestion limits from measured volume so a runaway service cannot starve the rest.
Querying with LogQL
LogQL starts with a stream selector, then filters:
{namespace="production", app="checkout"} |= "error" | json | status >= 500
A metric query counts errors per minute from logs, which helps when an application cannot yet be instrumented:
sum by (app) (rate({namespace="production"} |= "level=error" [5m]))
Connecting logs to metrics and traces
In Grafana, a derived field on the Loki data source extracts trace_id and links to the trace in Tempo. Going the other direction, Tempo’s trace to logs link reads shared labels like app and namespace. This only works if the same label names are used across Prometheus, Loki and Tempo, which is why we standardise them early. See Grafana data source setup and distributed tracing.
Decisions and trade-offs
Deployment mode. Single-binary is fine for small volume. The simple scalable mode splits read and write paths and suits most teams. Fully distributed microservices mode is for large scale and brings real operational weight.
Loki or Elasticsearch/OpenSearch. If your engineers search by service, time and a phrase, Loki is usually sufficient and cheaper. If analysts run ad hoc aggregations over arbitrary fields, or you need rich full-text search and dashboards over logs, a search engine is the better tool. Some estates run both with different purposes.
Self-hosted or Grafana Cloud. Self-hosting keeps data in your accounts but you operate it. A hosted service removes that, at an ongoing cost.
Do not log everything. The cheapest log line is the one never written. We help teams cut noisy debug output at source.
Failure modes
- Label cardinality explosions from putting pod names or IDs in labels. Loki’s per-stream overhead then dominates.
- Missing multiline handling, so stack traces become hundreds of separate entries.
- Dropped logs under load when agents lack buffering or limits are too tight.
- Sensitive data in logs. Tokens and personal data written by applications; we add scrubbing and review.
- No retention set, so storage grows without bound.
Protecting the cluster from noisy logs
A single misbehaving service can emit enough log volume to dominate storage and slow queries for everyone. We treat this as a normal operating risk and apply guardrails. Per-tenant or per-stream rate limits in Loki stop one team from overwhelming the shared installation. At the agent, filters drop known low-value lines such as health check requests before they leave the node. Log levels are reviewed per service, since debug logging left on in production is the most common cause of surprise growth.
We also agree a simple ownership rule: the team that produces the volume sees it. A dashboard showing log bytes per namespace turns an abstract storage concern into a conversation between the right people, and usually leads to a fix within the same week.
Delivery and handover
You receive the agent configuration, Loki deployment as Helm values in Git, label conventions, retention policy, example LogQL queries and dashboards, and an operations guide. We review volumes and costs after a few weeks of production traffic.
How to tell it is working
Watch ingestion rate against limits, query latency for typical incident searches, storage growth, the share of log lines with a trace ID, and how often engineers find the needed logs during an incident without asking where they are. To discuss your logging setup, get in touch.