Log Aggregation

Log aggregation moves logs off the machines that produce them and into a central, searchable store. In Kubernetes this matters because pods are short-lived and their local logs disappear with them.

How it works

Kubernetes captures what each container writes to standard output and standard error, and the runtime stores it in files on the node. The kubelet rotates these files, and kubectl logs reads from them. This is node-local: once a Pod is evicted or removed, its logs are gone. The Kubernetes documentation states that cluster-level logging needs a separate backend, since Kubernetes provides no native log storage.

A typical pipeline has a collector (often a DaemonSet agent on each node, or a sidecar), optional processing such as parsing and redaction, a store with an index, and a query interface. Alerts and dashboards sit on top.

ELK (Elasticsearch, Logstash, Kibana) indexes log content and offers rich full-text search at greater operational weight. Grafana Loki indexes only labels, not log contents, and stores compressed chunks in object storage, which keeps the index small. Its query language, LogQL, is modelled on PromQL.

A practical example

An incident scenario: errors spike in a checkout service across six replicas. With central logs you filter by service label and time range, find the failing request ID, and follow it across services. Using Loki, a query might select the stream with a label matcher such as {app="checkout"} and then filter lines for the word error. Without aggregation you would query each pod separately, and any restarted pod’s earlier logs may already be unavailable.

Common mistakes

• Logging unstructured text, which makes fields hard to query. Prefer structured (JSON) output.
• Putting high-cardinality values, such as user IDs, into Loki labels.
• Logging secrets or personal data, then retaining it for a long time.
• Keeping everything forever with no retention or tiering policy.
• Relying on logs alone when metrics and traces would answer the question faster.

When to use it and when not to

Any production system with more than a couple of services benefits from central logs, and compliance often requires retention. Choose full-text indexing when you need rich ad hoc search, and label-based indexing when cost and scale matter more. For a single small service, a managed platform’s built-in logging may be enough.

See also

• OpenTelemetry
• Distributed Tracing
• Prometheus
• ELK stack vs Grafana Loki
• Logging services