What goes wrong without a proper collector
Containers write to stdout and the container runtime writes that to files on the node. When a pod is evicted, a node is replaced by autoscaling, or a deployment rolls, those files go with it. Teams that rely on kubectl logs lose the one piece of evidence they needed, usually during the incident that mattered. Teams that bolt on a quick collector often hit the next problem: it runs without limits, drops records during a backend outage, or silently stops reading a rotated file.
A collector is small in code but central in effect. It is the first link in the chain described on our logging, ELK, Loki and OpenTelemetry service page, and anything it loses is gone for good.
What good looks like
- One agent per node, deployed through a DaemonSet, so scaling the cluster scales collection.
- Every record carries namespace, pod, container, node and selected labels.
- A backend outage causes delay, not loss, up to a stated duration.
- The agent has resource requests and limits, so it cannot starve workloads.
- The collector’s own health is monitored, including dropped and retried records.
How it works in practice
The pipeline
Fluent Bit has a simple model: inputs produce records, filters change them, outputs deliver them. For Kubernetes the standard shape is the tail input reading /var/log/containers/*.log, a parser for the container runtime format, the kubernetes filter to attach metadata from the API server, then one or more outputs.
[SERVICE]
Flush 1
Log_Level info
Parsers_File parsers.conf
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.max_chunks_up 128
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 50MB
storage.type filesystem
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name es
Match kube.*
Host elasticsearch.logging.svc
Port 9200
Logstash_Format On
Retry_Limit False
storage.total_limit_size 5G
Adjust the output block for Loki, Kafka or S3. The values shown are a starting point to tune, not a universal answer. Check the option names against the Fluent Bit version you deploy, since they change between releases.
Deploying it
We install the official Helm chart with values stored in Git and applied by Argo CD or Flux, so changes are reviewed and reversible. The pod mounts /var/log read-only, runs with a service account that can read pod metadata, and uses a hostPath or emptyDir for its buffer. Tolerations let it run on tainted nodes such as GPU or system pools, which matters because those nodes are often exactly where problems occur. See the glossary entry on taints and tolerations for background.
Resource and reliability settings
Four settings decide behaviour under stress. Mem_Buf_Limit bounds memory used per input. storage.type filesystem spills to disk. storage.total_limit_size caps disk use so the buffer cannot fill the node. Retry_Limit controls how long to keep trying. Together they define what happens when the backend is slow: the agent either applies back-pressure to the input or drops oldest data, and you should choose that deliberately.
Decisions and trade-offs
Fluent Bit or Logstash. Fluent Bit is written in C, starts fast and uses little memory, which suits one-per-node deployment. Logstash runs on the JVM and offers an extensive filter and plugin ecosystem. Running it on every node is wasteful. If you need heavy grok parsing or enrichment, we place Logstash as a central aggregation tier and keep the node agent thin.
Fluent Bit, Alloy or the OpenTelemetry Collector. If you are standardising on Grafana tooling, Grafana Alloy is a reasonable collector. If you are standardising on OpenTelemetry for traces and metrics, the OpenTelemetry Collector can also gather logs, which reduces the number of agents. We pick based on your destination and the rest of your telemetry plan, covered in our page on OpenTelemetry instrumentation.
Parsing at the edge or the centre. Parsing JSON at the agent keeps the backend simple. Heavy regex parsing at the agent costs CPU on every node. The best answer is to emit structured JSON from applications, as in our structured logging guidelines, so little parsing is needed at all.
When not to use a DaemonSet. On managed serverless container platforms such as AWS Fargate you cannot run node-level agents. There you use the platform’s built-in log router or a sidecar. Our team can advise for Fargate and similar environments.
Common failure modes
- Multiline stack traces split into many records. We configure multiline parsers so a Java or Python exception stays as one event.
- Log rotation races. Tail positions and inode handling are tested against the runtime’s rotation behaviour.
- Disk full from buffering. The total limit is set below node free space and monitored.
- Metadata API overload. On big clusters every agent querying the API server adds load. We tune the metadata cache and use the kubelet where appropriate.
- Silent sources. A namespace that stops logging looks the same as a namespace that is healthy. We alert on expected-source absence.
Upgrading the collector without losing logs
A DaemonSet touches every node, so an upgrade is a fleet-wide change and deserves a plan. We roll the new version to a single labelled node pool first, using a separate DaemonSet selector, and compare record counts against a node still on the old version. Only when the counts match and the agent’s error metrics stay flat do we widen the rollout.
Two details catch teams out. First, the updateStrategy should be RollingUpdate with a small maxUnavailable, so a bad image never takes out collection everywhere at once. Second, configuration changes that alter parsers should ship together with a test log line that exercises them, because a parser that silently stops matching produces unstructured records rather than an error. We keep a short list of sample lines per application in the repository and replay them through the new configuration before every release.
If you ever need to roll back, the previous Helm revision is the rollback, and the on-disk buffer lets the old version resume where the new one stopped.
Delivery and handover
You get the Helm values, parsers and dashboards in your repository, a rollout log from the canary stage, and a short runbook covering upgrades, buffer inspection and how to add a new source. We pair with your engineers during the first upgrade so the process is learned, not just documented.
How to tell it is working
Compare the number of records leaving the agents to the number arriving in the backend. The agent’s own metrics for dropped, retried and errored records should be flat near zero. Node CPU and memory attributable to the collector should sit comfortably inside limits. Finally, kill a pod and confirm its last lines are still searchable. If you want this set up or audited, get in touch.