What Falco is and why teams adopt it
Falco is a runtime security engine and a graduated project of the Cloud Native Computing Foundation. It watches what processes actually do on a host: files opened, programs executed, network connections made, privileges changed. It compares those events with rules and emits an alert when something matches.
This addresses a gap that build-time tools leave. A scanner such as Trivy tells you an image contains a known vulnerable package. It cannot tell you that, right now, someone has exec’d into a production pod and is reading a service account token. Falco can. Teams adopt it once they have the basics in place and want to know whether a workload is being abused.
The catch is that detection tools create work. Falco consulting is therefore less about installing a DaemonSet and more about making sure the alerts it produces are specific, owned and acted upon.
How we run Falco in production
Drivers and deployment
Falco collects system calls through a kernel module or an eBPF probe. We prefer the modern eBPF option where kernels support it, because it avoids building and loading kernel modules on every node image. We deploy with the official Helm chart as a DaemonSet, with explicit resource requests, a tolerance for any tainted nodes, and a priority class so it is not evicted first under pressure.
Managed Kubernetes imposes limits. On Amazon EKS managed node groups Falco works well, but serverless compute such as Fargate does not give you the node access it needs, so you would rely on provider detection there. We confirm this against your design before committing.
Rules as code
Falco rules are YAML. We keep the upstream rules unchanged, and put your exceptions and custom rules in a separate file under version control, delivered through the Helm values. A narrow custom rule might look like this:
- rule: Unexpected shell in payments namespace
desc: Detect an interactive shell in production payments pods
condition: >
spawned_process and container and
proc.name in (bash, sh, zsh) and
k8s.ns.name = "payments" and proc.tty != 0
output: >
Shell opened in payments pod
(user=%user.name pod=%k8s.pod.name container=%container.name cmd=%proc.cmdline)
priority: WARNING
tags: [container, shell, payments]
Field names and macros evolve between releases, so we test rules against the Falco version you run, and upgrade them through the same review process as application code.
Alert routing
Raw alerts to a chat channel become wallpaper. We use Falcosidekick to fan alerts out by priority: high-confidence events to your on-call tool with a runbook link, medium events to a security channel, and everything to long-term storage. If you already run Loki or an ELK stack, the full event stream lands there for investigation. Alert severity maps into your SRE and incident response process so there is a defined first responder.
Rolling out without blocking developers
Falco does not block anything by default, so it will not break deployments. The risk is different: developers get interrupted by alerts about legitimate debugging. Our rollout handles that.
- Observe first. Alerts go to a log destination only, on non-production and then a few production namespaces.
- Agree debugging rules. If engineers may exec into pods for troubleshooting, that is fine, but it should be logged and alerted at low priority, with higher priority in sensitive namespaces.
- Tune with the owners. Application teams explain what their containers legitimately do, and exceptions are added with a reason.
- Page last. Only when a rule has stayed quiet and accurate for weeks do we let it page anyone.
Tuning findings and common failure modes
Most noise comes from a few sources: package managers running at container start, agents and sidecars that write to unusual paths, and health check scripts. We write exceptions scoped by image, namespace or process lineage, not by disabling rules. Other failure modes:
- Dropped events. Under heavy system call load, Falco can drop events. We monitor its own metrics and size resources accordingly.
- Stale rules. New applications introduce new normal behaviour. Rules are reviewed on a schedule.
- No owner. Alerts that no one is accountable for are the most common cause of a failed programme.
- Over-trust. A quiet Falco does not prove nothing is wrong, since an attacker may avoid behaviour the rules look for.
Cost, capacity and day-2 operations
Falco is free software, so cost is node resources and operator time. Plan for per-node overhead, the storage and licensing of whatever receives the alerts, and someone to maintain rules. Day-2 tasks include upgrading Falco and its rules together, watching the engine’s metrics through Prometheus, and testing that alerts still arrive end to end, for example with a harmless test event after each upgrade.
Honest limits
Falco detects; it does not scan, prevent or investigate. It sees behaviour on the node, but sees less on serverless platforms. Its rules are heuristics and can be evaded by a careful attacker. It does not replace Kubernetes hardening, network policy or secrets management such as HashiCorp Vault.
When to choose something else
If you have no one to triage alerts, a managed detection service or cloud-native threat detection will give you more value than a self-tuned rule engine. If you mainly need to prevent bad deployments, begin with admission policy. If you want enforcement in the kernel, evaluate Tetragon.
Working with us
We typically start with a two-week observation pilot, deliver tuned rules and a response runbook, and then roll out across clusters. See our Kubernetes consulting service, or get in touch to scope a pilot.
Official documentation: falco.org