Implementing Kubernetes Network Policies Without Breaking Production

Why flat pod networking is a risk

Out of the box, Kubernetes networking is open: every pod can reach every other pod, in any namespace, on any port. That is convenient and, for production, dangerous. If an attacker gets code execution in one internet-facing pod, they can probe the database, internal admin tools and the cloud metadata endpoint without a single firewall in the way.

Network policies are the built-in answer. This page explains how we implement a Kubernetes network policy model as part of our Kubernetes consulting work, why rollouts so often go wrong, and when the effort is not worth it.

Step zero: does your CNI enforce policy?

A NetworkPolicy is only a declaration. The container network interface (CNI) plugin does the enforcing. If it does not support policy, objects are accepted and silently ignored, which is worse than having none because it feels safe.

Typical situations:

• EKS: the Amazon VPC CNI can enforce policy when the feature is switched on, or you can run Calico.
• AKS: choose a policy engine at cluster creation (Azure’s own, Calico or Cilium), and check it matches your CNI mode.
• GKE: Dataplane V2 (Cilium based) enforces policy, or Calico can be enabled.

Check, then test with a deliberate deny and a probe pod. Do not trust the YAML.

Native policy or Calico

Both implement the standard Kubernetes API. Calico also has its own resources with deny rules, ordered tiers, global policies and host endpoint protection. The decision:

Most startups are well served by native policy. We recommend Calico when a security team wants cluster-wide guardrails or deny semantics.

The rollout pattern we use

Applying default deny to a live namespace without preparation is how outages happen. The sequence:

1. Observe. Collect real flows, using CNI flow logs, Hubble on Cilium, or Calico flow logs. Include DNS, cluster add-ons, metrics scraping and calls to cloud APIs.
2. Label. Policies select by labels, so consistent labels on workloads and namespaces come first.
3. Write allows before denies. Draft the allow rules for a namespace and review them with its owners.
4. Apply default deny last, in non-production first, then production, one namespace at a time.
5. Watch. Monitor dropped connections and application errors for a full traffic cycle, including batch jobs.

A default deny baseline plus DNS and one real flow looks like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-ingress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress
      ports:
        - { protocol: TCP, port: 8080 }

The label on the DNS pods (k8s-app: kube-dns) is standard for CoreDNS but verify it in your cluster.

What breaks, and how we avoid it

• DNS. Covered above. Always ship a DNS allow with default deny egress.
• Ingress controller reachability. Your ingress controller namespace must be allowed to reach backends.
• Monitoring scrapes. Prometheus needs to reach metrics ports, or dashboards go blank. See cluster monitoring.
• Egress to managed services. Databases, object storage and third party APIs live outside the cluster, so egress rules need CIDR ranges or, on engines that support it, FQDN rules. IP-based rules go stale when providers change addresses.
• Webhooks and the API server. Admission webhooks and operators need control plane traffic allowed.
• Cloud metadata endpoint. Blocking pod access to the instance metadata address is a high-value, low-effort egress rule.

Worked example: a three tier namespace

Take a namespace with a web frontend, an API and a database. After default deny and DNS, the allow list is short and readable: ingress controller to frontend, frontend to API, API to database, and Prometheus to each metrics port. Anything else, such as the frontend talking straight to the database, is intentionally absent.

We then prove the negative. A temporary pod labelled like the frontend tries to open a connection to the database port and must be refused, while the API pod must succeed. Keeping these two checks in CI means a future label change that quietly widens or breaks access fails a build rather than surfacing in production. Teams often discover at this stage that a service was making a call nobody remembered, which is exactly the value of mapping flows first.

Trade-offs and when not to do it

Policies add friction. Every new service needs rules, and a mislabeled pod can lose connectivity in confusing ways. They do not encrypt traffic, and they do not understand HTTP paths or identities. If you need encryption between services or layer 7 rules, you are in service mesh territory, which we cover in Kubernetes consulting discussions case by case.

For a small cluster with one team and no sensitive data, a lightweight approach may be enough: default deny only on the namespaces holding databases and admin tools. For regulated workloads, a full model is usually worth it. Combine it with least privilege RBAC, since network and API access are separate attack surfaces.

Delivery and how to tell it works

We deliver the policy set as code in your repository, a labelling convention, automated connectivity tests that run in CI (for example a short-lived pod that must succeed on allowed paths and fail on denied ones), and a one page guide for requesting new flows. Success looks like: a probe pod in the wrong namespace cannot reach your database, deny counters are low and explainable, and no one needs a change ticket for ordinary traffic. If you want help designing your model, reach out.

References

• Network Policies, Kubernetes documentation
• Declare Network Policy, Kubernetes documentation