Container Networking Deep Dive Part 8: Network Policy Enforcement Compared, iptables vs eBPF

A Kubernetes NetworkPolicy is a Kubernetes API object. It says nothing about how it gets enforced. That’s deliberate: NetworkPolicy is an intent, and every CNI plugin that claims to support it is responsible for turning that intent into an actual packet-level drop decision, using whatever mechanism it already has for pod-to-pod forwarding, covered across Part 6 and Part 7. This post takes one policy and traces it through Calico’s default iptables-based enforcement and Cilium’s eBPF-based enforcement, to show where the two diverge in practice, not just in principle.

The policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Plain-language intent: pods labelled app: api-server in the production namespace accept TCP/8080 only from pods labelled app: frontend. Everything else, denied. This is a standard Kubernetes NetworkPolicy object, nothing Calico- or Cilium-specific about the resource itself.

Calico’s path: labels resolved to IPs, then to iptables rules

Calico’s controller watches the NetworkPolicy object, resolves podSelector: app: frontend against the current set of pods carrying that label (this is a live, continuously-reconciled set; pods come and go), and materializes the result as iptables rules referencing an ipset of the currently-matching IPs:

sudo ipset list cali40s:XyZ123

Name: cali40s:XyZ123
Type: hash:ip
Members:
10.244.1.20
10.244.1.21
10.244.2.14

sudo iptables -L cali-fw-cali abcd1234 -n

ACCEPT  tcp  --  10.244.0.0/16  10.244.1.20/32  match-set cali40s:XyZ123 src  tcp dpt:8080
DROP    all  --  0.0.0.0/0      10.244.1.20/32

The ipset is the mechanism that makes this tractable at scale. Calico isn’t writing one iptables rule per matching pod, it’s writing one rule that references a set, and updates the set’s membership (a cheap ipset add/ipset del) as pods are created or deleted, rather than regenerating the iptables rule itself. This is a meaningful improvement over kube-proxy’s own per-backend chain from Part 7, but the fundamental mechanism is the same family: a packet arriving at a pod’s veth still gets evaluated against an iptables chain, in order, and the enforcement point is the same FORWARD-adjacent hook every iptables-based tool in this series has used.

The practical consequence: a pod’s own IP has to actually appear in the ipset before traffic from it is permitted, which means there’s a real reconciliation lag between “a new frontend pod schedules and gets an IP” and “that IP is enforced as an allowed source.” Usually sub-second, but a real window, and a real thing to check when a freshly-scaled pod’s traffic is unexpectedly dropped for the first moment of its life.

Cilium’s path: labels stay labels, all the way to the datapath

Cilium’s enforcement doesn’t translate app: frontend into an IP set at any point. Every pod is assigned a numeric security identity derived directly from its labels when it starts, covered in the standalone Cilium post, and that identity, not the pod’s IP, is what the eBPF program checks:

cilium identity list | grep frontend

2210   k8s:app=frontend   k8s:io.kubernetes.pod.namespace=production

cilium bpf policy get 2456

DIRECTION   IDENTITY   PORT/PROTO   BYTES   PACKETS
Ingress     2210       8080/TCP     48213   612
Ingress     reserved:world          DENIED

The eBPF program attached to api-server’s veth checks the incoming packet’s carried security identity (embedded in the encapsulation header in tunnel mode, or derived from source in direct-routing mode) against this map, a single hash lookup, not an ipset membership check followed by a sequential iptables rule match. A pod’s identity is assigned by its labels at creation time, independent of whatever IP it happens to get, so there’s no equivalent reconciliation-lag window: the moment a new frontend pod exists with that label, its identity is already 2210, and the policy already covers it.

The tradeoff for this speed and lag-free enforcement is that it only holds inside a Cilium-managed cluster where every node runs the same Cilium agent maintaining consistent identity assignment. Mixing enforcement mechanisms (part of a cluster on Calico, part on Cilium) isn’t a supported migration path the way swapping one CNI’s config for another sometimes is; identity-based policy requires the eBPF datapath to be the thing doing forwarding everywhere policy needs to apply.

Watching a drop happen, each way

With Calico, diagnosing why a specific packet was dropped means correlating the ipset membership at the time against the iptables rule that matched: iptables -L -v counters incrementing on the DROP rule, cross-referenced against ipset list to confirm which pods were and weren’t in the allowed set at that moment:

sudo iptables -L cali-fw-cali_abcd1234 -n -v | grep DROP

   142  9384 DROP  all  --  *  *  0.0.0.0/0  10.244.1.20/32

142 packets, a byte count, no per-source detail beyond what’s in the rule itself. The investigation from here means checking which source IPs were sending to that pod and cross-referencing against what should have been in the ipset.

With Cilium, hubble observe --verdict DROPPED shows the same drop with the specific identity that was rejected, resolved back to labels, in one line:

hubble observe --verdict DROPPED --to-pod production/api-server-6b

Jul 23 15:41:02: default/reporting-svc (ID:3301, app=reporting) -> production/api-server-6b (ID:2210) Policy denied DROPPED (TCP)

The dropped packet’s source is immediately identifiable as app=reporting, a workload that was never granted access, without needing to cross-reference an ipset snapshot or a separate iptables counter at all.

What this means for choosing between them

Neither mechanism is wrong. Calico’s iptables/ipset enforcement is mature, well understood by anyone who already knows iptables from the rest of this series, and sufficient for the overwhelming majority of NetworkPolicy use cases: L3/L4 allow/deny between labelled pod groups. Cilium’s eBPF enforcement is faster at scale, has no IP-reconciliation lag window, and extends to L7 policy (HTTP method and path matching, shown in the standalone Cilium post) that iptables cannot express at all, because iptables has no concept of what’s inside a TCP payload. The cost of that capability is a datapath that’s genuinely newer, with a smaller pool of engineers who’ve debugged it in anger compared to two decades of iptables experience across the industry.

The next post moves from comparing mechanisms to using them under pressure: a vendor-neutral troubleshooting methodology for container networking generally, covering conntrack exhaustion, the VXLAN MTU problem from Part 4 showing up as an intermittent-looking failure, DNS resolution failures inside a pod, and cross-node connectivity checks that apply regardless of which CNI plugin a cluster runs.