Container Networking Deep Dive Part 7: eBPF-Native Networking and the End of kube-proxy

The standalone Cilium post on this site covers Cilium’s identity-based policy model, XDP acceleration, and Hubble observability in depth, and none of that is repeated here. This post picks up one thing that post mentioned but didn’t unpack: what kube-proxy actually does today, mechanically, and what specifically changes underneath a Service call once Cilium replaces it. Part 6 compared Flannel, Calico, and Cilium as pod-to-pod networking mechanisms; this is the Service-routing layer sitting on top of whichever one of those a cluster runs.

What kube-proxy actually does

A Kubernetes Service gets a stable ClusterIP that doesn’t correspond to any real interface anywhere. It’s a virtual address that has to be translated, per packet, to one of the Service’s actual backend pod IPs. kube-proxy is the component that makes that translation happen, and its default implementation does it with iptables:

sudo iptables -t nat -L KUBE-SERVICES -n | head -20

KUBE-SVC-XPGD46QRK7WJZT7O  tcp  --  0.0.0.0/0  10.96.0.1   /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0  10.96.10.5  /* production/api-server:http cluster IP */ tcp dpt:8080

Each Service gets its own KUBE-SVC-* chain. Inside that chain, one rule per backend pod, each with a --probability value forming a chain of weighted coin-flips: pod 1 gets picked with 1/N probability, pod 2 gets 1/(N-1) probability of the remaining traffic, and so on down the chain:

sudo iptables -t nat -L KUBE-SVC-NPX46M4PTMTKRN6Y -n

KUBE-SEP-AAA1  all  --  0.0.0.0/0  0.0.0.0/0  statistic mode random probability 0.33333
KUBE-SEP-BBB2  all  --  0.0.0.0/0  0.0.0.0/0  statistic mode random probability 0.50000
KUBE-SEP-CCC3  all  --  0.0.0.0/0  0.0.0.0/0

Every packet to that Service’s ClusterIP walks this chain, evaluating each probability rule in sequence until one matches, then gets DNAT’d to that specific pod’s real IP by the matched KUBE-SEP-* chain. This is functionally identical to the DNAT mechanism from Part 2’s port publishing, just generated per-backend instead of per-container, and it is a sequential match exactly like every other iptables chain covered in the nftables comparison from the standalone Cilium post. A Service with 500 backends is 500 rules walked, worst case, for every single packet to that ClusterIP, on every node, regenerated on every pod scale event.

kube-proxy’s alternative mode, IPVS, replaces the chain-walk with a real load-balancer’s hash table (the same in-kernel IPVS used for standalone load balancers), which fixes the O(n) lookup cost but still operates at the same point in the stack. A packet has to be built, routed to the ClusterIP, and translated, all after the fact.

What Cilium’s replacement changes

Cilium’s kube-proxy replacement doesn’t optimize the iptables chain-walk. It moves the decision earlier, to the point where the destination address is chosen, before a packet aimed at the ClusterIP is even constructed. This happens via eBPF programs attached to connect(), sendmsg(), and recvmsg() at the cgroup level, a hook point that sees the socket call itself, not a packet on the wire:

cilium bpf lb list

SERVICE ADDRESS    BACKEND ADDRESS
10.96.10.5:8080    10.244.1.12:8080 (weight 1)
                   10.244.2.31:8080 (weight 1)
                   10.244.3.8:8080  (weight 1)

When a process inside a pod calls connect() to 10.96.10.5:8080, the cgroup-attached eBPF program intercepts that call, looks up the ClusterIP in an eBPF map exactly like the one shown above, and rewrites the destination to a chosen backend pod’s real address before the socket layer ever hands a packet to the IP stack. The application’s own code still believes it connected to the ClusterIP (getsockname() and getpeername() are handled correctly by the same eBPF logic), but no packet, at any point, actually carries the ClusterIP as its destination. There is no DNAT to perform on the wire, because the translation happened before a wire-format packet existed at all.

This is a meaningfully different mechanism from “a faster iptables,” and it’s why Cilium’s kube-proxy replacement removes kube-proxy from the node entirely rather than making it quicker. kube-proxy itself doesn’t run, and neither do the KUBE-SERVICES/KUBE-SVC-* chains shown above:

sudo iptables -t nat -L KUBE-SERVICES -n

iptables: No chain/target/match by that name.

Watching a Service call resolve, live

Hubble, covered briefly in the standalone Cilium post, gets more useful here specifically because it can show the resolved backend for a Service call directly, rather than requiring a packet capture and manual DNAT correlation:

hubble observe --to-service production/api-server -f

Jul 23 14:02:11.334: default/checkout-7f9 (ID:1842) -> production/api-server-6b (ID:2210) to-endpoint FORWARDED (TCP Flags: SYN)

The destination shown is already the specific backend pod Cilium chose, api-server-6b, not the Service’s ClusterIP, because Hubble’s event stream is generated from the same eBPF program that made the load-balancing decision, at the point it made it. Diagnosing an uneven Service load distribution with kube-proxy’s iptables mode means correlating packet captures against the --probability chain by hand; with Cilium, hubble observe already shows which backend every call actually landed on.

Where this leaves the comparison

Flannel has no Service-routing story of its own at all. Every Flannel cluster still runs ordinary kube-proxy on top of Flannel’s pod network, iptables chain-walk and all. Calico can run either mode: ordinary kube-proxy, or, in newer Calico versions, its own eBPF dataplane that replaces kube-proxy the same conceptual way Cilium’s does, though with a different implementation. Cilium’s replacement was the first to reach broad production use and remains the reference implementation most commonly cited, which is why it’s the one detailed here. The mechanism, cgroup-level socket interception instead of packet-level DNAT, is the meaningful architectural point, more than which specific project implements it.

The next post takes the same three plugins from Part 6 and looks at the enforcement side specifically: how a NetworkPolicy resource gets turned into an actual drop decision, comparing Calico’s iptables-based enforcement against Cilium’s eBPF-based enforcement of the identical policy intent.