Container Networking Deep Dive Part 6: CNI Plugins Compared, Flannel, Calico, and Cilium

Part 5 established that every CNI plugin implements the same two verbs, and that the entire architectural difference between plugins lives in step 3 of the ADD call: how a pod’s IP becomes reachable from a different node. Flannel, Calico, and Cilium answer that question in three genuinely different ways. This is not a scored bake-off; it’s what each mechanism actually does to a packet.

Flannel: VXLAN, by default, and not much else

Flannel is deliberately the simplest of the three. Its default backend is VXLAN, the exact mechanism Part 4 walked through for Docker’s overlay driver, applied automatically to every pod instead of an opt-in network.

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Each node gets a slice of the cluster’s overall pod CIDR, a /24 out of a /16, typically, and Flannel’s daemon on each node maintains a simple mapping: “pod subnet X lives behind VXLAN tunnel endpoint at node Y’s IP.” A pod on node A sending to a pod on node B gets its packet picked up by the local flannel.1 VXLAN interface, encapsulated with node B’s IP as the outer destination, and decapsulated on arrival, identical in mechanism to Part 4’s Docker Swarm example, same 50-byte overhead, same MTU math.

ip -d link show flannel.1

8: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 vxlan id 1 ...

The 1450 MTU is Flannel doing the same overhead accounting Docker’s overlay driver does by default. What Flannel does not do is anything with network policy. There is no Flannel equivalent of a Kubernetes NetworkPolicy enforcement engine at all. Flannel answers “how does a packet get from pod A to pod B,” full stop, and nothing about whether it should be allowed to.

Calico: BGP-distributed routes, not a tunnel

Calico’s default mode skips encapsulation entirely in favor of a real routing protocol. Each node runs a BGP speaker (Bird, historically; increasingly Calico’s own), and nodes peer with each other, or with the cluster’s routers, to advertise “this node owns pod subnet X” as an ordinary BGP route:

calicoctl node status

Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.1.11    | node-to-node mesh | up    | 03:14:22 | Established |
| 10.0.1.12    | node-to-node mesh | up    | 03:14:19 | Established |
+--------------+-------------------+-------+----------+-------------+

A pod’s traffic to a pod on a different node is routed the ordinary way: a standard IP lookup against a route learned via BGP, forwarded out the node’s real interface, with no encapsulation, no extra header, no MTU tax at all. This works cleanly when the underlying network permits pod-subnet routes to actually be advertised and accepted between nodes (a flat L2/L3 network, or routers configured to peer with Calico), and falls back to VXLAN or IP-in-IP encapsulation automatically when it doesn’t. A cloud VPC that won’t route unrecognized subnets is the most common reason for the fallback, in which case Calico is back to Flannel’s overhead profile for exactly the same reason.

Where Calico earns its complexity is policy. Calico implements the full Kubernetes NetworkPolicy API, plus its own extended CRDs, enforced through iptables (or, in newer configurations, eBPF) rules generated from policy objects. This is covered in depth in the network-policy-enforcement post later in this series, and is the single biggest reason to choose Calico over Flannel: Flannel has nothing comparable at all.

Cilium: eBPF instead of either

Cilium, already covered in a standalone post on this site, takes a third approach: instead of a tunnel (Flannel) or BGP-distributed routes evaluated as ordinary IP lookups (Calico), Cilium attaches eBPF programs directly to each pod’s veth pair at the tc hook, and those programs make the forwarding decision in-line using hash-table lookups against eBPF maps rather than a routing table walk or an encapsulation/decapsulation step.

cilium status --wait

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

Cilium can run with or without encapsulation depending on the underlying network’s capabilities: VXLAN mode when the network won’t route pod CIDRs directly (same overhead profile as Flannel), or direct-routing mode when it will (same no-encapsulation profile as Calico’s BGP mode). But the forwarding decision itself, in either mode, happens in an eBPF program rather than a kernel route lookup or an iptables chain walk. The standalone Cilium post covers this datapath, the identity-based policy model, and Hubble observability in full detail and isn’t repeated here. What matters for this comparison is where Cilium sits architecturally: it can match either Flannel’s or Calico’s wire-level behavior, but replaces both plugins’ forwarding mechanism with the same eBPF datapath regardless of which mode it’s running in.

The comparison that actually matters

	Flannel	Calico (default)	Cilium
Cross-node mechanism	VXLAN (always)	BGP routes, VXLAN/IPIP fallback	eBPF datapath, tunnel or direct-routing mode
Encapsulation overhead	Always (50 bytes)	Only on fallback	Only in tunnel mode
Network policy	None	Full NetworkPolicy + CRDs	Full NetworkPolicy + identity-based L3-L7
Enforcement mechanism	N/A	iptables (or eBPF, newer versions)	eBPF only
Operational complexity	Lowest	Moderate (BGP peering to reason about)	Moderate-high (eBPF datapath to reason about)

Flannel’s simplicity is a real feature, not a limitation to apologize for. A cluster with no network policy requirement and no particular performance sensitivity gets a working pod network with almost nothing to misconfigure. Calico’s BGP mode removes encapsulation overhead entirely when the underlying network supports it, at the cost of a routing protocol to actually understand when something goes wrong. A BGP session down between two nodes is a different failure mode than a broken VXLAN tunnel, and needs different tools to diagnose (calicoctl node status and BGP session state, not tcpdump on port 4789). Cilium’s eBPF datapath is the most capable of the three and the one most worth understanding at the mechanism level rather than trusting as a black box, which is exactly what the next two posts do.

The next post goes deeper into Cilium’s eBPF datapath specifically, not repeating the standalone post’s ground, but focused on kube-proxy replacement and Hubble in the context of this series’ running comparison against Flannel and Calico’s mechanisms above.