Container Networking Deep Dive Part 9: Troubleshooting Container Networking

This series has covered bridge networking, Docker’s drivers, overlay VXLAN, CNI, and policy enforcement across Flannel, Calico, and Cilium as separate mechanisms. In practice, most container networking outages come from a small set of failure modes that show up regardless of which of those mechanisms a given environment runs. This post is the vendor-neutral checklist, the same kind of methodology the OSPF Deep Dive series built for routing, applied here to container networking instead.

1. conntrack exhaustion

Every NAT’d connection, every bridge-network container reaching out through MASQUERADE, every kube-proxy iptables DNAT, every Calico fallback-mode encapsulated flow, consumes an entry in the kernel’s connection tracking table. That table has a fixed maximum size, and once it fills, new connections silently fail to establish rather than producing an obvious error:

sysctl net.netfilter.nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max

net.netfilter.nf_conntrack_count = 262143
net.netfilter.nf_conntrack_max = 262144

One entry away from full. This is the single most common cause of “connections work fine most of the time, then start failing under load, then recover” on a host running a large number of short-lived container connections: a batch job opening thousands of brief outbound connections, or a Service under heavy traffic, exhausts the table faster than entries time out. The fix is either raising nf_conntrack_max (a sysctl change, requiring enough RAM to back the larger table since each entry costs real memory) or reducing the connection churn causing exhaustion (connection pooling, keep-alives, on the application side).

Confirm this is the actual cause before changing anything, rather than assuming it:

dmesg | grep -i "nf_conntrack: table full"

A hit here is unambiguous. The kernel logs the exact moment it started dropping new connections because the table was full.

2. VXLAN MTU fragmentation

Covered mechanically in Part 4, this failure mode is worth restating as a diagnostic pattern because of how it presents: small requests succeed, large ones hang or time out, and it looks intermittent because most traffic in a typical workload is small enough to survive. The tell is size-dependent failure, not connection-dependent failure.

# From inside a pod or container, with the DF bit set:
ping -M do -s 1400 -c 2 <destination>
ping -M do -s 1421 -c 2 <destination>

A binary search across sizes, same technique as the PMTUD sweeper automates for routed paths, finds the exact boundary. If the failure boundary lines up with 1500 minus whatever encapsulation overhead is in play (50 bytes for VXLAN, per Part 4’s breakdown), the diagnosis is confirmed without needing to inspect a single packet capture.

3. DNS-in-pod failures

A pod that can reach other pods by IP but not by name has a DNS problem specifically, not a networking problem generally, worth separating early rather than chasing a routing issue that doesn’t exist. Confirm the split directly:

kubectl exec test-pod -- getent hosts api-server.production.svc.cluster.local
kubectl exec test-pod -- curl -m 3 http://10.244.3.8:8080/

If the IP-based request succeeds and the name-based one doesn’t, the problem is CoreDNS (or whatever cluster DNS is running), not the CNI plugin. Common causes, in roughly the order worth checking them: the pod’s /etc/resolv.conf pointing at a nameserver that doesn’t match the actual cluster DNS Service IP (a symptom of a badly configured dnsPolicy, or a pod running with host networking and inheriting the host’s own resolv.conf instead of the cluster’s), CoreDNS pods themselves in a CrashLoopBackOff or otherwise unhealthy, and NetworkPolicy rules from Part 8’s mechanism that permit pod-to-pod traffic generally but don’t explicitly allow egress to the DNS Service’s port, a real and easy-to-miss gap in a “default deny” NetworkPolicy stance.

kubectl exec test-pod -- cat /etc/resolv.conf
kubectl get pods -n kube-system -l k8s-app=kube-dns

4. Cross-node connectivity breaks

When pod-to-pod traffic works on the same node but not across nodes, the fault is almost always in whatever cross-node mechanism the CNI plugin uses: VXLAN tunnel state (Flannel, or Calico/Cilium in tunnel mode), BGP session state (Calico’s default mode), or the eBPF datapath’s own view of which node owns which pod (Cilium). Isolate which side of the boundary the failure is on first:

# Same-node test (should always work; if this fails, the problem isn't cross-node)
kubectl exec pod-a -- ping -c 2 <pod-on-same-node-ip>

# Cross-node test
kubectl exec pod-a -- ping -c 2 <pod-on-different-node-ip>

If same-node traffic works and cross-node doesn’t, check the specific mechanism in play. For Flannel or VXLAN-mode Calico, confirm the tunnel interface is actually up and the node’s own overlay routes are populated:

ip -d link show flannel.1
ip route show table all | grep flannel

For Calico’s BGP mode, confirm the BGP session to the affected node specifically is established, not just that Calico is “running”:

calicoctl node status

A BGP session down to one specific node explains cross-node failures to that node only, and working connectivity to every other node, a distinguishing signature from a broken tunnel, which tends to be more uniformly broken across all cross-node pairs at once.

5. tcpdump inside a network namespace

Every technique above eventually needs a packet capture to confirm, and the namespace boundary that makes container networking isolated in the first place also means a capture on the host’s own interface won’t show a pod’s internal view of a conversation. You need to run tcpdump inside the namespace, exactly as covered mechanically in the namespaces post:

# Find the pod's network namespace
crictl inspectp <pod-id> | grep namespace

# Or, from the host, using the container's PID:
sudo nsenter -t <container-pid> -n tcpdump -i eth0 -n

Capturing on the host-side veth end shows the same traffic after it’s crossed into the bridge or CNI plugin’s forwarding path, useful for confirming whether a packet ever left the pod at all, versus whether it arrived correctly on the other side. Capturing on both ends of a suspected break, the source pod’s eth0 and the destination pod’s eth0, and comparing what’s actually seen on each side is the fastest way to localize which of the four failure modes above is actually in play, rather than guessing from symptoms alone.

Putting it together

None of these five checks requires trusting a CNI plugin’s own status output as the final word. nf_conntrack_count, ping -M do, resolv.conf, ip link show, and tcpdump inside a namespace are all tools that predate Kubernetes and work identically regardless of which plugin from Part 6 is running. That’s deliberate: a troubleshooting methodology built on kernel-level primitives generalizes across Flannel, Calico, and Cilium, whereas one built on trusting each plugin’s own dashboard doesn’t transfer at all when the next cluster runs a different one.

The closing post in this series ties all nine of the above into one real, multi-layer bug: a cross-node connectivity failure that turns out to be two separate problems stacked on top of each other, diagnosed start to finish using nothing but the techniques from this series.