Container Networking Deep Dive Part 10: A Full Walkthrough, Chasing a Cross-Node Connectivity Bug

This closes the series with one bug, worked start to finish, using the techniques from Parts 1 through 9 rather than introducing anything new. The scenario: a checkout service, running on a Cilium-managed cluster, gets scaled from 3 replicas to 12 ahead of an expected traffic spike. Within minutes, a subset of checkout pods start failing calls to payment-gateway, an internal service on a different set of nodes. Not all of them, roughly a third.

First: isolate what’s actually failing

The report is “checkout can’t reach payment-gateway.” That’s a symptom, not a diagnosis. First move, from Part 9’s DNS-versus-network split:

kubectl exec checkout-8f2 -- getent hosts payment-gateway.production.svc.cluster.local

;; connection timed out; no servers could be reached

DNS resolution itself is failing, from this specific pod. Not every checkout pod, though. Running the same command against checkout-3a1 (one of the original 3 replicas, not one of the newly scaled ones) succeeds immediately. The split is old pods versus new pods, not checkout versus payment-gateway.

Problem one: a NetworkPolicy gap that only bites newly-labeled pods

checkout runs behind a NetworkPolicy that denies all egress by default and explicitly allows specific destinations, the same default-deny pattern Part 8 covered. Checking the policy:

kubectl get networkpolicy checkout-egress -n production -o yaml

spec:
  podSelector:
    matchLabels:
      app: checkout
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: payment-gateway
      ports:
        - protocol: TCP
          port: 8443

No rule permitting egress to CoreDNS at all. The original 3 replicas have been running long enough that this gap should, in theory, affect them too, and checking their identity assignment against Part 8’s mechanism explains why it doesn’t:

cilium identity list | grep checkout

3301   k8s:app=checkout    k8s:io.kubernetes.pod.namespace=production
3301   k8s:app=checkout,k8s:version=canary

Two different identities. The original 3 replicas were deployed before a version: canary label was added to the checkout Deployment’s pod template months ago; the new 12 replicas from today’s scale-out picked up that label and, with it, a different security identity than the old pods still carry. A previously-applied CoreDNS-allow rule, added after the original policy above and never audited against the newer label set, only matches app: checkout without the version label. Part 8’s identity-based enforcement is doing exactly what it’s configured to do; the configuration itself has a gap that only a subset of pods fall into.

kubectl get networkpolicy checkout-dns-allow -n production -o yaml | grep -A3 podSelector

  podSelector:
    matchLabels:
      app: checkout

Confirmed with Hubble, matching Part 8’s drop-diagnosis pattern exactly:

hubble observe --verdict DROPPED --from-pod production/checkout-8f2

Jul 23 16:04:02: production/checkout-8f2 (ID:3301, app=checkout,version=canary) -> kube-system/coredns-7d9 (ID:1105) Policy denied DROPPED (UDP)

Fix: broaden the DNS-allow policy’s selector to match on app: checkout alone, the way it was presumably intended, or add an explicit second rule for the canary-labeled identity. Either resolves the DNS failure. It does not, on its own, resolve the original report.

Problem two: DNS was never the whole story

With the policy fixed, getent hosts now succeeds from every checkout pod. The original symptom, failed calls to payment-gateway, is mostly gone, but not entirely. A small number of calls still fail, and this time only for a specific request type: large order payloads with many line items, not simple ones.

This is Part 4 and Part 9’s MTU signature exactly: size-dependent failure, not connection-dependent. payment-gateway’s nodes are on a different rack than checkout’s newest nodes, added in today’s scale-out, and Cilium on this cluster runs in VXLAN tunnel mode between those two node groups specifically, because a firewall between the racks blocks the direct-routing mode’s requirements.

kubectl exec checkout-8f2 -- ping -M do -s 1400 -c 2 <payment-gateway-pod-ip>
kubectl exec checkout-8f2 -- ping -M do -s 1421 -c 2 <payment-gateway-pod-ip>

1400 bytes: received
1421 bytes: Frag needed and DF set, but no route to host

The boundary lands almost exactly where Part 4’s 50-byte VXLAN overhead math predicts. The new nodes added in today’s scale-out were provisioned from an updated base image with a host NIC MTU of 1500 but no corresponding adjustment to account for the tunnel overhead on cross-rack traffic. The older node pool, provisioned months earlier, had this already accounted for by a since-forgotten manual ip link set mtu 1450 applied by hand during initial cluster build-out, and that manual step was never captured in the newer nodes’ provisioning process.

ip link show cilium_vxlan

9: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 vxlan id 4 ...

1500, not 1450, the gap. Fix: correct the MTU on the tunnel interface (or, properly, fix the node provisioning process so it’s set correctly at boot rather than patched by hand again), and confirm:

sudo ip link set cilium_vxlan mtu 1450
kubectl exec checkout-8f2 -- ping -M do -s 1421 -c 2 <payment-gateway-pod-ip>

1421 bytes: received

Large-payload requests to payment-gateway succeed from the affected pods immediately after.

What made this a two-problem bug

Neither issue alone explains the full symptom set on its own, and that’s exactly why isolating “what’s actually failing, for which pods, under what conditions” mattered more than jumping straight to a fix for the first plausible cause found. The DNS gap explained the total failures from newly-scaled pods. The MTU gap explained the partial, payload-size-dependent failures that remained after the DNS gap was fixed, and that would otherwise have been invisible until the DNS issue stopped masking them entirely.

The tools involved across both problems: getent hosts and Hubble’s drop verdict for the policy gap, cilium identity list to explain why old and new pods behaved differently under the same policy, ping -M do for the MTU boundary, and ip link show to confirm the actual interface configuration rather than assuming it matched the older node pool. None of it required anything beyond what Parts 1 through 9 already covered: the namespaces and veth pairs underneath every pod, the NAT and policy mechanisms layered on top, and the encapsulation overhead that both CNI plugins and Docker’s own overlay driver carry by the same 50-byte tax.