Container Networking Deep Dive Part 4: Multi-Host Overlay Networking and the VXLAN Underneath It

Every network built in this series so far has lived on one host: one bridge, one kernel, veth pairs that never leave the machine. Real deployments span hosts, and containers on different hosts still need to reach each other by a stable address without every service having to know the physical topology underneath it. Docker Swarm’s overlay driver, and the CNI plugins covered later in this series, solve this the same way: encapsulate the container’s traffic inside a tunnel between hosts, so the containers behave as if they share a bridge even though they don’t share a kernel.

Standing up an overlay network

Overlay networks require Swarm mode:

# on the first host
docker swarm init --advertise-addr 10.0.0.11

# on the second host, using the join token printed above
docker swarm join --token <token> 10.0.0.11:2377

Create the overlay network from the manager:

docker network create -d overlay --attachable demo-overlay

Run one container on each host, attached to the overlay:

# host A
docker run -d --name svc-a --network demo-overlay alpine sleep infinity

# host B
docker run -d --name svc-b --network demo-overlay alpine sleep infinity

docker exec svc-a ping -c 2 svc-b

This works, by name, across two hosts that otherwise have no route to each other’s container IP ranges. Nothing about the physical network between the two hosts was reconfigured to make this possible. No VLANs added, no routes added on the intervening switches. The overlay network did all of it by wrapping every packet.

What VXLAN actually does to a packet

VXLAN (Virtual Extensible LAN) is the encapsulation Docker’s overlay driver uses. The mechanism: take the entire original Ethernet frame from the container, source MAC, destination MAC, the container’s private IP, all of it, and wrap it inside a new UDP packet addressed between the two physical hosts. The intervening network only ever sees host-to-host UDP on port 4789; it has no idea an entire second Ethernet frame is riding inside the payload.

Original frame (container A to container B):
[Eth: container MACs][IP: 10.0.9.x -> 10.0.9.y][TCP][payload]

After VXLAN encapsulation (host A to host B):
[Eth: host A to host B MACs][IP: host A to host B][UDP dport 4789][VXLAN header][original frame above]

Capture the traffic on the physical interface of either host to see this directly:

sudo tcpdump -i eth0 -n udp port 4789 -vv

Every packet shown is UDP between the two hosts’ real IPs. The container-to-container conversation only becomes visible again once tcpdump decodes the VXLAN payload. The -T vxlan and packet-detail flags reveal the inner Ethernet frame, MACs and all, inside what the wire otherwise treats as an ordinary UDP datagram.

The 50-byte tax

This is the part that causes real outages, not a theoretical footnote. The VXLAN header, plus the UDP header, plus the outer IP header, plus the outer Ethernet header, add up to 50 bytes of overhead added to every original frame before it goes on the physical wire:

Outer Ethernet header:    14 bytes
Outer IP header:          20 bytes
Outer UDP header:          8 bytes
VXLAN header:               8 bytes
                          --------
Total overhead:            50 bytes

A standard 1500-byte Ethernet MTU on the physical network leaves only 1450 bytes for the entire original frame once VXLAN wraps it. If the container’s own interface still advertises a 1500-byte MTU, which it will, unless something explicitly lowers it, any original packet close to that size gets fragmented at the encapsulating host, or dropped outright if the DF (don’t fragment) bit is set and nothing along the path returns an ICMP Fragmentation Needed message that actually makes it back.

Part 4 of the Pivoting and Tunneling series covers this exact class of problem from the other side: an SSH or Chisel tunnel silently eating MTU budget the same way VXLAN does here. The mechanism is identical. Something in the path adds bytes to every packet, nothing tells the endpoints to lower their MTU accordingly, and the failure mode is asymmetric: small packets work fine, large ones vanish or fragment, and the symptom looks like “the connection hangs on big transfers” rather than an obvious outright failure.

Docker’s overlay driver actually accounts for this by default. Container interfaces on an overlay network get an MTU of 1450, not 1500, specifically to leave room for the VXLAN header without relying on fragmentation:

docker exec svc-a ip link show eth0 | grep mtu

5: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 ...

The failure case worth knowing about is when this number is wrong for the actual path. A physical network with a lower MTU than 1500 somewhere in the middle (a WAN link, an additional tunnel already in place, a cloud provider’s own overlay underneath yours) leaves even less room than 1450 assumes, and Docker has no way to know about a constraint it didn’t create.

Diagnosing it

The standard PMTUD-style binary search from the tunneling series applies here directly. Send progressively larger pings with the DF bit set from inside the container, and find where they stop returning:

docker exec svc-a ping -M do -s 1400 -c 2 svc-b
docker exec svc-a ping -M do -s 1420 -c 2 svc-b
docker exec svc-a ping -M do -s 1421 -c 2 svc-b

-M do sets the DF bit; the first size that fails without an ICMP response (rather than fragmenting cleanly) marks the boundary. This is precisely the technique the standalone PMTUD sweeper tool automates for a routed path. Here it’s applied to a tunnel wrapped around two Docker hosts instead of a chain of routers, but the underlying question, what’s the largest DF-set frame that survives this specific path, is identical.

Why this matters before it becomes a Kubernetes problem

Every CNI plugin covered later in this series, Flannel specifically, and Calico in its default configuration, builds its cross-node pod network the same way: VXLAN (or a close variant), encapsulating pod traffic between nodes, subject to the exact same 50-byte overhead and the exact same MTU miscalculation risk if the underlying network’s own MTU isn’t accounted for. A misconfigured MTU on a Kubernetes cluster’s CNI shows up as “large requests time out, small ones work fine, and it looks intermittent,” the same signature this section just walked through, one layer up the stack.

The next post moves from Docker’s own overlay networking to Kubernetes’ CNI interface, a different problem than the one overlay networking solves, but one that exists for a reason directly tied to how Docker’s own model falls short at Kubernetes’ scale.