IPsec Deep Dive Part 3: NAT vs IPsec — NAT-T, Port Forwarding, and the Fortinet SD-WAN Reality

Part 1 built the control channel; Part 2 built the data plane. Both rested on an assumption so basic it never got stated: that the IP address each peer sees is the address the other peer actually has. On a real SD-WAN estate that assumption is false more often than it’s true — the spoke behind a carrier CPE doing NAT, the branch on CGNAT’d 5G backup, the FortiGate-VM behind a cloud provider’s 1:1 NAT, the home-office FortiGate behind a domestic router. This post is about exactly what breaks when NAT touches IPsec, how the protocol detects and repairs it, what port forwarding is still for, and how all of it surfaces in a Fortinet SD-WAN overlay — including where it complicates ADVPN.

A precise statement of the problem

“NAT breaks IPsec” is true but lazy. There are three distinct breaks, with different mechanisms, different severities, and different fixes — and conflating them is why so much NAT-T advice is cargo cult.

Break 1: AH is cryptographically incompatible with NAT — unfixable

Part 1 flagged it; here’s the payoff. AH’s integrity check covers the immutable fields of the outer IP header, source and destination addresses included. A NAT exists to rewrite exactly those fields. Every AH packet crossing any NAT — even a polite 1:1 NAT that touches nothing else — arrives with an ICV that no longer matches, and is dropped as tampered. Which is, from AH’s perspective, a correct diagnosis: the packet was tampered with. NAT is a man-in-the-middle that we’ve collectively decided to tolerate, and AH was built to detect men-in-the-middle.

There is no fix, no workaround, no encapsulation trick. AH and NAT are mutually exclusive by design, and this — far more than the lack of encryption — is what removed AH from practical relevance. ESP’s “flaw” of not covering the outer header turned out to be the survival trait.

Break 2: ESP has no ports — NAPT has nothing to hold on to

The NAT everyone actually runs is NAPT — port address translation, many inside hosts multiplexed onto one public address using L4 source ports as the demultiplexing key. Walk an ESP packet through one and watch it fail on the device’s own terms:

• Outbound: the NAPT wants to rewrite source IP and source port, recording the mapping. ESP is IP protocol 50 — there is no L4 port. There’s an SPI in roughly the right place, but it’s not a port and doesn’t behave like one.
• Inbound: a reply ESP packet arrives at the public address. Which inside host gets it? The lookup key the NAPT was built around doesn’t exist. Worse — and this is the subtle one — the inbound packet’s SPI is from the other SA of the pair (Part 1: SAs are unidirectional, each direction has its own SPI), so even an SPI-aware device can’t trivially correlate it with the outbound mapping it recorded.

Some home routers attempt exactly that correlation under the name “IPsec passthrough” — track the outbound SPI, guess that the next inbound protocol-50 packet from that peer belongs to the same session. It works for exactly one inside IPsec client at a time and fails confusingly the moment a second one starts; two laptops, two tunnels, one public IP is the canonical breakage. Passthrough is a heuristic, not a solution, and turning it off once real NAT-T is in play removes a source of interference.

• Drop case: plenty of NAPT implementations don’t even try — no ports, no table entry, silently discard. Protocol 50 just dies at the CPE, and from the FortiGate’s perspective Phase 1 completes flawlessly and then no data flows.

Break 3: IKE’s own UDP gets its source port rewritten

IKE at least runs over UDP and can cross a NAPT. But it conventionally uses source port 500 = destination port 500, and a NAPT will rewrite that source port to something ephemeral. Pre-NAT-T responders commonly insisted on the 500↔500 convention, or worse, tried to reply to port 500 rather than to the rewritten source port, sending answers to a port the NAT had no mapping for. And with IKEv1 main mode + PSK, Part 1’s identity problem compounds it: the responder selecting a PSK by source address now sees the NAT’s address — shared, in a CGNAT scenario, by thousands of customers.

One more casualty worth naming, because it explains a NAT-T design choice below: transport mode. The inner TCP/UDP checksum is computed over a pseudo-header that includes the original IP addresses. NAT rewrites the outer addresses; in transport mode those are the addresses the checksum was computed against; the checksum is now wrong, and it’s encrypted, so the NAT’s usual checksum fix-up can’t reach it. Tunnel mode dodges this entirely — the inner packet, addresses, checksum and all, rides untouched inside the encrypted payload. One more reason site-to-site is tunnel mode.

The fix, step 1: detect the NAT (NAT-D)

The protocol machinery (RFC 3947 for IKEv1, native in IKEv2) starts from a neat insight: each peer knows what addresses and ports it’s actually using, and can tell the other — in a form a NAT can’t quietly edit. During the first IKE exchange (IKE_SA_INIT in IKEv2), each side sends NAT discovery payloads: hashes of (SPIs, IP address, port) for both its own end and the addresses it believes its peer has.

The receiver recomputes those hashes from the packet it actually received. If the hash of the claimed source address/port doesn’t match the observed source address/port, something rewrote them in flight — a NAT, and the mismatch even reveals which side is behind it. Because the values travel as hashes rather than literal addresses, a NAT can’t helpfully “fix” them the way ALGs mangle FTP and SIP; the comparison stays honest.

Both sides now agree on whether a NAT is present and on whose side. If none: proceed classically — UDP 500, raw protocol-50 ESP, done. Otherwise:

The fix, step 2: float to UDP 4500 and encapsulate (NAT-T)

When NAT is detected, both peers float to UDP port 4500, and two things change shape per RFC 3948:

IKE continues on UDP 4500 — with a four-zero-byte non-ESP marker prefixed to every IKE message. Why: port 4500 is about to carry two different protocols on one socket, and the receiver needs an instant demultiplexer. The first four bytes after the UDP header are either zero (cannot be a valid SPI — SPI 0 is reserved) meaning IKE, or nonzero meaning that’s an SPI, this is ESP. One four-byte sentinel, no second port, no heuristics.

ESP gets wrapped in UDP — outer IP (protocol 17 now, not 50), a plain 8-byte UDP header (4500→4500), then the ESP packet exactly as Part 2 dissected it, SPI onward, byte for byte. The NAPT is finally happy: it has UDP ports to rewrite, a clean conntrack entry per inside host, and inbound replies demultiplex on the rewritten port like any other UDP flow. Multiple FortiGates behind one public IP each get their own port mapping — the multi-client failure mode of “IPsec passthrough” evaporates.

Two operational consequences travel with the design:

Keepalives. A NAT’s UDP mapping is use-it-or-lose-it, with idle timeouts as low as 30 seconds on cheap CPE. A quiet tunnel would lose its mapping, and the responder’s next packet inbound would find no entry and die — the tunnel works until it mysteriously doesn’t, the diagnostic signature being “spoke can always re-initiate, hub can never reach a quiet spoke.” NAT-T’s answer is a one-byte (0xFF) keepalive trickled out during idle periods; FortiOS defaults to 10 seconds, tunable per phase1 with set keepalive.

Eight more bytes of overhead. The UDP header joins Part 2’s overhead bill — its MSS-clamp arithmetic already budgeted for this, which is why the recommendation was 1350 and not something tighter against the raw-ESP number. An estate where some spokes are NATed and some aren’t runs mixed encapsulation, and clamping for the worst case everywhere is cheaper than diagnosing the difference later.

On FortiOS the whole mechanism is three settings under phase1-interface:

config vpn ipsec phase1-interface
    edit "overlay-hub"
        set nattraversal enable     # enable | disable | forced
        set keepalive 10            # NAT-T keepalive interval, seconds
        ...
    next
end

enable trusts NAT-D: encapsulate only if NAT was detected. forced encapsulates unconditionally — the tool for the genuinely maddening case where NAT-D sees no address rewriting but something in the path silently eats protocol 50 (overzealous “VPN passthrough” CPE, certain mobile carriers, the odd cloud security group). If Phase 1 completes but no ESP ever arrives, before a long pcap session, try forced. disable is for the one scenario NAT-T harms: some legacy or hardened peers refuse encapsulated ESP, and if there’s provably no NAT in path, raw protocol 50 it is.

Where port forwarding still earns its keep

NAT-T solves the initiator-behind-NAT problem — the outbound IKE packet builds the NAT mapping, and everything flows back through it. It does nothing for the opposite case: the responder behind NAT. An initiator out on the internet sends IKE_SA_INIT to the responder’s public address; the NAT receives it, finds no mapping (nothing outbound ever created one), and drops it. No amount of NAT-T helps, because NAT-T never gets to run.

The repair is static: tell the NAT, ahead of time, where unsolicited IPsec traffic goes —

Forward	When
UDP 500 → inside FortiGate	Always — the first IKE packet arrives here before any float
UDP 4500 → inside FortiGate	Always in practice — everything after the float
IP protocol 50 → inside FortiGate	Only if a peer will negotiate raw ESP (no NAT-T support, or nattraversal disable) — note this is a protocol forward, which consumer routers often can’t express; with NAT-T available, it’s unnecessary

Three sharp edges. First, with PSK + IKEv1 main mode the responder-behind-NAT also trips Part 1’s identity problem — the initiator addresses the NAT’s public IP while the responder identifies by its private one; the fixes are set localid on the responder or, properly, IKEv2 where identity never depended on addresses. Second, both ends behind NAT works fine — but only if at least one side has this forwarding (or a full 1:1 NAT) so that someone can be the responder. Two NATed peers with no forwarding on either side have no rendezvous point — the limitation hole-punching systems like WireGuard-with-a-coordinator or Tailscale exist to solve, and which static site-to-site IPsec simply doesn’t: one end must be reachable. Third, on a FortiGate that is the NAT (a hub fronting an inside concentrator), this is a VIP — config firewall vip forwarding UDP 500/4500 — plus a policy allowing it, the same first-match logic as everything else in the policy pipeline.

The Fortinet SD-WAN reality

Time to land this on the estate every other post assumes. A production Fortinet SD-WAN overlay is, concretely: a hub (or several) with stable public addressing, and tens-to-thousands of dialup IKEv2 spokes initiating inward — from fibre with real addresses, from CPE NAT, from CGNAT’d LTE/5G backup circuits, from anything procurement signed off. NAT shapes that design more than it first appears:

The dialup direction is the NAT strategy. Spokes initiate, hubs respond — which means every NAT mapping is created outbound by the spoke, NAT-T floats whoever needs floating, and no branch site ever requires port forwarding, coordination with the circuit provider, or even a static IP. That’s not a happy accident; it’s why the dialup-hub pattern won. The only place reachability must be engineered is the hub — public UDP 500/4500, or the VIP arrangement above if the hub itself sits behind a 1:1 NAT, standard for FortiGate-VM hubs in AWS/Azure where the elastic/public IP is always a 1:1 NAT to the instance’s private address (cloud on-ramp territory: tunnels to TGW/vWAN run NAT-T as a matter of course).

Identity must not be address-based, because addresses are lies. Behind CGNAT, hundreds of unrelated customers share the spoke’s apparent source address, and it changes on reconnect. Part 1’s hub config already took the position this requires: peertype one with IKE IDs (or, better, certificates) — the spoke is its credential, wherever it appears from. Address-keyed PSK main mode isn’t just deprecated here, it’s structurally impossible.

ADVPN is where NAT bites back. Hub-and-spoke data planes want to become meshes on demand — ADVPN builds spoke-to-spoke shortcuts when traffic justifies them, the hub relaying shortcut offers over the IKE channel (auto-discovery-sender / auto-discovery-receiver from Part 1’s config). Now re-run this post’s logic on a spoke-to-spoke tunnel: the would-be initiator spoke learns its peer’s address — as observed by the hub, i.e. the peer’s NAT public address — and sends IKE at it. If the target spoke’s NAT has no reason to deliver unsolicited UDP 4500 (and it usually doesn’t — its mappings point at the hub), the shortcut times out. Both spokes NATed: worse. FortiOS mitigates honestly: shortcut attempts carry the NAT-T machinery, monitoring detects failed shortcuts, and traffic falls back to the hub path — degraded to the baseline, never black-holed. Practical posture: spokes on real addresses or 1:1 NAT mesh freely; spokes behind NAPT/CGNAT mostly hairpin via the hub, so place hubs where hairpinning is cheap (the hub-placement series’ argument, arriving from a new direction). FortiOS 7.4’s ADVPN 2.0 reworked shortcut signalling partly to negotiate these cases better, but physics hasn’t changed: someone reachable must exist.

Reading the state off a live box. diagnose vpn ike gateway list name <tunnel> shows NAT-T in the established gateway: the peer on port 4500, and a nat: peer / nat: this-side annotation straight out of NAT-D’s verdict. diagnose vpn tunnel list shows natt: mode=... per tunnel. On the wire, diagnose sniffer packet wan1 'port 4500' 4 shows the keepalive heartbeat (66-byte frames of 0xFF) between data bursts — and one of the great minor satisfactions in this job is watching a NAT mapping get kept alive in real time, eight bytes of UDP at a stretch, knowing every layer of machinery from Part 1’s DH exchange to this keepalive is doing its quiet work so that BGP, SLA probes, and user traffic can pretend the underlay is simple.

The series, closed out

Part	Topic
1	The architecture: SAs, ESP vs AH, tunnel vs transport, IKEv1 main/aggressive, IKEv2
2	Phase 2: child SAs, selectors, PFS, rekeying, anti-replay, the ESP packet field by field
3 (this post)	NAT vs IPsec: the three breaks, NAT-D, NAT-T on UDP 4500, port forwarding, ADVPN behind NAT

The arc in one paragraph: IKE Phase 1 spends nine (or four) messages building an authenticated control channel; Phase 2 mints unidirectional SA pairs whose ESP packets give every byte a job; and because the world is full of devices that rewrite addresses, the whole edifice carries detection hashes and a UDP escape hatch so it can survive paths it was never originally designed for. Every overlay tunnel in every SD-WAN post on this blog — every BGP session in the resilience series, every probe packet in an SLA rule, every byte to a cloud on-ramp — rides on exactly this machinery. Now it’s written down.