IPsec Deep Dive Part 1: ESP, AH, and How IKE Phase 1 Actually Brings a Tunnel Up

Every Fortinet SD-WAN design on this blog stands on the same foundation and has so far gotten away with waving at it. The resilience series runs BGP over IPsec overlays. The cloud on-ramp posts terminate IPsec into AWS and Azure. The PKI series spent three parts replacing IPsec’s pre-shared keys with certificates without ever explaining what those keys were for at the protocol level. Even NSE4 Part 8 covers IPsec the way an exam guide has to — what to configure, not what’s on the wire.

This series is the missing foundation, and the rule is no stone unturned. Part 1 covers the architecture — security associations, ESP versus AH, tunnel versus transport — and then walks IKE Phase 1 message by message: IKEv1 main mode, why aggressive mode exists and why it leaks, and how IKEv2 collapses the whole dance into four packets. Part 2 does the same for Phase 2 and dissects an ESP packet field by field. Part 3 takes on the topic that generates more VPN tickets than everything else combined: what NAT does to IPsec, and how NAT-T and port forwarding put it back together — ending where this blog always ends up, on a FortiGate SD-WAN overlay.

IPsec is not a protocol

The first mental adjustment: IPsec (RFC 4301) is a framework — an architecture with three moving parts that are routinely confused for each other:

Component	IP protocol / port	Job
ESP — Encapsulating Security Payload	IP protocol 50	Carries the protected data: encryption + integrity
AH — Authentication Header	IP protocol 51	Integrity only, no encryption — and covers parts of the IP header itself
IKE — Internet Key Exchange	UDP 500 (and 4500, Part 3’s territory)	The control plane: negotiates parameters, performs key exchange, authenticates the peers

Notice what that table implies and most quick explanations skip: a working IPsec tunnel involves two completely different kinds of traffic. IKE is a UDP application-layer protocol doing negotiation. ESP is its own IP protocol — not TCP, not UDP, protocol number 50 sitting right alongside them — doing bulk transport. The distinction sounds pedantic until Part 3, where it becomes the entire reason NAT and IPsec don’t get along: NAT devices were built for protocols 6 and 17, and protocol 50 has no port numbers to translate.

The split is also a clean control-plane/data-plane separation, the same shape as everything in the SD-WAN world: IKE is the routing protocol of the arrangement, ESP is the forwarding plane. On a FortiGate the split is physical — IKE runs in the iked process on the CPU, while established ESP flows get pushed down to the NP7 for hardware forwarding, exactly as covered in packet flow Part 1.

Security associations: the data structure everything hangs off

Before any messages make sense, one concept has to be solid: the security association (SA). An SA is a simplex (one-direction) contract: “traffic matching this selector, protected with this cipher and these keys, identified on the wire by this SPI.” Three consequences follow directly:

SAs come in pairs. Because an SA is unidirectional, a working tunnel needs at least two — one inbound, one outbound — each with its own keys and its own 32-bit SPI (Security Parameters Index). The SPI travels in the clear at the front of every ESP packet; it’s the receiver’s lookup key into its SAD (Security Association Database) to find which keys decrypt this packet. When diagnose vpn tunnel list on a FortiGate shows different SPIs for dec and enc, that’s the pair.

The SPD decides, the SAD executes. The Security Policy Database answers “should this packet be protected, bypassed, or dropped?” — on a route-based FortiGate VPN that decision is effectively made by routing plus firewall policy, which is why route-based tunnels feel so natural in an SD-WAN context. The SAD answers “with which SA?” Keep the two databases separate in your head and half of IPsec’s apparent complexity evaporates.

Keys have lifetimes, so SAs are born and die constantly. Everything about rekeying — soft limits, hard limits, what happens to in-flight traffic — lands in Part 2. For now: an SA is not “the tunnel.” The tunnel is a long-lived intent; SAs are the short-lived instantiations of it.

ESP vs AH, and tunnel vs transport

Two independent choices define what actually goes on the wire, giving four combinations — of which modern practice uses essentially one.

ESP or AH. ESP encrypts the payload and authenticates everything from the ESP header to the end of the trailer — but deliberately not the outer IP header. AH encrypts nothing and instead authenticates the payload plus the immutable fields of the IP header itself, source and destination addresses included. That design decision made sense in a 1990s world worried about source-address spoofing; in a world full of NAT it’s fatal, because any device that rewrites an address corrupts every AH packet irreparably. Hold that thought for Part 3 — it’s the first of NAT’s three kills. ESP with an authentication algorithm does everything AH does except cover the outer header, and not covering the outer header turns out to be a feature. AH survives mostly in exam questions; FortiOS doesn’t even expose it for the VPNs that matter here.

Tunnel or transport. Transport mode protects the L4 segment and reuses the original IP header — fine for host-to-host traffic where the endpoints are also the crypto endpoints (its one mainstream habitat is GRE-over-IPsec on Cisco kit, where GRE already provides the tunnel). Tunnel mode wraps the entire original IP packet — addresses and all — inside a brand-new IP packet between the two gateways. Site-to-site VPNs, and therefore every SD-WAN overlay, are tunnel mode: the original source and destination are RFC 1918 addresses on LANs that the internet has never heard of, so they have to ride inside something publicly routable. A FortiGate phase1-interface VPN is tunnel-mode ESP, full stop.

So the working combination is ESP, tunnel mode, and the diagram above is worth a long look, because three later plot points are sitting in it: the outer IP header is outside the authenticated span (NAT-T’s opening), the entire inner packet — inner addresses included — is encrypted (why a NAT can’t even see what it would need to fix up), and everything from SPI onward is opaque except the SPI and sequence number themselves (why the receiver can demultiplex without decrypting).

IKE Phase 1: building the control channel

Nothing about ESP explains where its keys come from. That’s IKE’s job, and it happens in two phases that people merge in their heads at their peril:

• Phase 1 authenticates the two peers to each other and builds one bidirectional, encrypted control channel — the IKE SA (IKEv1 calls it the ISAKMP SA).
• Phase 2 runs inside that protected channel to negotiate the actual IPsec SAs that carry data — as many pairs as needed, rekeyed on their own schedule.

The two-phase design is an amortization argument: the expensive parts (Diffie-Hellman, peer authentication) happen once in Phase 1; the cheap part (negotiating data-plane SAs) can then happen repeatedly under that umbrella without redoing the expensive parts each time.

IKEv1 main mode, message by message

Six messages, three pairs, each pair doing one job:

Messages 1–2 — agree on how to talk. The initiator sends one or more proposals: encryption algorithm, hash, DH group, authentication method, lifetime. The responder picks one or replies with nothing useful — and this is the origin of the single most common Phase 1 failure, the proposal mismatch. The FortiOS log line negotiation failure with no SA proposal chosen is this exact exchange falling at the first hurdle. Everything so far is cleartext, which is fine: nothing secret has been said.

Messages 3–4 — Diffie-Hellman. Each side sends its DH public value (KEi/KEr) and a nonce. After this pair, both sides — and only these two sides — share a secret that no observer of the exchange can compute. From it, IKEv1 derives SKEYID and three working keys: a derivation key, an authentication key, and the encryption key for the rest of the IKE conversation. The DH group chosen back in message 1 governs how hard this secret is to attack: group 14 (2048-bit MODP) is the floor today, and the elliptic-curve groups 19–21 buy more security per CPU cycle — relevant at SD-WAN hub scale, where a hub re-establishing 500 dialup spokes after a reboot does 500 DH computations before any spoke passes traffic.

A subtlety worth pinning down: DH provides secrecy, not identity. After message 4 the channel is encrypted against outsiders, but either end could be anyone — a man-in-the-middle could have run two separate DH exchanges, one with each victim. That’s what the last pair fixes.

Messages 5–6 — prove who you are, under cover. Now that encryption keys exist, each side sends its identity and an authenticator, encrypted: with PSK authentication, a hash keyed with material derived from the pre-shared key; with certificates, a signature plus the cert itself (the upgrade path the PKI series is about). Main mode’s quietly elegant property lives here: identities are exchanged only after the channel is encrypted, so a passive observer never learns who is tunnelling to whom.

That property has a sharp PSK-shaped corner, though. In messages 5–6 the responder must look up the right PSK before it can verify the hash — and with main mode the only identity information available at lookup time is the initiator’s source IP address. Static-IP site-to-site: fine. A dialup spoke arriving from an address the hub has never seen: impossible — the hub can’t pick a per-peer PSK for an address it doesn’t recognize. The classical answers were a single group PSK for all dialup peers (one compromised spoke can impersonate the hub to every other spoke — genuinely bad), or:

Aggressive mode — the shortcut that leaks

Aggressive mode crunches Phase 1 into three messages by front-loading everything: message 1 carries the SA proposal, the DH public value, the nonce, and the initiator’s identity — necessarily in cleartext, since no keys exist yet. Now the responder can select a per-peer PSK by identity rather than by source IP, which is the entire reason aggressive mode shipped: it made dialup PSK workable.

The price is two leaks. First, identities travel in the clear — an observer learns who is connecting to whom. Second and worse, the responder’s message 2 includes its authentication hash, computed over values an eavesdropper has now mostly seen. With PSK auth, that hash is offline-crackable: capture one aggressive-mode exchange, take it home, and grind dictionary candidates against it at leisure — no further contact with either peer needed. Every PSK-strength lecture ever delivered traces back to this exposure. Certificates don’t leak this way even in aggressive mode, but the identity exposure remains.

The real fix isn’t picking a better IKEv1 mode. It’s the protocol revision that made the whole dilemma obsolete.

IKEv2 — the same job, redesigned with the lessons learned

IKEv2 (RFC 7296) is not “version 2 of the same wire protocol” — it’s a redesign that keeps the architecture (two phases, IKE SA then data-plane SAs) and rebuilds the mechanics:

Four messages, not nine. IKE_SA_INIT request and response carry proposals, DH values, and nonces — the work of main mode 1–4 in one round trip. IKE_AUTH request and response, already encrypted, carry identities, authentication, and the negotiation of the first data-plane SA — so one round trip after the keys exist, the tunnel passes traffic. Main mode plus quick mode needed nine messages for the same outcome.

Identity protection without the dialup dilemma. Identities ride in the encrypted IKE_AUTH exchange (main mode’s virtue), but because authentication and SA negotiation were restructured, the responder no longer needs the PSK before learning the identity — dialup with per-peer credentials just works (aggressive mode’s virtue, without the leak). EAP support comes along for free, which is what makes IKEv2 the basis of modern remote-access VPNs.

Reliability is in the protocol, not bolted on. Every IKEv2 message is a request that expects a response, with message IDs, retransmission, and a clear definition of when a peer is dead. IKEv1 ran over bare UDP with vendor-specific retransmission heuristics and DPD as an add-on (RFC 3706); entire categories of interop bugs lived in that gap.

NAT awareness is native. IKE_SA_INIT carries NAT_DETECTION_SOURCE_IP and NAT_DETECTION_DESTINATION_IP payloads as a first-class part of the exchange — the full mechanics are Part 3’s opening act.

For Fortinet SD-WAN specifically, the calculus is one-sided: dialup spokes authenticate by IKE ID or certificate rather than source address, hubs converge faster after failover because each spoke needs four packets instead of nine, ADVPN’s shortcut signalling rides the IKE channel, and there is no aggressive-mode foot-gun left to step on. New overlay builds on this blog are IKEv2 unless a peer forces otherwise; the cloud on-ramp tunnels to AWS TGW and Azure vWAN support it on the provider side too.

What Phase 1 looks like on a FortiGate

Tying the wire protocol to the config that drives it — a hub-side IKEv2 dialup Phase 1 of the kind every overlay post on this blog assumes:

config vpn ipsec phase1-interface
    edit "overlay-hub"
        set type dynamic                  # dialup: accept spokes from any address
        set interface "wan1"
        set ike-version 2
        set peertype one                  # authenticate spokes by IKE ID
        set peerid "spoke-estate"
        set proposal aes256gcm-prfsha384  # AEAD — Part 2 explains why GCM
        set dhgrp 19                      # ECC group: cheaper per spoke at scale
        set add-route disable             # routing comes from BGP, not IKE
        set auto-discovery-sender enable  # ADVPN — Part 3 territory
        set psksecret <long, random, per-estate>
    next
end

Every line maps to something above: ike-version 2 selects the four-message exchange, peertype one/peerid is the dialup-identity capability IKEv1 main mode couldn’t offer PSKs, proposal and dhgrp are the message-1 negotiation content, and type dynamic is the hub declaring it will be a responder for unknown initiators. And when it doesn’t come up:

diagnose vpn ike log-filter dst-addr4 <peer-ip>     # FortiOS 7.4+: diagnose vpn ike log filter
diagnose debug app ike -1
diagnose debug enable

Reading the output is just reading this post backwards: find which message of the exchange appears last, and the failure names itself. Dies at message 1–2 (or IKE_SA_INIT)? Proposal or DH-group mismatch — compare proposal/dhgrp line by line. Completes the DH pair then fails? Authentication: PSK mismatch, wrong peerid, or certificate trouble. Never see a reply at all? Nothing about IKE yet — UDP 500 isn’t arriving, which is a routing, firewalling, or (very often) NAT problem. The third case is the whole of Part 3.

Where this series goes

Part	Topic
1 (this post)	The architecture: SAs, ESP vs AH, tunnel vs transport, IKEv1 main/aggressive, IKEv2
2	Phase 2: quick mode and child SAs, selectors, PFS, rekeying, anti-replay — and an ESP packet dissected field by field
3	NAT vs IPsec: the three ways NAT breaks it, NAT-D, NAT-T on UDP 4500, port forwarding, and what it all means for a Fortinet SD-WAN overlay

Phase 1’s deliverable is modest for all the ceremony involved: one encrypted, authenticated control channel and not a single byte of user data protected. Part 2 picks up at the moment that channel goes live, and follows it into the negotiation that actually moves packets.