IPsec Deep Dive Part 2: Phase 2, Child SAs, and the Anatomy of an ESP Packet

Part 1 ended with a deliberately deflating observation: after all of Phase 1’s ceremony — proposals, Diffie-Hellman, authenticated identities — exactly zero bytes of user data are protected. What exists is one encrypted control channel between two peers who now trust each other. This post is about what that channel is for: negotiating the IPsec SAs that carry actual traffic, keeping them fresh, and the precise byte-level shape of the ESP packets they produce. By the end, every field of an ESP packet on the wire will have a name and a reason.

Phase 2’s job, stated precisely

Phase 2 negotiates IPsec SAs — IKEv2 calls them child SAs, which is the better name since it makes the relationship explicit: they’re created under, and protected by, the Phase 1 IKE SA. Each negotiation produces a pair (Part 1’s rule: SAs are one-directional), each member with its own SPI and its own key material. Three things get agreed per negotiation:

• What traffic this SA pair covers — the traffic selectors (IKEv1: proxy IDs; IKEv2: TSi/TSr payloads).
• How it’s protected — the ESP cipher and integrity proposal, negotiated independently of Phase 1’s. The two proposals answer different questions: Phase 1’s protects IKE itself, Phase 2’s protects your data. They can differ, and a mismatch in either is its own distinct failure mode.
• The keys — derived from Phase 1’s key material plus fresh nonces, and optionally a brand-new DH exchange (PFS, below).

In IKEv1 this is quick mode: three messages inside the encrypted channel — request, response, and a third message proving liveness. IKEv2 folds the first child SA into IKE_AUTH (which is how it gets traffic flowing in four packets total) and uses CREATE_CHILD_SA exchanges for every subsequent SA and every rekey.

Traffic selectors: where policy-based and route-based VPNs actually differ

A selector is a promise: “this SA pair carries traffic from these sources to those destinations.” Both ends must agree, and IKEv1 is brittle about it — the proxy IDs must match essentially exactly, which is why “Phase 1 up, Phase 2 won’t establish” against Cisco, Palo Alto, or cloud-provider kit is so often a selector mismatch. The FortiOS log line to look for is no matching phase2 found, and the fix is comparing both ends’ selector definitions, not staring at crypto settings.

This is also the layer where the policy-based versus route-based distinction — which NSE4 presents as a FortiGate design choice — turns out to be a protocol posture:

• Policy-based: many narrow SA pairs, one per traffic rule. The SPD literally enumerates interesting traffic, and each entry can negotiate its own Phase 2. More SAs, more rekeys, more selector negotiations to mismatch.
• Route-based (FortiGate phase1-interface + phase2-interface): typically one SA pair with selectors of 0.0.0.0/0 ↔ 0.0.0.0/0 — “everything routed into this tunnel interface is interesting.” Traffic steering moves up to the routing table, where BGP, SD-WAN rules, and everything else this blog cares about already live.

That wide-open selector is the enabling decision for SD-WAN: the resilience series runs BGP over the overlay precisely because the tunnel will carry whatever routing hands it, no renegotiation required when a new prefix appears behind a spoke. A policy-based VPN would need a Phase 2 negotiation per selector change; a route-based VPN needs a routing update.

config vpn ipsec phase2-interface
    edit "overlay-hub-p2"
        set phase1name "overlay-hub"
        set proposal aes256gcm
        set pfs enable
        set dhgrp 19
        set keylifeseconds 3600
        # src/dst subnet default to 0.0.0.0/0 — the route-based posture
    next
end

PFS: what the checkbox actually buys

Phase 2 keys derive from Phase 1 key material. Follow that dependency to its uncomfortable conclusion: an adversary who records encrypted traffic for months and later compromises the Phase 1 secrets can derive every Phase 2 key ever produced from them, and retroactively decrypt the archive. Perfect Forward Secrecy breaks the chain by running a fresh, independent DH exchange inside every Phase 2 negotiation — KEi/KEr in quick mode, an optional KE payload in CREATE_CHILD_SA. Each SA pair’s keys then stand alone: compromising one window of traffic says nothing about any other.

The cost is one DH computation per rekey, per tunnel. On a 60-spoke estate rekeying hourly that’s background noise; the 50G packet-flow series made the point that control-plane crypto rides the CPU, not the NP7 — but a DH op per tunnel per hour is not where a hub CPU goes to die. Enable PFS; match the group on both ends (a PFS group mismatch is yet another “Phase 1 fine, Phase 2 dead” generator).

Rekeying: the part everyone configures and nobody thinks about

Keys age. The more data and time a single key covers, the more material a cryptanalyst has and the bigger the blast radius if it leaks — so SAs carry lifetimes (FortiOS defaults Phase 2 to 43200 s; the config above tightens it to an hour). The mechanics matter operationally:

Soft before hard. A well-behaved implementation starts negotiating the replacement SA at a soft threshold before the hard expiry, so the new pair is installed while the old one still works. Traffic never sees a gap — there are simply two valid SA pairs for a few seconds, and diagnose vpn tunnel list showing two SPIs per direction mid-rekey is healthy, not a bug.

Both ends count independently. Lifetime mismatches don’t prevent establishment — the shorter timer simply fires first and that end initiates rekey. Trouble arrives with byte-based lifetimes mixed against time-based ones across vendors, or with one end (historically, certain cloud endpoints) that expects always to be the rekey initiator. The symptom is a tunnel that drops for a few seconds at suspiciously round intervals; the diagnosis is comparing lifetime configuration, not crypto.

Rekey ≠ re-authentication. Rekeying a child SA derives new keys under the existing IKE SA. The IKE SA itself also has a lifetime, and its renewal (IKEv2 can rekey it in-place; IKEv1 builds a new Phase 1 alongside) is the heavyweight event. The distinction shows up at SD-WAN hub scale: a hub absorbing 500 spokes’ Phase 2 rekeys per hour barely notices; 500 simultaneous Phase 1 re-authentications after a hub failover is the thundering herd the resilience series’ staggered-lifetime advice exists for.

The ESP packet, field by field

Everything Phase 2 negotiated becomes concrete here — every field below exists to serve something from the last two posts.

SPI (32 bits, cleartext). The receiver’s index into its SAD: which SA, therefore which keys, therefore how to decrypt. It must be cleartext — it’s read before any decryption can begin. This is also why the receiver can host thousands of tunnels on one IP address and demultiplex instantly.

Sequence number (32 bits, cleartext, monotonic). Feeds anti-replay, below. It increments per packet per SA and is never allowed to wrap: approaching 2³² packets forces a rekey regardless of time or byte lifetimes (extended 64-bit sequence numbers exist for the genuinely high-throughput case, with only the low 32 bits on the wire).

IV (cipher-dependent). AES-CBC carries a 16-byte IV; AES-GCM an 8-byte one. It travels in clear because an IV isn’t a secret — its one job is never repeating under the same key. The GCM footnote is sharper: a repeated IV under one key doesn’t just weaken GCM, it breaks it catastrophically — one of several reasons counter-mode ciphers make rekey discipline non-negotiable.

Payload. In tunnel mode, the entire original IP packet — the inner header with its RFC 1918 addresses, the L4 header, the data. Encrypted in full, which is why a middlebox can’t see — let alone fix up — anything inside. Remember that for Part 3.

Padding, pad length, next header. Padding aligns the plaintext to the cipher’s block size (and incidentally blurs true payload lengths against traffic analysis). Next header says what the payload is: 4 means IP-in-IP — tunnel mode — and 6 would mean a raw TCP segment, transport mode. The receiver learns which mode this packet used from this one byte, after decryption.

ICV. The integrity check: HMAC-SHA256 output (truncated to 128 bits) with classic cipher+MAC suites, or carried within the authentication tag with AEAD ciphers. Coverage runs from the SPI through next header — and not over the outer IP header, the deliberate gap Part 1 flagged and Part 3 exploits.

On AEAD: aes256gcm in the proposals above selects authenticated encryption — encryption and integrity in one primitive, one pass, instead of AES-CBC followed by a separate HMAC. One negotiation knob fewer to mismatch, less per-packet work, and on FortiGate hardware the practical headline: NP7 offload of ESP is what makes line-rate IPsec possible at all, and AES-GCM is the suite the offload path is built around. The packet-flow series’ rule — first packet on the CPU, rest of the flow in hardware — applies to ESP exactly: IKE and the SA install are CPU work, then the NP7 encrypts/decrypts in the fast path.

Anti-replay: the window

Without protection, an attacker who can’t read an ESP packet can still record and re-send it — replaying yesterday’s “unlock the door” message verbatim. The defence costs almost nothing: the receiver tracks a sliding window (commonly 64–1024 packets wide) anchored at the highest sequence number seen. A packet beyond the right edge advances the window; inside the window, a bitmap marks it seen-or-not and duplicates are dropped; behind the left edge it’s dropped unconditionally — too old, no questions.

The operational wrinkle is QoS-induced reordering: aggressive priority queueing on an upstream device can delay bulk packets so far behind interactive ones that they fall off the window’s left edge and die as replay suspects. The symptom — unexplained loss on exactly one traffic class inside a tunnel — looks like nothing else, and diagnose vpn tunnel list exposing replay counters per SA is how it’s caught. FortiOS sizes windows generously by default; the fix when it bites is widening the window or taming the reordering, not disabling replay protection.

The overhead bill, and where the MTU bodies are buried

Tunnel-mode ESP with AES-256-GCM adds, on a 1500-byte path: new outer IP header (20) + SPI/sequence (8) + IV (8) + pad/pad-length/next-header (2–17) + ICV (16) — roughly 54–70 bytes, leaving an effective inner MTU around 1430–1446. Three failure tiers follow:

1. Inner packet small enough: fits, ships, nobody notices. Ping works.
2. Inner packet too big, DF clear: the FortiGate fragments. Slow path, reassembly cost at the far end, but functional.
3. Inner packet too big, DF set — every modern TCP flow: the gateway must send ICMP Fragmentation Needed back to the source, and if that gets filtered anywhere, the connection black-holes. “Ping works, HTTPS hangs” is this tier, and it’s an IPsec-overhead story so reliably that pmtud-sweeper was written largely to find which hop is eating the ICMP.

The standing mitigation, on every overlay policy this blog ships: clamp TCP MSS at the tunnel so endpoints never build an oversized segment — set tcp-mss-sender 1350 / set tcp-mss-receiver 1350 on the policy governing tunnel traffic. 1350 is deliberately conservative: it leaves room for ESP overhead plus the further 8 bytes Part 3 is about to add.

Verifying Phase 2 on a FortiGate

diagnose vpn tunnel list name overlay-hub

The single most information-dense IPsec command in FortiOS. In its output, everything from this post has an address: the SA pairs with their enc/dec SPIs (two per direction mid-rekey is normal), the negotiated proposal per SA, selectors, byte/packet counters per direction (a tunnel passing traffic one way only is a routing or selector asymmetry, not a crypto failure), replay drop counters, and the NPU flag showing whether ESP is actually riding the NP7 offload path or quietly burning CPU.

get vpn ipsec tunnel summary
diagnose vpn ike gateway list name overlay-hub

The summary for estate-wide eyeballing; the gateway list to confirm the Phase 1 SA the child SAs hang from — which closes the loop back to Part 1.

Where this lands

Part	Topic
1	The architecture: SAs, ESP vs AH, tunnel vs transport, IKEv1 main/aggressive, IKEv2
2 (this post)	Phase 2: child SAs, selectors, PFS, rekeying, anti-replay, and the ESP packet field by field
3	NAT vs IPsec: the three breaks, NAT-D detection, NAT-T on UDP 4500, port forwarding, and Fortinet SD-WAN

Both phases are now accounted for: a control channel that authenticates and negotiates, and data-plane SAs producing ESP packets whose every byte has a job. All of it assumed one quiet precondition — that the IP addresses each peer sees are the addresses the other peer actually has. Part 3 is about what happens when a NAT makes that assumption false, which it does on more or less every spoke circuit a real SD-WAN estate has ever deployed.