Proving the Path Part 1: The Chain, and the Fault We're Chasing

Every troubleshooting session starts with someone’s conclusion already half-formed. The tunnel’s up, so it’s not IPsec. The member’s alive, so it’s not SD-WAN. Ping works, so it’s not the network. Each of those sentences is wrong often enough that I’ve learned to treat them as suspects rather than facts, and the only thing that settles the question is evidence tied to a specific stage of the packet’s path, not a dashboard tile that’s green.

This series is about that discipline. A Day in the Life of a Packet on a 50G FortiGate already walked the architecture of that path in detail: the NP7 fast path, the session table, the FIB, firewall policy, NPU offload. If you haven’t read it, this series will make more sense alongside it, because I’m going to lean on the same mechanics without re-explaining them from scratch. What I want to do here is ask a different question of the same material: given a live fault, how do you prove which stage is lying to you?

Rather than inventing a fresh toy example for every stage, I’m threading one fault through all six parts. That’s closer to how a real ticket behaves. Symptoms show up downstream of the actual cause, and every stage looks fine in isolation right up until you check it against the others.

The fault

The lab for this series is the same dual-hub, BGP-on-loopback topology from SD-WAN Resilience: BGP on Loopback: a branch, SPOKE-14, with two overlay members. HUB-1 is preferred (SD-WAN member priority 10), HUB-2 is standby (priority 20), and a Performance SLA health-check (the ICMP-plus-HTTP pair from SD-WAN Resilience: Performance SLAs and Service Steering) drives failover for the Teams and Microsoft 365 service rule.

A user on a Teams call from SPOKE-14 reports the call going choppy for two or three minutes, then settling. The health-check history shows exactly what you’d expect: a packet-loss spike on the HUB-1 path around the same time, an SLA breach, a failover event on the service rule, then a clean recovery a couple of minutes later. That reads like a closed case: the SLA layer did its job and the network self-healed.

Except the choppy call didn’t get better when the dashboard did. It stayed bad until the user hung up and redialled, and the next call was clean from the start. Something in that story doesn’t fit “SD-WAN failed over and fixed it,” and the only way to find out what is to go and check evidence at a specific stage rather than pattern-match from a graph.

The chain

A packet’s path through a FortiGate breaks down into a small number of stages, and each one answers a specific, narrow question:

  1. Ingress. Did the frame arrive on the interface and VRF I expect?
  2. Session. Does an existing session already hold this flow’s policy, NAT, route and egress decision?
  3. Policy route. Does explicit PBR or an SD-WAN service rule match this flow’s 5-tuple?
  4. Routing. Is there a valid route to the destination, resolved recursively where it needs to be?
  5. SD-WAN. Is the currently-selected member SLA-eligible, and which member does the rule actually pick right now?
  6. Policy and NAT. Which firewall policy, NAT mode and IP pool applies?
  7. IPsec. Are the phase 1 and phase 2 SAs for the chosen path actually up, with correct selectors?
  8. Egress and return. Did the packet leave, and did a reply come back on the path I expect?
  9. NPU offload. Could hardware acceleration be hiding what happens to the flow after its first few packets?

None of that is exotic, and if you’ve read the packet-flow series it’s the same territory. The part that’s easy to skip under pressure is stage 2, and almost every wrong conclusion in this series traces back to evaluating stages 3 through 9 against the current configuration while a live flow is still working off a decision it made minutes or hours earlier. An SD-WAN failover changes what a new session will do next. It says nothing about what an already-established session is doing, because FortiOS doesn’t re-run policy, NAT, route or egress selection for a session once it’s created. That behaviour has a name: session pinning. It’s the thread running through the rest of this series.

Where this is going

Six parts, each working the Teams-call fault one stage further:

  • Part 2. Ingress and the session table: does the choppy call’s session still exist, and what does it say about itself?
  • Part 3. Policy route and routing: what would a new flow to the same destination do right now, and does that match what the old session is doing?
  • Part 4. SD-WAN member and SLA selection: what actually triggered the failover, and did it ever apply to the call already in progress?
  • Part 5. Policy, NAT and IPsec: which policy ID and which tunnel is the pinned session using, and is that tunnel itself healthy?
  • Part 6. Egress, NPU offload and the verdict: proving where the packet actually went during the bad window, and what fixing it properly looks like.

Part 2 starts where every diagnosis should: not at the dashboard, and not at the configuration, but at the session table.