Proving the Path Part 4: SD-WAN Member, SLA, and Rule Selection

Part 3 confirmed that a brand-new Teams call from SPOKE-14 would route correctly right now, which rules out routing and policy route as the ongoing cause. That pushes the question onto the SD-WAN layer itself: what actually happened during the SLA breach, and did the failover mechanism do what it was supposed to?

What the health-check actually recorded

The Performance SLA design here is the same pair from SD-WAN Resilience: Performance SLAs and Service Steering: a fast ICMP probe and a slower HTTP probe against a DCE target, feeding a service rule set to sla-compare-method order so it always prefers the highest-priority member that’s currently passing:

diagnose sys sdwan health-check
Health Check(dce-probe):
  Seq(1 to-hub1): state(alive), packet-loss(2.400%) latency(41.200), jitter(6.800) sla_map=0x0
  Seq(2 to-hub2): state(alive), packet-loss(0.100%) latency(58.900), jitter(4.100) sla_map=0x1

state(alive) on both members throughout the window is the detail worth sitting with. HUB-1’s link never went down: it stayed up, stayed reachable, and would pass a naive “is the member alive” check the entire time. What actually tripped was sla_map, because packet loss on HUB-1 briefly crossed the configured threshold (1%) while HUB-2 stayed clean. That’s the SLA layer catching something link-state monitoring alone never would.

diagnose sys sdwan service for the Teams rule during the incident, then again now, tells the rest of the story:

# during the breach
Service(1): dce-services  ... sla_map=0x1 -> selected(to-hub2)

# now, after recovery
Service(1): dce-services  ... sla_map=0x0 sla_map=0x1 -> selected(to-hub1)

HUB-1 failed SLA, the rule failed new flows over to HUB-2 for the duration of the breach, HUB-1 recovered, and, per the recoverytime setting on the health-check, the rule failed back to HUB-1 once it had stayed clean for the required number of consecutive passing probes.

The SD-WAN layer did exactly what it was designed to do

That’s the finding, and it’s worth stating without qualification: this isn’t a bug, a misconfiguration, or a health-check that fired late. A brief loss spike happened, the SLA caught it faster than a link-state check would have, new flows were steered around it, and the rule failed back cleanly once the path was genuinely healthy again. Everything about the SD-WAN and SLA mechanism, evaluated on its own terms, worked.

False conclusion checkpoint: the member is alive, so SD-WAN should use it

This is the inverse of the trap and it’s worth naming directly, because it’s the instinct that would have led an investigation astray in the other direction. “HUB-1 never went down, so how could SD-WAN possibly have moved traffic off it?” Alive and SLA-eligible are different things, and the entire reason a Performance SLA health-check exists on top of link and BGP monitoring is to catch exactly this kind of soft failure: a member that’s up, routable, and BFD-healthy, but briefly failing to meet a latency, jitter or loss threshold. state(alive) was true the whole time; sla_map is what actually decided the routing outcome, and it’s the field worth checking, not the interface status.

What this mechanism never touches

The detail that matters for the rest of this investigation: everything above (the health-check, the SLA state, the service rule’s selected() output) governs what a new session evaluation does. It has no mechanism for reaching into an already-established session and telling it to re-run policy route, routing, SD-WAN and NAT selection. FortiOS doesn’t offer a “re-steer existing sessions on SLA change” option, by design; renegotiating an in-flight flow’s path mid-session is disruptive in its own right, and the platform doesn’t do it automatically.

Where this leaves us

The SD-WAN and SLA layer isn’t the fault, and it never was. It correctly protected every new Teams call during the breach window. The one call that stayed choppy did so precisely because it was already running before the breach started, and nothing in the SD-WAN mechanism was ever going to touch it. Part 5 checks the layer that call’s session is actually still committed to (the firewall policy, NAT and IPsec tunnel it selected at creation) to confirm there’s nothing else going on there before I call this settled.