SDWAN Resilience Part 6: Building It Right — Full DCI and Dual-Active ADVPN

8 May 2026 FortinetSDWANResilienceBGPADVPNNetworking

The first five parts of this series walked through a real topology with real constraints — HA FortiManager, dual hubs in active/standby, no DCI between data centres, source-IP-pinned backends. The design was defensible against Fortinet’s published Best Practice where the BP fit the constraints, and challenged where it didn’t. The result is a network that converges in 1–3 seconds and survives every failure mode the constraints expose.

This post is the version without those constraints. If you’re greenfielding, or if you can talk the budget out of a DCI, this is what Fortinet’s SD-WAN Architecture for Enterprise points at as the reference design — and what you should build if nothing is stopping you. Less compromise, more bandwidth, simpler steady state.

What we get to drop

Three constraints from Part 1 disappear here:

Active/standby preference. Both DCs carry production traffic in steady state. DC2 is no longer half-idle insurance.
No DCI. A dedicated, redundant interconnect between DC1 and DC2 means the hubs share a control plane and a fast path between each other.
Source-IP pinning. The application stack accepts session origination from either DC’s egress range, or sits behind a load balancer that doesn’t pin on source. (If the apps still pin, you don’t get to drop this constraint and you should stay with the design from Parts 1–5.)

Drop those three and the design simplifies in ways that compound. iBGP between hubs becomes obvious. ADVPN becomes safe. Symmetric routing becomes solvable. The SD-WAN SLA layer stops being failover plumbing and starts being application-aware steering.

The reference topology

                    +----------------------+
                    |   DCE (AS 65500)     |
                    |  Application Stack   |
                    +----+------------+----+
                         |            |
                    eBGP |            | eBGP
                  (with comms)   (with comms)
                         |            |
                +--------+---+ DCI +--+--------+
                |  HUB-1     |=====|  HUB-2    |
                |  (DC1)     | iBGP|  (DC2)    |
                |  AS 65000  | BFD |  AS 65000 |
                +-----+------+     +-----+-----+
                      | IPsec  +------+ IPsec
                      |       both    |
                      |       active  |
                      +-+---+-----+---+
                        |   |     |
                        |   | ADVPN shortcuts
                        |   |     |
                +-------+---+-+   +-+--------+
                |  SPOKE-1    |   | SPOKE-2  |
                |  AS 65101   |   | AS 65102 |
                +-------------+   +----------+

The pieces that change:

DCI between HUB-1 and HUB-2. Two physical paths if you can get them — typically one private (DWDM, dark fibre, or a dedicated MPLS LSP) and one public-internet IPsec as a backup. Run BFD on both, use ECMP, and the iBGP session ECMPs across them.
iBGP between hubs, sourced from loopbacks, with BFD-for-BGP. Both hubs see each other’s spoke prefixes and DCE-learned prefixes.
ADVPN is enabled. Spoke-to-spoke shortcuts are negotiated dynamically via the hubs.
Hubs cooperate on DCE policy. Communities tag prefixes by origin DC; the DCE chooses based on preference but both paths are in the RIB.

DCI: design the dependency you’ll lean on

The DCI is the thing that the rest of the simplification depends on, so it’s worth building it once and properly.

config system interface
    edit "dci-primary"
        set vdom "root"
        set ip 10.0.255.1 255.255.255.252
        set bfd enable
        set bfd-required-min-rx 200
        set bfd-desired-min-tx 200
        set bfd-detect-mult 3
    next
end

config vpn ipsec phase1-interface
    edit "dci-backup"
        set type static
        set interface "wan2"
        set ike-version 2
        set peertype any
        set remote-gw <HUB2-public-ip>
        set net-device disable
        set add-route disable
        set network-id 99
        set psksecret <preshared>
    next
end

config router static
    edit 10
        set dst 10.255.0.2 255.255.255.255
        set device "dci-primary"
        set priority 10
    next
    edit 11
        set dst 10.255.0.2 255.255.255.255
        set device "dci-backup"
        set priority 20
    next
end

tx 200 / mult 3 = 600 ms is more aggressive than the spoke-side BFD because the DCI underlay is, by construction, more reliable than internet to a branch. If your DCI is dual-pathed at L1, drop to tx 100 / mult 3 = 300 ms and you’ll get sub-second iBGP convergence on a DCI fibre cut.

The DCI is only iBGP and BFD. No customer traffic on it directly — customer traffic chooses a hub and stays at that hub for the egress decision. The DCI is the control-plane interconnect.

iBGP between hubs

config router bgp
    set as 65000
    set router-id 10.255.0.1
    set ibgp-multipath enable

    config neighbor
        edit "10.255.0.2"
            set remote-as 65000
            set update-source "lo0"
            set bfd enable
            set capability-graceful-restart enable
            set next-hop-self enable
            set route-reflector-client enable
        next
    end
end

Two things to note. First, next-hop-self between two hubs in the same AS is technically unnecessary if the spokes can resolve the other hub’s loopback (they can — every spoke has both hub loopbacks in static and BGP). Set it anyway: it makes the spoke RIB cleaner and removes a dependency on transitive recursion that doesn’t pull its weight.

Second, route-reflector-client on each side of a two-hub iBGP session is harmless and useful: if you ever scale to a third hub, the iBGP topology becomes a route-reflector pair, and the config doesn’t have to change.

ADVPN, briefly

ADVPN is the feature that lets two spokes establish a direct shortcut after the first packet between them goes via the hub. The hub recognises the flow, signals both spokes, and they negotiate a direct IPsec SA.

The config delta from Part 2 is small — add set auto-discovery-sender enable on the hub and set auto-discovery-receiver enable on the spokes:

# HUB
config vpn ipsec phase1-interface
    edit "spokes-h1"
        set auto-discovery-sender enable
        set auto-discovery-shortcuts enable
    next
end

# SPOKE
config vpn ipsec phase1-interface
    edit "to-hub1"
        set auto-discovery-receiver enable
    next
end

The implication is that spoke-to-spoke traffic stops hairpinning through the hub, which matters at scale (a hub stops being a bandwidth chokepoint) and for latency (spoke-to-spoke is one IPsec hop instead of three).

ADVPN is only safe when you trust spoke-to-spoke directly — which means you’ve thought about per-spoke firewall posture, segmentation, and what shortcuts mean for your blast radius. In a real enterprise build this is also the point at which you start using FortiManager templates aggressively, because the spoke security policy needs to be uniform.

Dual-active routing: how the spoke chooses

In Parts 1–5 the spoke applied a local-pref 50 route-map to HUB-2 advertisements to enforce active/standby. Strip that route-map. Both hubs advertise with default local-pref 100, the spoke’s BGP best-path tie-breaks on AS-path length (equal — both are 65000 from the spoke’s view), then on MED (set both hubs to MED 0), then on cluster-list / router-id. The hubs are configured for set ibgp-multipath enable and set ebgp-multipath enable, and the spoke is configured similarly:

config router bgp
    set ebgp-multipath enable
    config neighbor
        edit "10.255.0.1"
            set remote-as 65000
        next
        edit "10.255.0.2"
            set remote-as 65000
        next
    end
end

The result: every DCE prefix has two equal-cost paths in the spoke RIB, one via each hub. ECMP installed in the FIB. Flows are hashed across the two paths.

Per-flow ECMP is good for bandwidth, neutral for latency (assuming the two paths are comparable), and bad for stateful inspection if the return path doesn’t match. That last problem is the next subsection.

Symmetric routing: solving it once

The classic dual-active failure mode: a flow from SPOKE-1 hashes onto HUB-1’s path, the request crosses DC1’s egress firewall, lands at the application; the application replies, the DCE chooses HUB-2 as best path back to SPOKE-1, the reply crosses DC2’s egress firewall — which has no state for the connection, drops the packet, and the user blames you.

There are three serious answers and they’re not mutually exclusive:

Don’t put stateful inspection at the egress. Put inspection at the hub itself, before traffic enters the DC at all. Each hub inspects on the way in and the way out, and the DC interior is a transit fabric. This is the cleanest approach and the one Fortinet’s reference designs implicitly assume — the FortiGate hub is the inspection point.
Symmetric path enforcement via DCE-side policy. The DCE sees two paths to SPOKE-1 (one via HUB-1, one via HUB-2) and chooses based on community/local-pref per spoke prefix. The hub-side route-maps tag spoke advertisements with a community indicating “originated by DC1” or “originated by DC2”. The DCE applies a route-map that prefers DC1 for half the spokes and DC2 for the other half — coarse, but it makes return symmetric per spoke. Combine with the spoke-side outbound NAT below for full bidirectional pinning.
Source NAT at the hub. Each hub source-NATs spoke→DCE traffic to a hub-local pool. Now the DCE always returns to the hub the request came from, regardless of which spoke originated it. Costs you visibility of spoke source IP at the application layer (logs show the hub NAT pool), and breaks if the application authorises by source IP. Useful as a fallback.

Option 1 is the architecturally clean answer. Options 2 and 3 are workarounds for environments where the inspection points aren’t movable.

SD-WAN SLA: still here, doing a different job

In Parts 1–5 the SD-WAN SLA layer was the failover detector for soft failures. In dual-active it’s still important, but its primary role shifts: application-aware steering.

Two members are equally healthy at L3. The SLA tells you which is healthier for this application class. Voice traffic prefers the lower-jitter member. Bulk SMB prefers the lower-loss member. Neither is “the failover path” — both are first-class.

config system sdwan
    config service
        edit 1
            set name "voice"
            set mode sla
            set dst "voice-dst"
            set health-check "voice-jitter-check"
            set sla-compare-method order
            config sla
                edit "voice-jitter-check"
                    set id 1
                next
            end
            set priority-members 1 2
        next
        edit 2
            set name "bulk"
            set mode sla
            set dst "bulk-dst"
            set health-check "bulk-loss-check"
            set sla-compare-method order
            config sla
                edit "bulk-loss-check"
                    set id 1
                next
            end
            set priority-members 2 1
        next
    end
end

The two service rules pick opposite priority orderings. Steady state, voice prefers HUB-1 and bulk prefers HUB-2. If either degrades, both classes converge to the surviving hub. Failover is automatic; steady state spreads load by application class.

Comparison table

Property	Constrained design (Parts 1–5)	Reference design (this post)
DCI	None	Dedicated, redundant, with BFD
Hub-to-hub control plane	None	iBGP over DCI
DC mode	Active/standby	Dual-active
Spoke-to-spoke	Hairpin via hub	ADVPN shortcut
Spoke RIB for DCE prefix	Single best-path (HUB-1 LP 100, HUB-2 LP 50)	ECMP via both hubs
Symmetric routing	Trivial (single egress)	Requires hub-as-inspection or community-driven DCE policy
Bandwidth utilisation	~50% (DC2 idle)	~100%
Failure-domain blast radius	Smaller — one DC at a time	Larger — design must assume both can fail in different ways
SLA layer purpose	Failover for soft failures	Application-aware steering + failover
Convergence target	1–3 s	1–3 s (same; same mechanisms)
Operational complexity	Lower steady state, harder failure modes to reason about	Higher steady state, easier failure modes
Fortinet BP alignment	Calls out as supported	Explicit reference design

The convergence numbers are identical. BFD doesn’t care which design it’s running in. The difference is what happens after convergence: in active/standby you’re now on the only working path; in dual-active you’ve shed half your bandwidth and the other half is fine.

What’s still hard, even with the full shebang

Don’t read this post as “do all of the above and you have nothing to worry about.” Three things stay hard:

Asymmetric routing during partial failures. When one path of two ECMP paths is degraded but not failed, flows hashed onto the bad path don’t automatically re-hash. Per-flow ECMP is sticky by design. The fix is application-layer (TCP retransmits move the conversation) or per-flow SLA (the service rule re-pins on threshold breach). Tune carefully.
DCI as a single failure domain. A DCI fibre cut on both physical paths simultaneously is rare but not zero, and the design assumes it doesn’t happen for very long. If your two DCs must survive total DCI loss, you’ve reverted to the constrained design until the DCI is back. Have an operational runbook that says so.
FortiManager HA across the DCI. The HA pair now has a fast path between members, which is good. It also means the HA pair can’t survive DCI loss as cleanly as before — both members will see each other as “remote” and may fight over primary. Configure HA priorities to deterministically prefer one member regardless, and test the partition behaviour.

When to choose which design

The decision tree is short:

Pick this (Part 6) reference design if you’re greenfielding, you can build a DCI, your applications don’t pin on source, and you want the bandwidth.
Pick the constrained design (Parts 1–5) if you have any of: legacy source-pinning apps you can’t change, no budget for a DCI, smaller-blast-radius operational preference (a real and respectable choice in some shops), or DCs operated by separate teams with different SLAs.
Pick a phased migration if you’re somewhere in between. Start with the constrained design (it works), build the DCI, switch to dual-active when the DCI is proven, enable ADVPN when the spoke security posture is uniform.

Either design, properly built, will converge in 1–3 seconds. Either will survive a hub failure without users noticing if the application is well-behaved. The choice is about what you’re optimising for outside the failure mode — bandwidth utilisation, blast radius, simplicity of reasoning.

Series wrap (for real this time)

Six parts, two designs, one reference topology family. The bits worth taking away:

Pick the design that matches the constraints, not the one that looks newest. Active/standby still ships in 2026. Defend it when it’s right; replace it when the constraints lift.
BGP-on-loopback, BFD on tunnels and BGP, hold/keepalive at 3/9, OSPF (where used) at 1/4, BFD tx 250 / mult 5 — those are the convergence-tuning numbers that work, regardless of which design is running.
SD-WAN SLA is the layer that catches failures the routing can’t see. In active/standby it’s the soft-failure detector. In dual-active it’s the application steering layer. Either way, design the health-check target carefully.
The DCI is the dependency that unlocks everything else. If you can build it, build it.

If a future post is going to push this further, it’s about the bits that needed their own series the moment they came up: FortiManager templating to make either design operable at scale, FortiSASE if cloud-delivered hubs change the failure model, and the segmentation story if ADVPN shortcuts cross trust boundaries. None of that fit here. Series done.