SDWAN Resilience Part 2: BGP on Loopback

In Part 1 we set the topology and the AS plan. This post lays down the BGP design end-to-end: why we peer on loopbacks instead of tunnel interfaces, how to make IPsec on the hub support that pattern, and how the no-DCI constraint shapes the hub-to-hub story.

By the end you should be able to copy the CLI snippets straight onto a hub and a spoke and have a working BGP overlay.

Why peer on loopbacks at all?

The naive way to run BGP across an IPsec overlay is to peer on the tunnel-interface IPs. It works on a brand-new lab build without any of the awkwardness below. The problem is what happens during a failure.

Tunnel-interface IPs on FortiOS dynamic IPsec are assigned by the hub via mode-config, and they change when the tunnel re-establishes. Even with set exchange-interface-ip enable, the IP a spoke ends up with after a flap is whatever the hub had free at the time, not “the same one as before”. If your BGP peer is pinned to that address, every flap is a peer-reconfigure event.

Loopback peering avoids that. The loopback IPs are static, they survive tunnel flaps, and they are independent of which underlay the tunnel happens to be running over. That stability is also the foundation that BFD-for-BGP and ECMP-on-overlay rest on later in the series.

A second reason: a loopback gives BGP a stable router-id that doesn’t drift when interfaces come and go. FortiOS picks the highest interface IP if you don’t pin a router-id, and on a hub with dozens of dynamic spokes that “highest IP” will move. Pin the router-id to the loopback and forget about it.

Fortinet’s SD-WAN Architecture for Enterprise and the BGP on loopback recipe both recommend this pattern. The recipe is the design template for any non-trivial Fortinet SD-WAN.

Loopback plan

From Part 1:

Device	Loopback
HUB-1	10.255.0.1/32
HUB-2	10.255.0.2/32
SPOKE-N	10.255.1.N/32

Tunnel-interface underlay is per spoke per hub, assigned via mode-config from a /24 per hub. The loopback subnets are advertised by both ends so they are reachable across the overlay.

Hub-side IPsec (HUB-1)

Dynamic dial-up phase 1 with IKEv2, multiple overlays, and mode-config:

config vpn ipsec phase1-interface
    edit "spokes-h1"
        set type dynamic
        set interface "wan1"
        set ike-version 2
        set peertype any
        set net-device disable
        set mode-cfg enable
        set ipv4-start-ip 10.254.0.1
        set ipv4-end-ip 10.254.0.254
        set ipv4-netmask 255.255.255.0
        set add-route disable
        set network-overlay enable
        set network-id 1
        set exchange-interface-ip enable
        set dpd on-idle
        set psksecret <preshared-secret>
    next
end

config vpn ipsec phase2-interface
    edit "spokes-h1"
        set phase1name "spokes-h1"
        set proposal aes256-sha256
        set pfs enable
        set dhgrp 14
    next
end

The crucial bits:

• set add-route disable — by default the hub installs a /32 for each spoke’s tunnel-interface IP via the tunnel itself. Turn it off; we’ll let BGP install the proper routes (including the spoke loopback). Without this, a static /32 shadows the BGP route and the next-hop-self trick later misbehaves.
• set network-overlay enable and set network-id 1 — required when the hub has more than one dynamic phase 1, and good hygiene even with one. The second hub uses network-id 2 so the two are distinguishable.
• set exchange-interface-ip enable — the spoke learns the hub’s tunnel-interface IP, which is required for the recursive lookup that resolves the hub’s loopback over the tunnel.

On HUB-2, the configuration is identical except network-id 2 and a different mode-config range (10.254.1.0/24) so the two hubs never hand out the same tunnel address to a spoke.

Hub-side loopback and BGP (HUB-1)

config system interface
    edit "lo0"
        set vdom "root"
        set type loopback
        set ip 10.255.0.1 255.255.255.255
        set allowaccess ping
    next
end

config router bgp
    set as 65000
    set router-id 10.255.0.1
    set graceful-restart-time 120
    set ibgp-multipath enable

    config neighbor-group
        edit "spokes"
            set remote-as-filter "spoke-as-list"
            set update-source "lo0"
            set ebgp-enforce-multihop enable
            set route-reflector-client enable
            set next-hop-self enable
            set capability-graceful-restart enable
            set bfd enable
            set route-map-in "in-from-spokes"
            set route-map-out "out-to-spokes"
        next
    end

    config neighbor-range
        edit 1
            set prefix 10.255.1.0 255.255.255.0
            set neighbor-group "spokes"
        next
    end
end

config router as-path-list
    edit "spoke-as-list"
        config rule
            edit 1
                set action permit
                set regexp "^651[0-9][0-9]$"
            next
        end
    next
end

Notes:

• neighbor-range plus neighbor-group is the dynamic-peer pattern. Any spoke whose loopback falls in 10.255.1.0/24 and whose ASN matches the regex peers automatically. No per-spoke configuration on the hub.
• set route-reflector-client enable is set even though there is no iBGP fabric here — it’s a no-op on eBGP peers and protects you when ADVPN is added later.
• set next-hop-self enable — without this, when the hub reflects a route from one spoke to another (if/when ADVPN is added) the next-hop will be the originating spoke and unreachable. Even on day one it’s the right default.
• set bfd enable — sets BFD-for-BGP. Configuration of the BFD timers themselves is in Part 4.

Static route to make spoke loopbacks reachable via the tunnel:

config router static
    edit 10
        set dst 10.255.1.0 255.255.255.0
        set device "spokes-h1"
        set blackhole disable
    next
end

A single route-via-tunnel-interface for the entire spoke loopback range is the only manual routing on the hub. BGP does the rest.

Spoke-side IPsec (SPOKE-1)

config vpn ipsec phase1-interface
    edit "to-hub1"
        set interface "wan1"
        set ike-version 2
        set peertype any
        set mode-cfg enable
        set network-overlay enable
        set network-id 1
        set add-route disable
        set exchange-interface-ip enable
        set dpd on-idle
        set remote-gw <HUB1-public-ip>
        set psksecret <preshared-secret>
    next
    edit "to-hub2"
        set interface "wan1"
        set ike-version 2
        set peertype any
        set mode-cfg enable
        set network-overlay enable
        set network-id 2
        set add-route disable
        set exchange-interface-ip enable
        set dpd on-idle
        set remote-gw <HUB2-public-ip>
        set psksecret <preshared-secret>
    next
end

set add-route disable on the spoke side stops it from auto-installing a /32 to the hub’s tunnel-interface IP via the tunnel — same reasoning as on the hub, we want BGP to drive routing.

Spoke loopback, reachability, and BGP

config system interface
    edit "lo0"
        set type loopback
        set ip 10.255.1.1 255.255.255.255
        set allowaccess ping
    next
end

config router static
    edit 10
        set dst 10.255.0.1 255.255.255.255
        set device "to-hub1"
    next
    edit 11
        set dst 10.255.0.2 255.255.255.255
        set device "to-hub2"
    next
end

config router bgp
    set as 65101
    set router-id 10.255.1.1
    set graceful-restart-time 120

    config neighbor
        edit "10.255.0.1"
            set remote-as 65000
            set update-source "lo0"
            set ebgp-enforce-multihop enable
            set capability-graceful-restart enable
            set bfd enable
            set route-map-out "advertise-local"
        next
        edit "10.255.0.2"
            set remote-as 65000
            set update-source "lo0"
            set ebgp-enforce-multihop enable
            set capability-graceful-restart enable
            set bfd enable
            set route-map-out "advertise-local"
            set route-map-in "prefer-h1"
        next
    end
    config network
        edit 1
            set prefix 10.20.1.0 255.255.255.0
        next
    end
end

The two static /32s pin reachability to each hub’s loopback over the corresponding tunnel. Without them, the spoke knows the hub’s tunnel-interface IP (via mode-config) but has no way to resolve the loopback recursively.

The prefer-h1 route-map (applied inbound on the HUB-2 peer) bumps local-preference down on routes received from HUB-2:

config router route-map
    edit "prefer-h1"
        config rule
            edit 1
                set set-local-preference 50
            next
        end
    next
end

Routes from HUB-1 keep the default 100; routes from HUB-2 are 50. The spoke prefers HUB-1. That is the active/standby preference enforced in BGP.

You could equivalently enforce the same outcome by having HUB-2 AS-path-prepend its advertisements toward spokes. Either direction works. Setting it on the spoke is cleaner because the policy is “this spoke prefers DC1” — which is a property of the spoke, not of the hub. On the hub, the same operator change would need to be coordinated across all spokes.

Hub-to-hub iBGP — and why we don’t run it

In a single-AS hub design, the obvious next step is iBGP between HUB-1 and HUB-2. But:

• There is no DCI.
• Running iBGP through the DCE means the DCE has to carry our internal routes, which it shouldn’t.
• Running iBGP across the spoke overlay means using a spoke as transit, which is a security and capacity disaster.
• Building a dedicated IPsec tunnel between the two hubs over the public internet is technically a DCI-by-stealth and reintroduces every problem we said we accepted by not building one.

The cleanest answer for this topology is: don’t run iBGP between hubs. Each hub maintains its own eBGP relationships:

• eBGP downstream to its set of spokes (covered above).
• eBGP / OSPF / static upstream to the DCE (Part 3).

Spoke prefixes show up in the DCE from both hubs, so the DCE has two paths to every spoke. The DCE picks one (Part 5 will talk about how to influence that choice with MED) and the route is symmetric per active/standby.

What the hubs give up by not running iBGP:

• They don’t see each other’s spokes in their RIBs. That’s fine — they don’t need to.
• They can’t share liveness signals. That has to come from BFD per peering, plus end-to-end SLA at the spoke.
• They can’t act as transit for each other on a partial outage. That’s a feature in this design — we want hubs to fail closed, not fail open.

If you later add a DCI, that calculus changes immediately and iBGP between hubs becomes the obvious thing to do. The recommendation here is specific to “no DCI, active/standby.” Worth re-evaluating any time the topology changes.

What about ADVPN?

Out of scope for this design. Active/standby with no DCI is already a constraint that doesn’t compose well with ADVPN shortcuts. ADVPN announcements piggyback on the BGP session, so the wiring above is compatible with adding ADVPN later, but turning it on without addressing the DCI gap creates spoke-to-spoke shortcuts that can cross unintended trust boundaries.

Verification

On the hub:

get router info bgp summary
get router info bgp neighbors 10.255.1.1 received-routes
diagnose ip rtcache list

get router info bgp summary should show every spoke as Established with sane Up Time. If a spoke is bouncing between Active and Idle the usual culprits are add-route still enabled (a static route shadowing BGP’s /32), update-source not pointing at the loopback, or ebgp-enforce-multihop missing.

On the spoke:

get router info bgp summary
get router info bgp neighbors 10.255.0.1 advertised-routes
diagnose vpn ike gateway list name to-hub1

get router info bgp neighbors <hub> advertised-routes is the single best command for “is the spoke actually advertising what I think it is to that hub?”.

Common gotchas

• add-route still enabled. Symptom: BGP comes up, routes look right, but traffic blackholes intermittently. The auto-installed /32 from IPsec mode-config is shadowing the BGP next-hop resolution. Disable on both ends.
• Mode-config IP exhaustion. A /24 hands out 254 spoke addresses, which sounds generous until you realise every hub flap re-leases. Pick a /22 if you have any scale ambitions, or watch for phase1: ip-pool exhausted log lines.
• IKEv1 left over from a copy-paste. If you mix IKEv1 and IKEv2 across spokes you’ll spend a happy afternoon tracing why some spokes don’t pick up the right network-id.
• Loopback /32 not advertised. Easy to forget. Without it, the other end’s BGP comes up but the next-hop is unreachable, so the routes are hidden. Pin the loopback into a network statement or redistribute connected with a prefix-list.
• Router-ID drift. If set router-id isn’t pinned and the loopback bounces, FortiOS picks a new ID and BGP everywhere resets. Always pin it.

Where Part 3 picks up

The hubs now have BGP to spokes. The next problem is what happens above the hub: how each hub talks to the DCE. Three flavours — static, OSPF, and BGP — each with its own trade-offs. Part 3 walks through them, including which one actually behaves correctly during a partial DCE failure given the no-DCI constraint we just locked in.