FortiGate BGP Cookbook Part 1: Dual-ISP Edge Recipes

I have written about BGP on FortiGate from the overlay side: peering on loopbacks across IPsec, BFD tuning, VRF route leaking. What I have not written is the plain internet edge, a FortiGate with two ISPs, a public AS number and a /24 of its own. That is where a lot of people meet BGP on a FortiGate for the first time, and it is where a small mistake gets announced to the rest of the internet.

This post and its pair are a cookbook. Each recipe has the problem, the config, what to check, and what bites.

One thing to be clear about: I did not lab this. Every command here was checked against Fortinet’s CLI reference, administration guide and knowledge base articles, and I describe what the output should show rather than pasting captured output I do not have. If something differs on your firmware, trust your box over this page and tell me. The theory behind the policy choices is in the BGP Deep Dive, Part 8 on filtering and Part 9 on traffic engineering. Part 2 of this pair covers verification and debugging.

The scenario

All addresses and AS numbers are from the documentation ranges, so substitute your own.

ItemValue
Our AS64500
Our prefix198.51.100.0/24
ISP1AS 64496, peer 203.0.113.1, our side 203.0.113.2/30 on wan1
ISP2AS 64497, peer 192.0.2.1, our side 192.0.2.2/30 on wan2

The goal is ordinary: use ISP1 as primary, ISP2 as backup, take a default route from each, and announce only our own /24.

Recipe 0: the naive baseline

This is the config most people start with, and the reason the rest of the post exists.

config router bgp
    set as 64500
    set router-id 198.51.100.1
    config neighbor
        edit "203.0.113.1"
            set remote-as 64496
        next
        edit "192.0.2.1"
            set remote-as 64497
        next
    end
end

Both sessions come up. Nothing is filtered in either direction, and nothing is authenticated. Everything after this is a change to this block.

Recipe 1: do not become a transit AS

The problem. BGP advertises its best path for a prefix to every eBGP peer by default. A FortiGate with two ISPs and no outbound policy will take routes it learned from ISP1 and offer them to ISP2, and the reverse. You have volunteered to carry traffic between two providers, over whatever link you bought for your office. Even if you only accept a default route, you would hand ISP1’s default to ISP2.

The fix. Say exactly what you intend to announce. A prefix list holds the prefixes, a route map applies it, and the route map goes on each neighbor in the outbound direction.

config router prefix-list
    edit "OUR-PREFIXES"
        config rule
            edit 1
                set action permit
                set prefix 198.51.100.0 255.255.255.0
                unset ge
                unset le
            next
        end
    next
end

config router route-map
    edit "ISP1-OUT"
        config rule
            edit 1
                set match-ip-address "OUR-PREFIXES"
            next
            edit 100
                set action deny
            next
        end
    next
end

config router bgp
    config neighbor
        edit "203.0.113.1"
            set route-map-out "ISP1-OUT"
        next
    end
end

Rules in a FortiOS route map are processed by rule ID, so the permit at ID 1 is checked before the catch-all deny at ID 100. I add the explicit deny rather than rely on what the platform does when nothing matches. Build the same map for ISP2 (it becomes ISP2-OUT in recipe 3).

There is a second way to express the same intent, using the AS path instead of the prefix:

config router aspath-list
    edit "LOCAL-ONLY"
        config rule
            edit 1
                set action permit
                set regexp "^$"
            next
        end
    next
end

You reference it from a route map rule with set match-as-path "LOCAL-ONLY". The regular expression ^$ matches an empty AS path, which is what a route this router originated looks like. Routes learned from an ISP carry that ISP’s AS number, so they do not match. Note the command is aspath-list, not as-path-list.

For an internet edge I prefer the prefix list. It states the exact prefixes you mean to announce, so a stray static route redistributed into BGP cannot leak out. The AS-path version says “anything I originate,” which is the better rule when the set of originated prefixes changes often.

After any outbound policy change, push it without dropping the session:

execute router clear bgp all soft out

Recipe 2: accept only a default route, and prefer ISP1 outbound

The problem. You rarely need a full table on a small edge, and you do not want an ISP’s mistake to land as 900,000 routes in your RIB. You also need to decide which ISP carries outbound traffic.

The fix. Accept only 0.0.0.0/0 from each ISP, and use local preference to choose between the two copies.

config router prefix-list
    edit "DEFAULT-ONLY"
        config rule
            edit 1
                set action permit
                set prefix 0.0.0.0 0.0.0.0
                unset ge
                unset le
            next
        end
    next
end

config router route-map
    edit "ISP1-IN"
        config rule
            edit 1
                set match-ip-address "DEFAULT-ONLY"
                set set-local-preference 200
            next
            edit 100
                set action deny
            next
        end
    next
    edit "ISP2-IN"
        config rule
            edit 1
                set match-ip-address "DEFAULT-ONLY"
                set set-local-preference 100
            next
            edit 100
                set action deny
            next
        end
    next
end

config router bgp
    config neighbor
        edit "203.0.113.1"
            set route-map-in "ISP1-IN"
        next
        edit "192.0.2.1"
            set route-map-in "ISP2-IN"
        next
    end
end

Higher local preference wins, so the default from ISP1 (200) beats the default from ISP2 (100). If ISP1 fails, its default disappears and ISP2’s takes over with no further action.

FortiOS also has a per-neighbor set weight. It does the same job on one box, but weight never leaves the FortiGate. If you later add a second FortiGate and run iBGP between them, local preference travels with the route and weight does not. I default to local preference for that reason.

Run execute router clear bgp all soft in after changing inbound policy, then check what you kept (Part 2 shows how to see the routes before and after the filter).

Recipe 3: pull inbound traffic towards ISP1

The problem. Local preference only controls the traffic you send. Traffic coming back to your /24 is decided by the rest of the internet, and right now both ISPs announce your prefix with the same path length.

The fix. Make the path through ISP2 look longer by prepending your own AS on the announcement to ISP2. This is the ISP2-OUT map from recipe 1, with a set-aspath action added.

config router route-map
    edit "ISP2-OUT"
        config rule
            edit 1
                set match-ip-address "OUR-PREFIXES"
                set set-aspath "64500 64500"
            next
            edit 100
                set action deny
            next
        end
    next
end

config router bgp
    config neighbor
        edit "192.0.2.1"
            set route-map-out "ISP2-OUT"
        next
    end
end

Prepending is a hint, not a command. Path length is only one step in the best path decision, and an ISP that ranks routes by customer relationship or by a community you did not set can ignore it entirely. If you see traffic still arriving through ISP2, read the ISP’s published community list before adding more prepends. I would also confirm the final path length on the ISP’s looking glass instead of counting on the FortiGate to tell you what it sent. Part 9 of the Deep Dive explains why inbound is the hard direction.

Recipe 4: advertise your /24 when you do not have it as a connected route

The problem. You add a network statement for 198.51.100.0/24 and the ISP never sees it. BGP only advertises a prefix it can find in the routing table, and a network statement needs an exact match. Your /24 is probably carved into several smaller internal subnets, so no single /24 route exists.

The fix. Anchor the prefix with a blackhole static route, then announce it.

config router static
    edit 100
        set dst 198.51.100.0 255.255.255.0
        set distance 254
        set blackhole enable
    next
end

config router bgp
    config network
        edit 1
            set prefix 198.51.100.0 255.255.255.0
        next
    end
end

The high distance means any real route for the same /24 wins. Longest prefix match means your internal /26s and /27s still beat the blackhole for everything you actually use, so the /24 only swallows traffic for addresses you have not routed anywhere.

Two things to know. Size the blackhole to the advertised prefix exactly, because a broader one drops traffic for real subnets inside it. And because the blackhole is always in the table, BGP keeps announcing the /24 even if everything behind the FortiGate is down. On a single edge that is acceptable. With several sites announcing parts of the same space, it is not.

If you only want to announce a specific connected network or a single loopback, the alternative is redistribute connected with a route map that matches a prefix list, rather than a network statement.

Recipe 5: originate a default route to a downstream router

The problem. The FortiGate also peers with an internal core router, say 10.0.0.2 in AS 64501, and that router needs a default route. You do not want to redistribute a static default or write a route map for it.

The fix. Tell the neighbor to originate one.

config router bgp
    config neighbor
        edit "10.0.0.2"
            set remote-as 64501
            set capability-default-originate enable
        next
    end
end

If you want the default to carry different attributes from the other routes you send that neighbor, a dedicated route map applies only to it:

config router bgp
    config neighbor
        edit "10.0.0.2"
            set default-originate-routemap "DEFAULT-ATTRS"
        next
    end
end

Fortinet’s own tip on this notes that the neighbor’s ordinary route-map-out does not touch the originated default. Only default-originate-routemap does, and the option appears in the CLI once capability-default-originate is on.

The thing I would test before relying on it is whether the default is advertised when no default exists in the FortiGate’s own routing table. Fortinet’s documentation does not say either way. If your design needs the downstream router to lose its default when both ISPs fail, check that behavior yourself rather than assuming it. And this is for internal or downstream peers. Never originate a default towards an ISP.

Recipe 6: authenticate the session with TCP-AO

The problem. Anyone who can reach TCP 179 on your edge and guess your peer’s address can try to open a session or reset an existing one. The session needs to authenticate every segment, not just the BGP messages inside it.

The fix. FortiOS 7.4.2 and later support the TCP Authentication Option (TCP-AO, RFC 5925) for BGP, on 7.4 and 7.6. It is the option I would use. You define a key chain, then bind it to the neighbor with auth-options.

config router key-chain
    edit "ISP1-AO"
        config key
            edit "1"
                set accept-lifetime 00:00:00 01 01 2024 2147483646
                set send-lifetime 00:00:00 01 01 2024 2147483646
                set key-string "<key-from-isp1>"
                set algorithm hmac-sha256
            next
        end
    next
end

config router bgp
    config neighbor
        edit "203.0.113.1"
            set auth-options "ISP1-AO"
        next
    end
end

The lifetime values are copied from Fortinet’s example. Check them against your firmware, and make sure the send and accept lifetimes overlap with your peer’s. The key string is a placeholder for the shared secret you agree with the ISP. Verify the result with:

diagnose sys tcp-auth-options

Fortinet documents three algorithms for TCP-AO: hmac-sha1, hmac-sha256 and cmac-aes128. Use the strongest one both ends support. If you are not sure what the peer can do, Fortinet’s knowledge base suggests hmac-sha1, because some peers support nothing else.

Two interop points matter on an ISP edge, and the first is the one most likely to cost you an afternoon. FortiOS assigns the TCP-AO send and receive key IDs itself and gives you no way to set them by hand. Cisco and Juniper peers expect matching IDs, so you need to agree the ID mapping with the ISP and confirm it on both sides. The second is that any mismatch in key, algorithm, lifetime or TCP options is silent. The peer drops your SYN packets, so the symptom is a session stuck in Connect or Idle, not an authentication error. If you see tcb is not available in the diagnostics, Fortinet’s tip is to recreate the key chain, reapply it to the neighbor and confirm the TCP control block is active.

Why not MD5. MD5 authentication (RFC 2385) is the older mechanism, and FortiOS still supports it with a single line on the neighbor:

config router bgp
    config neighbor
        edit "203.0.113.1"
            set password <key-from-isp1>
        next
    end
end

It works, and it is better than nothing. It also has real limits. It is built on an MD5 digest, a hash that is no longer considered strong. There is one static key per session with no defined way to roll it over, so changing the key means a coordinated change on both ends and a session reset. TCP-AO was written to replace it, and RFC 5925 obsoletes RFC 2385. Fortinet’s pages that I read do not rank the two against each other, so that comparison comes from the RFCs and not from a Fortinet statement.

My rule is TCP-AO where both ends support it, and MD5 only where the peer or the firmware leaves no choice. ISPs differ here, so ask which your provider supports before you assume. If you do end up on MD5, the failure mode looks the same as a blocked port. The MD5 signature rides in TCP option 19 on every segment, so a peer with a different key silently discards your packets, the session never leaves Connect or Active, and get router info bgp summary shows an up/down time of “never”. A packet capture on port 179 shows SYNs going out with no SYN-ACK back.

Putting it together

This is the full edge config with every recipe applied, except the downstream peer from recipe 5. Authentication is TCP-AO.

config router key-chain
    edit "ISP1-AO"
        config key
            edit "1"
                set accept-lifetime 00:00:00 01 01 2024 2147483646
                set send-lifetime 00:00:00 01 01 2024 2147483646
                set key-string "<key-from-isp1>"
                set algorithm hmac-sha256
            next
        end
    next
    edit "ISP2-AO"
        config key
            edit "1"
                set accept-lifetime 00:00:00 01 01 2024 2147483646
                set send-lifetime 00:00:00 01 01 2024 2147483646
                set key-string "<key-from-isp2>"
                set algorithm hmac-sha256
            next
        end
    next
end

config router prefix-list
    edit "OUR-PREFIXES"
        config rule
            edit 1
                set action permit
                set prefix 198.51.100.0 255.255.255.0
                unset ge
                unset le
            next
        end
    next
    edit "DEFAULT-ONLY"
        config rule
            edit 1
                set action permit
                set prefix 0.0.0.0 0.0.0.0
                unset ge
                unset le
            next
        end
    next
end

config router route-map
    edit "ISP1-IN"
        config rule
            edit 1
                set match-ip-address "DEFAULT-ONLY"
                set set-local-preference 200
            next
            edit 100
                set action deny
            next
        end
    next
    edit "ISP2-IN"
        config rule
            edit 1
                set match-ip-address "DEFAULT-ONLY"
                set set-local-preference 100
            next
            edit 100
                set action deny
            next
        end
    next
    edit "ISP1-OUT"
        config rule
            edit 1
                set match-ip-address "OUR-PREFIXES"
            next
            edit 100
                set action deny
            next
        end
    next
    edit "ISP2-OUT"
        config rule
            edit 1
                set match-ip-address "OUR-PREFIXES"
                set set-aspath "64500 64500"
            next
            edit 100
                set action deny
            next
        end
    next
end

config router static
    edit 100
        set dst 198.51.100.0 255.255.255.0
        set distance 254
        set blackhole enable
    next
end

config router bgp
    set as 64500
    set router-id 198.51.100.1
    config neighbor
        edit "203.0.113.1"
            set remote-as 64496
            set auth-options "ISP1-AO"
            set soft-reconfiguration enable
            set route-map-in "ISP1-IN"
            set route-map-out "ISP1-OUT"
        next
        edit "192.0.2.1"
            set remote-as 64497
            set auth-options "ISP2-AO"
            set soft-reconfiguration enable
            set route-map-in "ISP2-IN"
            set route-map-out "ISP2-OUT"
        next
    end
    config network
        edit 1
            set prefix 198.51.100.0 255.255.255.0
        next
    end
end

I added soft-reconfiguration so that Part 2 can show the routes as received, before the inbound route map touches them. With default-only inbound policy the memory cost is negligible.

Where this goes wrong

  • Filtering one direction and forgetting the other. An inbound filter protects you from the ISP. An outbound filter protects the internet from you. The transit problem in recipe 1 is an outbound failure, and it will not show up in your own routing table.
  • Pinning the router ID. Set router-id to a stable address. If you leave it unset, FortiOS picks one from the interfaces, and a change to that interface resets the sessions.
  • Policy changes that do nothing. Changes to a route map apply after a soft clear, not before. Use execute router clear bgp all soft in and execute router clear bgp all soft out, and wait for the advertisement interval if you do not clear.
  • Prepend that goes the wrong way. Prepend on the announcement to the backup ISP, to make it less attractive. Prepending towards the primary does the opposite of what you want.
  • Authentication that fails quietly. A TCP-AO or MD5 mismatch looks like a blocked port, not a bad password. Agree algorithm, key, lifetimes and key IDs with the ISP before the change window.

Next

Part 2 is the reference I use after making any of these changes: how to see what you receive and what you send, how to tell where a prefix went, and a triage list for a session that will not establish.