BGP TCP-AO on FortiGate: Replacing the MD5 Password and Proving It on the Wire
The usual way to protect a BGP session is the MD5 password. TCP-AO, from RFC 5925, is its replacement, and FortiOS has supported it for BGP since 7.4.2. Setting it up takes a key chain and one line on each neighbor. What the documentation does not show is what you see when it works, what you see when it does not, and what happens to a session that is already up when you change the key. I tested all three on FortiOS 7.6.8, and this post is the evidence.
I covered the configuration from the documentation in recipe 6 of the FortiGate BGP cookbook, and the protocol theory in BGP Deep Dive Part 2. Here I am looking at the packets.
The lab
Two FortiGate-VMs on FortiOS 7.6.8: a hub that acts as a route reflector, and a spoke. They run iBGP in AS 65000 between loopbacks (lo-bgp, VRF 100), carried over an IPsec tunnel. The session is TCP port 179 from 10.0.1.2 to 10.0.1.11.
A fair question is why authenticate a BGP session that already runs inside an encrypted tunnel. The tunnel keeps outsiders away. It does nothing about something that is already inside the overlay. With one key per spoke, a spoke that is compromised cannot impersonate another spoke to the hub, and anything that reaches the BGP port in the overlay cannot inject or reset the session without that spoke’s key. That is my reasoning for the design, not something I tested.
The configuration
A key chain per peer, with one key, and the key bound to the neighbor with auth-options:
config router key-chain
edit "GT-BGP-SPOKE1"
config key
edit 1
set accept-lifetime <start> <duration>
set send-lifetime <start> <duration>
set key-string "<48 hex characters>"
set algorithm cmac-aes128
next
end
next
end
config router bgp
config neighbor
edit "10.0.1.11"
set remote-as 65000
set update-source "lo-bgp"
set auth-options "GT-BGP-SPOKE1"
next
end
end
The 7.6.8 algorithm choices are md5, hmac-sha1, hmac-sha256, hmac-sha384, hmac-sha512 and cmac-aes128. I used cmac-aes128. RFC 5926 makes cmac-aes128 and hmac-sha1 the two that every implementation must support, so those are the safe choices for a peer that is not a FortiGate. The key is a random 48 character hex string generated on a management host. It lives in a gitignored secrets file and never in the repository.
The spoke has the mirror image: a key chain called GT-BGP-RR with the same key, bound to its one neighbor, the hub.
To see what the unit has loaded:
diagnose sys tcp-auth-options
VFID=0 send-id=1 recv-id=1 flags=0x784 keylen=48 alg=2(aes128) addr=10.0.1.11
The output lists the peer, the algorithm, the key length and the key’s send and accept times. It never prints the key.
The first trap: a bound key does not touch a running session
I bound auth-options to a neighbor whose session was already Established. Nothing happened. The uptime carried on counting. diagnose sys tcp-auth-options listed the peer on both ends, and the session was still the one that had been set up before the key existed. It was almost certainly unauthenticated.
TCP-AO is negotiated on the SYN. An established connection keeps whatever it was set up with. You only know the key is in use after you reset the session:
execute router clear bgp ip 10.0.1.11
So “it shows up in diagnose sys tcp-auth-options and BGP is Established” proves nothing. It needs a reset and a capture.
On the wire
After the reset I captured on the tunnel interface:
diagnose sniffer packet H1-S1-ISP1 'tcp port 179' 6 6 l
Every segment carries TCP option kind 29, on the SYN, the SYN-ACK, the ACK, the keepalives, the OPEN, the NOTIFICATION and the FIN. Decoded:
| Bytes | Meaning |
|---|---|
1d | option kind 29 (TCP-AO) |
10 | length 16 |
01 | KeyID: the key the sender used |
01 | RNextKeyID: the key the sender wants the peer to use |
| 12 bytes | the MAC |
A keepalive is a 19 byte BGP message (0013 04), and with the option the TCP header grows to 48 bytes: 20 for the base header, 16 for TCP-AO and 12 for the padded timestamp option.
What a wrong key looks like
I changed the key on the spoke through FortiManager and left the hub on the old one. The established session kept exchanging keepalives with TCP-AO for more than 50 seconds. It was not affected, which is the same trap again from the other side. After a clear bgp, both ends sat in Connect with an Up/Down of never.
The capture of just the SYNs showed what was going on:
diagnose sniffer packet H1-S1-ISP1 'tcp port 179 and tcp[tcpflags] & tcp-syn != 0' 6 20 l
Four SYNs in total, from both ends, one of them a retransmit with the same sequence number. All of them carried option 29, and there was no SYN-ACK in either direction. A peer that cannot verify the MAC drops the SYN without replying. There is no reset and no error, only silence. On the sender it looks like a firewall blocking port 179, and my BGP cookbook says as much: a TCP-AO or MD5 mismatch looks like a blocked port. The troubleshooting order I would use: check the state (Connect, never), sniff the SYNs, see option 29 leaving and nothing coming back, then compare the keys and the algorithm on both ends.
It fails closed, which is what you want from an authentication feature. It is also the reason rotation needs care.
Rotating without dropping the session
Because a running session does not notice a key change, a rotation has to be done with overlapping lifetimes. The new key is installed on both ends and accepted well before anyone sends with it. Then, at a set time, both ends start sending with the new key and the old one stops. At every moment each end accepts whatever the other might send.
For the capture I used a third key rather than reusing an id. Reusing an id with new key material would mean the two ends disagree about the key until both installs finish, which is a few minutes, and a live session that depends on it could drop. A new id avoids that.
The schedule, in UTC:
- Key 3 accepted from 19:20:00, sending from 19:55:00.
- Key 2 stops sending at 19:55:00 and is accepted until 20:10:00.
- Both ends installed through FortiManager at 19:24 and 19:25, thirty minutes before the switch.
I ran the sniffer on the hub from 19:52:48 to 19:58:52. Before 19:55:00 every segment carries 1d 10 02 03: KeyID 2, RNextKeyID 3. The sender is still sending with key 2 and is already telling the peer that key 3 is ready to receive. From 19:55:30 every segment, in both directions, is 1d 10 03 03.
The last key 2 segment was at 19:54:56 and the first key 3 segment at 19:55:30. No BGP traffic fell in between, and with a 60 second keepalive interval it would not be expected to. The capture holds 30 packets. The TCP sequence and acknowledgement numbers run on unbroken across the switch, there is no SYN in the capture, and the keepalives arrive on schedule. The spoke’s session uptime read 5 days 1 hour 15 minutes at 19:56:08, counting from the same epoch as before the switch. No reset.
Cleaning up afterwards
Expired keys stay listed, and they are harmless, but the chain should not grow with every rotation. After the windows closed I deleted keys 1 and 2 on both ends, leaving key 3 as the only key, and the session did not reset on either install. One practical point: the delete of a key that no longer exists fails the whole stage in FortiManager, so a one-shot delete list has to come out of the configuration after it has been applied once.
One key per spoke
The key chain is per peer on purpose. A hub with ten spokes has ten neighbors, ten key chains and ten keys, and none of the keys is shared. The overhead on the hub is one neighbor, one key chain (two keys while rotating), one entry in whatever holds the keys and one rotation event per cycle for each spoke. I handle that with three things:
- The configuration is generated from the inventory, so a new spoke adds a loop iteration and not a hand-built block.
- Keys are created by a script, not typed.
- Rotation is scheduled, with overlapping lifetimes, so it needs no change window.
For a provider with many customers, a key per customer rather than per spoke is a defensible lower tier, as long as it is written down as a risk accepted: one compromised spoke then exposes that customer’s sessions and no one else’s.
Where the key lives matters as much as how often it changes. FortiManager masks key-string in its install preview, but a script object that pushes the key holds it in clear. On the device, show router key-chain prints an ENC string, and in my lab, with private data encryption off, that string is reversible. Treat it as the key itself, never paste that output anywhere, and turn on private data encryption wherever the key matters.
What I did not test
- Neighbor ranges and neighbor groups with TCP-AO. My hub uses explicit neighbors, one per spoke. A range with a shared key is the usual dial-up shape, and I have not tried it.
- Algorithms other than cmac-aes128. Fortinet’s knowledge base suggests HMAC-SHA1 if you are unsure what a third-party peer supports, and advises overlapping lifetimes. I did not run an interoperability test with another vendor.
- MD5 against TCP-AO on the same session, and the migration between the two.
- Table limits for key chains at large spoke counts.
Summary
- Binding a key to an established session changes nothing. Reset it, and capture to be sure.
- Option kind 29 on every segment is the proof:
1d 10 KeyID RNextKeyIDplus a 12 byte MAC. - A wrong key is a silent SYN drop:
Connect,never, no SYN-ACK. - Rotate with overlapping lifetimes, a new key id each time, and install both ends well before the send time. The session stays up across the switch.
- One key per spoke, generated and rotated by automation, kept out of the repository.