iperf3 on FortiOS 7.6.8: Getting the Built-In Server Working Inside a VRF
I wanted to load an IPsec tunnel to see what it does to BFD, without building extra test VMs. FortiOS has iperf3 built in, so that looked like a free test harness. The first attempt connected to nothing and printed nothing useful. This post is how I got it working inside a VRF, and what the failure looked like at each stage.
The setup: two FortiGate-VMs on FortiOS 7.6.8, a hub and a spoke, joined by one IPsec tunnel. The tunnel and the BGP loopbacks (lo-bgp) are in VRF 100. The test traffic has to go loopback to loopback so it matches the firewall policy that already permits the overlay.
What is built in
The feature lives under diagnose traffictest. On 7.6.8 these are the settings and the defaults I found:
diagnose traffictest show
server-intf: port1
client-intf: port2
port: 162
proto: TCP
run is the iperf3 client. run -s starts a server. Server mode arrived in FortiOS 7.4.8; on older versions it fails with iperf3: parameter error - cannot be both server and client. Fortinet’s own tips on configuring a FortiGate as an iperf server show the server interface with allowaccess speed-test, which matters later in this post.
Every run prints a banner saying the function is designed for connection verification, not bandwidth measurement. Treat the numbers below as a way to compare paths and settings, not as a benchmark.
Problem 1: “Network is unreachable”
From the hub, the obvious command fails:
diagnose traffictest run -c 10.0.1.11 -B 10.0.1.2 -t 10
traffictest behaved as if it runs in the default VRF (0). The target, the spoke’s lo-bgp address, is in VRF 100, so there was no route. Setting the interface the test uses fixes the routing:
diagnose traffictest port 5201
diagnose traffictest client-intf lo-bgp
On the spoke, the server side needs the matching setting:
diagnose traffictest port 5201
diagnose traffictest server-intf lo-bgp
diagnose traffictest run -s
Check diagnose traffictest show on both devices before every run. The settings are per device, and I found one device back at its defaults between sessions.
Problem 2: connection timed out, no log, no RST
With routing sorted, the client printed Connection timed out. A sniffer on the spoke showed the SYN arriving on the tunnel and nothing leaving:
diagnose sniffer packet any 'tcp and port 5201' 4 20 l
S1-H1-ISP1 in 10.0.1.2.7463 -> 10.0.1.11.5201: syn 640126127
S1-H1-ISP1 in 10.0.1.2.7463 -> 10.0.1.11.5201: syn 640126127
...
The retransmits came at roughly 1, 2, 4, 8, 17 and 33 seconds. The server was listening: diagnose sys tcpsock | grep 5201 showed 10.0.1.11:5201 ... state=listen process=iperf3. The flow debug said where the packet died:
fw_local_in_handler line=630 msg="iprope_in_check() check failed on policy 0, drop"
Policy 0 is the implicit deny. This is the same drop I wrote about in FortiGate Management Loopback Unreachable: Why It Drops at Policy 0, so I was sure the answer was the policy. It was not that simple. diagnose firewall iprope list 100004 showed policy 901 already allowed TCP 5201 from the hub’s loopback to the spoke’s loopback, and the SYN still hit policy 0.
Three things that were not the cause
A local-in-policy. I added a temporary config firewall local-in-policy accepting TCP 5201 on the tunnel interface. It made no difference, and it never appeared in the iprope group that makes the decision, so it was never consulted for this traffic.
TCP in general. I bounced the BGP session to force a new connection to TCP 179 through the same policy. The fresh SYN passed straight through. A ping to the loopback passed as well. So new TCP sessions to the loopback are fine, and only 5201 is dropped.
The VRF. I moved the test out of the overlay completely: the hub’s port2 to the spoke’s port2 across the underlay, both in VRF 0, no tunnel. Same result, a SYN that arrives and is never answered. That ruled out VRF 100.
The cause: allowaccess speed-test
FortiOS gates the built-in iperf3 server with the speed-test access option, in the same way it gates HTTPS or SSH on an interface. If the interface that owns the destination address does not allow it, the connection is dropped. On port2, allowaccess was empty (the underlay port is hardened). The CLI lists the option:
config system interface
edit port2
set allowaccess ?
...
speed-test Speed test access.
With set allowaccess speed-test on the interface that owns the destination address, the same underlay test connected immediately:
[ ID] Interval Transfer Bitrate Retr
[ 6] 0.00-10.00 sec 2.55 GBytes 2.19 Gbits/sec 761 sender
For the overlay, the setting goes on the destination loopback, not on the tunnel interface:
config system interface
edit lo-bgp
append allowaccess speed-test
next
end
And the policy is still required
allowaccess is one control. The firewall policy is the other. With speed-test set and the service removed from policy 901, the client hung again. With the service restored, it connected. In the overlay case you need both:
allowaccess speed-teston the interface that owns the destination address.- A firewall policy permitting TCP 5201 from the tunnel to that loopback.
If either is missing, the symptom is identical: a silent drop, SYN retransmits, no RST, no log. The service object is a plain custom service:
config firewall service custom
edit "GT-IPERF-TEST"
set tcp-portrange 5201
next
end
The UDP trap
I also wanted a flood that does not back off, so I tried UDP. In traffictest the protocol is numeric:
diagnose traffictest proto 1
0 is TCP and 1 is UDP. The client then failed with:
iperf3: error - unable to read from stream socket: Resource temporarily unavailable
iperf3 opens a TCP control connection first, then sends the data as UDP. The control connection to 5201 worked, so the client got as far as “Connecting to host”. The sniffer on the spoke saw a single 4-byte UDP packet, the iperf3 UDP handshake, arrive and get no answer. My service object was TCP only. Adding the UDP port fixed it:
config firewall service custom
edit "GT-IPERF-TEST"
set udp-portrange 5201
next
end
The flood then ran at 100 Mbit/s with the receiver reporting loss, which is what I wanted.
What it measured
| Path | Rate | Notes |
|---|---|---|
| Underlay, VRF 0, one stream | 2.19 Gbit/s | 761 retransmits in 10 s |
| Tunnel, VRF 100, one stream, hub to spoke | about 540 Mbit/s | 5 retransmits in 60 s |
| Tunnel, two streams | 669 Mbit/s total | about 335 Mbit/s each |
| Tunnel, one stream, spoke to hub | 547 Mbit/s | 30 s run |
The tunnel is IKEv2 with AES-256-GCM, with no hardware offload, because these are VMs. During the transfer the receiving device’s first CPU core sat at 93% system with 2% idle in both directions, while the sender stayed lower. The second stream added only about 20%, which fits all of one tunnel’s traffic being decrypted on one core. I did not prove that, so I would call the ceiling a receiver-CPU limit on a VM, not a tunnel limit.
The working recipe
Spoke (the server):
config system interface
edit lo-bgp
append allowaccess speed-test
next
end
diagnose traffictest port 5201
diagnose traffictest server-intf lo-bgp
diagnose traffictest proto 0
diagnose traffictest run -s
Hub (the client):
diagnose traffictest port 5201
diagnose traffictest client-intf lo-bgp
diagnose traffictest proto 0
diagnose traffictest run -c 10.0.1.11 -B 10.0.1.2 -t 10
The firewall policy from the tunnel to lo-bgp must include TCP 5201, and also UDP 5201 if you use UDP. Run diagnose traffictest show on both devices first.
Clean up afterwards
allowaccess speed-test opens a service, and on a device with a hardened interface that is exactly the kind of thing that should not be left behind. In my templates it is a variable that defaults to false and sits next to a second variable that adds the firewall service. Both switch on for a test window and back off afterwards, so no one has to remember a command typed by hand.
What I used it for
The point was to put a loaded tunnel next to BFD and watch what happens. How I shaped the queue on the Proxmox side is in my netem post. The short version of what the load showed: bulk TCP does not take BFD down, because TCP backs off before the queue stays full, but a flood of UDP into a smaller link does.