FortiGate Management Loopback Unreachable: Why It Drops at Policy 0 and the One Policy That Fixes It

A loopback is meant to be the one address on a device that is always up. So it is a little rude when a FortiGate loopback turns out to be the one address you cannot reach.

This is what I hit while building a management plane in its own VRF, with a /32 loopback on every FortiGate as the address the NOC connects to. Ping and SSH to the physical management port worked. Ping and SSH to the loopback, from the same host, timed out. The routing was right. The loopback had allowaccess set. Nothing was obviously wrong, which is the worst kind of wrong.

Everything below was tested on FortiOS 7.6.8, on FortiGate-VMs, in my lab.

The setup

  • lo-mgmt: a loopback interface, 10.0.0.15/32, in VRF 20, allowaccess ping https ssh.
  • port1: 10.10.0.8/24, in VRF 20, allowaccess ping https ssh fgfm.
  • A jump host, 10.10.0.6, with a host route to 10.0.0.15/32 via the port1 address.
  • Local-in policies on the device: an allow entry for my NOC address group, and a deny-the-rest entry on port1.

From the jump host:

ping 10.10.0.8        # fine
ssh admin@10.10.0.8   # fine
ping 10.0.0.15        # 100% packet loss
ssh admin@10.0.0.15   # connection timed out

The packets are arriving. The route is correct. Something on the FortiGate is dropping them.

What the FortiGate says

Flow debug is the right first tool:

diagnose debug reset
diagnose debug flow filter addr 10.0.0.15
diagnose debug flow filter proto 1
diagnose debug flow show function-name enable
diagnose debug flow trace start 10
diagnose debug enable

Then a ping from the jump host:

received a packet(proto=1, 10.10.0.6:3->10.0.0.15:2048) ... from port1. type=8
allocate a new session-00000787
find a route: flag=80000000 gw-0.0.0.0 via root
gnum-100004, use addr/intf hash, len=1
fw_local_in_handler line=630 msg="iprope_in_check() check failed on policy 0, drop"

Policy 0 is the hidden default deny. The packet was dropped by a policy I did not write, because it matched nothing.

Two things in that trace are worth noticing. The packet arrived on port1, and it was looked up in policy group 100004. That group is the firewall policy table, and diagnose firewall iprope list 100004 shows nothing in it except the catch-all action=drop ... last-deny entry.

Why the local-in policy didn’t help

My first guess was a local-in policy. I already had one allowing the NOC group to this device’s addresses, and I extended it to cover 10.0.0.15. It compiled cleanly into the right group, with the right sources, destinations and services. The ping still failed.

The iprope dump explains why:

diagnose firewall iprope list | grep -B14 -A6 10.0.0.15

Two kinds of entry matter here. The implicit entries that allowaccess creates for the loopback are keyed to the loopback as the ingress zone:

group=0010000f ... zone(1): 15 -> zone(1): 0
source(1): 10.10.0.6-10.10.0.6 ... dest(1): 10.0.0.15-10.0.0.15 ... tcp 443

Zone 15 is lo-mgmt (diagnose ip address list gives you the zone numbers). The entries for the physical port are keyed to its own zone:

group=0010000f ... zone(1): 3 -> zone(1): 0
dest(1): 10.10.0.8-10.10.0.8

Zone 3 is port1. My local-in policies, which are written for intf port1, sit in that same zone 3 group as the port’s own entries.

So the packet arrives on port1, but the loopback’s permissions are keyed to the loopback. They never get a chance to apply.

The fix: one ordinary firewall policy

Fortinet’s own knowledge base says the same thing. The community article “Technical Tip: Access the FortiGate via the Loopback Interface” describes traffic arriving on another port being dropped at local-in (policy 0) even with allowaccess on the loopback, and adds a firewall policy from the ingress port to the loopback. A second article, “Understanding policyid 0 drops for local-in policy traffic”, describes the hidden default deny that does the dropping. Neither shows a passing test, so here is mine.

config firewall policy
    edit 900
        set name "GT-NOC-TO-LO-MGMT"
        set srcintf "port1"
        set dstintf "lo-mgmt"
        set action accept
        set srcaddr "GT-NOC-MGMT"
        set dstaddr "GT-SELF-LOMGMT"
        set schedule "always"
        set service "GT-MGMT-SVC"
        set logtraffic all
    next
end

The source interface is where the traffic comes in. The destination interface is the loopback. The source address is the NOC group, the destination is the loopback’s own address, and the service is a group of HTTPS, SSH and ping. No VIP, no NAT.

After installing it:

ping -c 3 10.0.0.15     # 0% packet loss, about 0.5 ms
ssh admin@10.0.0.15     # banner, login, hostname

The iprope dump now shows what changed. Group 100004 holds a new entry:

policy index=900 action=accept ... zone(1): 3 -> zone(1): 13

Zone 3 is port1, zone 13 is lo-mgmt on that device, and the sources, destination and services are the ones from the policy. In the flow trace, len=1 becomes len=2, the policy 0 line disappears, and the session is confirmed:

gnum-100004, use addr/intf hash, len=2
ip_session_confirm_final ... hook=1

The reply then shows up as from local. One detail to be aware of: the trace does not print Allowed by Policy-900. Traffic to an address the FortiGate owns does not print that line. The accept shows as the longer group and the session being confirmed.

It takes two gates

The firewall policy is the first gate. The loopback’s own allowaccess is the second. I proved this on a second FortiGate, where the loopback had allowaccess https ssh and no ping:

  • With policy 900 in place permitting ICMP, SSH to the loopback worked and ping did not.
  • Adding ping to the loopback’s allowaccess made ping work immediately: 0% loss.
  • Taking ping out again put it back to 100% loss.

So the model is: the packet needs a firewall policy from the ingress interface to the loopback, and then it needs the loopback’s allowaccess for the service. A policy alone does not open services the loopback does not allow, and allowaccess alone never opens anything from another interface.

What about local-in policies?

My earlier, port-based local-in policies never saw this traffic. On one of the test devices I had a deny entry covering all sources and all destinations on port1, live the whole time, and SSH to the loopback worked regardless.

What does catch it is a local-in policy whose interface is the loopback. I added a temporary deny with intf "lo-mgmt", all sources, all destinations, all services. The existing SSH session to the loopback was closed and new connections timed out. So local-in policy can control loopback traffic, but only when written against the loopback as the interface.

It fails closed

An allow rule that also lets everyone in would not be much use. I added a second address to the jump host that was not in the NOC group:

sudo ip addr add 10.10.0.99/24 dev eth0

From that address:

  • ping -I 10.10.0.99 10.0.0.15: 100% loss. No firewall policy matches.
  • ping -I 10.10.0.99 10.10.0.8: 100% loss. The local-in deny on the port catches it.
  • ssh -b 10.10.0.99 admin@10.10.0.8: timed out, even though the admin trusthost covers the whole /24. The local-in deny stopped it before the trusthost was consulted.

Straight afterwards, from the real jump host address, everything worked. The firewall policy table’s implicit deny gives deny-by-default on the loopback path, and the source address group is the allow-list. That is a better security story than it first appears: the policy-0 drop is the thing keeping the loopback closed.

The same thing happens on an overlay tunnel

In a managed service design the loopback is reached over the overlay, not over a physical port, so I repeated the test with an IPsec tunnel as the ingress. Two FortiGates, a certificate-authenticated IKEv2 tunnel between them, a loopback on each end with allowaccess ping.

One thing came up before I could test anything: with no firewall policy referencing the tunnel interface at all, IKE never started.

ignoring request to establish IPsec SA, no policy configured

A sniffer on the hub saw no IKE packets. A tunnel needs at least one firewall policy that references it, even if it only ever carries traffic between the two devices’ own loopbacks.

Stage one used a policy from the tunnel to the loopback that allowed SSH only. The tunnel came up, and ping to the peer’s loopback failed. On the receiving FortiGate:

received a packet(proto=1, 10.0.1.2:8->10.0.1.11:2048) tun_id=10.100.0.1 from S1-H1-ISP1
gnum-100004 ... len=1
fw_local_in_handler line=630 msg="iprope_in_check() check failed on policy 0, drop"

The same signature as the physical port. The policy existed, but it did not match ICMP, so it fell through to the implicit deny.

Stage two changed the policy’s service to the full management group. Ping now reached the loopback, the trace showed len=2 and a confirmed session, and the reply went back out through the tunnel. A tunnel interface is just another ingress.

Housekeeping: debug is global

One lesson from the console, not the config. diagnose debug enable is system-wide, not per session. While I was tracing over SSH, the same flow lines were scrolling on the VM console, including output from both of my test rounds. Run diagnose debug disable and diagnose debug reset when you are done, or every open session gets flooded.

The summary

If a FortiGate loopback is reachable from the interface it lives behind but not from another one:

  1. Packets to a loopback IP arriving on another interface need a firewall policy from the ingress interface to the loopback. Without one, flow debug shows iprope_in_check() check failed on policy 0, drop.
  2. The loopback’s allowaccess is a second gate. It applies after the policy, per service.
  3. Local-in policies written for the ingress port do not see this traffic. One written for the loopback interface does.
  4. The source address group on the firewall policy is the allow-list, and everything else is dropped by the implicit deny.
  5. The same applies when the ingress is an IPsec tunnel, and the tunnel needs a policy referencing it before IKE will start at all.

A caveat on how I got to the model: the two-stage explanation is inferred from the zone keys in the iprope listings and the flow traces, not from Fortinet documentation, and it fits every observation I made. I tested on 7.6.8 only, so check it on your own build before relying on it.

For the design this came from, the answer is a management loopback per device in its own VRF, with one firewall policy per ingress interface, each with a tight source group. That is easy to template and easy to audit.