OSPF Deep Dive Part 6: Authentication and Security Hardening

Every post in this series so far has assumed CORE-CSR, BRANCH-FGT, and EDGE-MX trust each other by default, which is exactly what an unauthenticated OSPF segment does. Anything that can put a correctly-formed hello packet onto the wire, area ID matching, hello and dead timers matching, becomes a neighbor. That includes a rogue device plugged into the same broadcast segment, and it includes a compromised host on a VLAN that was never supposed to have a router on it in the first place.

What’s actually at risk

Three failure modes make unauthenticated OSPF worth taking seriously rather than treating it as a secondary concern behind the interface ACLs already protecting a segment.

An attacker forming a full adjacency can inject Type 5 LSAs advertising routes that don’t exist, redirecting traffic for real prefixes toward a device the attacker controls, effectively a routing-layer man-in-the-middle that doesn’t require touching the actual destination at all. An attacker can also just originate a higher-priority hello and win a DR election on a segment where the real DR was assumed to be stable, disrupting the flooding topology for everyone on that segment. And even without forming an adjacency at all, a flood of malformed or mismatched hellos is enough to keep a router’s OSPF process busy processing and rejecting packets, a cheap denial-of-service against the control plane rather than the data plane.

None of this requires sophistication. It requires physical or logical access to a segment where OSPF is running unauthenticated, which in practice means most enterprise networks that never explicitly configured authentication, because none of the three vendors in this series enable it by default.

Plaintext and MD5 are both weaker than the name suggests

OSPFv2 supports three authentication types historically: null (none), plaintext (Type 1), and MD5 (Type 2, “cryptographic” in Cisco’s terminology though the algorithm itself is showing its age). Plaintext authentication puts the password directly in every hello packet, unencrypted, which stops the most casual rogue-adjacency scenario and stops essentially nothing else, since anyone capturing traffic on the segment reads the password directly off the wire.

MD5 is a real improvement, since the password itself never travels on the wire, only a keyed hash of the packet contents does. But MD5 as a hash algorithm has known collision weaknesses that make it a poor long-term choice for anything security-sensitive, and RFC 5709 exists specifically to replace it with HMAC-SHA (SHA-1, SHA-256, SHA-384, or SHA-512) using the same key-chain concept MD5 already introduced. All three vendors in this series have supported HMAC-SHA for OSPF for years at this point, Cisco since IOS 15.4(1)T, FortiOS since 7.0.1, and Junos more recently, 23.4R1 for key chains and 24.2R1 for the HMAC-SHA2 algorithms specifically. There’s very little reason to configure new MD5 authentication today instead of HMAC-SHA256 from the start.

Key chains on all three vendors

Cisco IOS-XE, HMAC-SHA256 via a key chain applied at the interface, since Cisco’s OSPF cryptographic authentication is interface-level only, not area-level:

key chain OSPF-KEYS
 key 1
  key-string MyStrongKeyHere
  cryptographic-algorithm hmac-sha-256
!
interface GigabitEthernet0/1
 ip ospf authentication key-chain OSPF-KEYS

FortiOS, using the same key-chain concept, configured under router key-chain and referenced from the OSPF interface settings:

config router key-chain
    edit "OSPF-KEYS"
        config key
            edit 1
                set accept-key ENC MyStrongKeyHere
                set algorithm hmac-sha256
            next
        end
    next
end
config router ospf
    config area
        edit 0.0.0.0
            config interface
                edit "port1"
                    set authentication message-digest
                    set keychain "OSPF-KEYS"
                next
            end
        next
    end
end

Junos, using its own authentication-key-chains construct under security, referenced from the OSPF interface stanza:

set security authentication-key-chains key-chain OSPF-KEYS key 1 secret MyStrongKeyHere
set security authentication-key-chains key-chain OSPF-KEYS key 1 algorithm hmac-sha2-256
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 authentication keychain OSPF-KEYS

Older MD5 authentication (still worth knowing since it’s what most existing deployments actually run) is configured without a key chain at all on any of the three: ip ospf message-digest-key 1 md5 MyStrongKeyHere on Cisco, set authentication message-digest-key 1 md5 MyStrongKeyHere inline under the FortiOS interface block, and set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 authentication md5 1 key MyStrongKeyHere on Junos.

Area-wide vs interface-level, and the mismatch trap

Cisco and Junos both allow authentication to be configured once at the area level as a default for every interface in that area, in addition to per-interface configuration that overrides it. FortiOS configures authentication per interface only, with no area-level default to fall back on. This isn’t a security weakness on FortiOS’s part, but it does change the operational discipline required: adding a new interface to an OSPF area on a FortiGate never inherits authentication automatically the way it might on a Cisco or Junos box configured with an area-level default, and forgetting the per-interface line on a newly added FortiGate interface produces exactly the same silent gap in coverage described at the top of this post.

A mismatched authentication configuration between two neighbors doesn’t produce a helpful error either. Like the hello parameter mismatches from Part 1, it produces no adjacency at all, with log messages that vary by vendor in how directly they point at authentication as the cause versus just reporting a generic hello rejection. Part 10’s troubleshooting methodology covers how to actually confirm authentication as the specific cause rather than assuming it once other causes are ruled out.

That closes the vendor-neutral half of this series. Parts 7 through 9 take everything from Parts 1 through 6 and go through each vendor’s actual implementation in turn, starting with Cisco, to find where their real-world defaults and quirks depart from the theory just covered.