OSPF Deep Dive Part 12: A Multi-Vendor Outage, Start to Finish

Every fault in this series so far has been introduced on its own, in isolation, to teach one specific mechanism. This post runs the whole thing the other way: a single vague complaint, every adjacency reporting healthy, and the actual cause buried in a detail that Part 3 already warned about and Part 7 already showed printing a warning that’s easy to ignore.

The complaint

The backup job that copies data from Area 1’s server subnet to a system reachable through EDGE-MX’s Area 2 has started taking noticeably longer some nights and not others. Nothing paged. No interface is down. Nobody would call this an outage from a monitoring dashboard’s point of view, which is exactly why it’s a better closing scenario than a hard failure: hard failures are what Parts 1 and 10 already trained for. This is the harder case, where the network appears entirely healthy and isn’t.

First pass: confirm the adjacencies, per Part 10’s own advice

Part 10’s methodology says start with neighbor state before anything else. On all three routers, every neighbor reports Full:

CORE-CSR#  show ip ospf neighbor
10.0.0.2    1   FULL/DR    00:00:33   10.0.0.6   Gi0/1
10.0.0.3    1   FULL/BDR   00:00:38   10.0.0.10  Gi0/2

BRANCH-FGT # get router info ospf neighbor
10.0.0.1    1   Full/DR    00:00:31   10.0.0.5   wan1
10.0.0.3    1   Full/BDR   00:00:35   10.0.0.9   wan2

EDGE-MX> show ospf neighbor
10.0.0.5   ge-0/0/1.0   Full   10.0.0.1   1   34
10.0.0.6   ge-0/0/2.0   Full   10.0.0.2   1   36

Nothing here points anywhere. Per Part 10’s table, Full-with-wrong-behavior is a different investigation from a stuck-state investigation, so the neighbor table’s job is done: it confirms this isn’t a Part 1/7/8/9 adjacency problem at all.

The red herring

While checking every interface for good measure, an old point-to-point GRE tunnel between CORE-CSR and EDGE-MX turns up, a backup path from before the current three-area design, never fully decommissioned. Its neighbor is stuck at ExStart:

CORE-CSR# show ip ospf neighbor
10.0.0.3    1   EXSTART/  -   -   10.0.0.14  Tunnel14

Part 1’s diagnosis applies directly: ExStart almost always means an MTU mismatch. Checking confirms it, the tunnel interface on CORE-CSR is still set to the old default MTU while EDGE-MX’s side was changed months ago during unrelated cleanup. It’s a real fault. Fixing it (matching the tunnel MTU on both ends) is worth doing. It also has nothing to do with the backup job’s slowness, because this tunnel isn’t in the current SPF tree for any traffic between Area 1 and Area 2 at all, it’s an orphaned backup path that was never actually carrying anything. This is worth sitting with as its own lesson: finding a genuine, fixable problem during an investigation is not the same as finding the problem that matches the reported symptom, and it’s tempting to declare victory here anyway because the fix is clean and the diagnosis was textbook.

Back to the actual path, and the LSDB checks out too

Per Part 10’s remaining steps: the LSDB has the expected LSAs, Area 1’s Type 3 summary for 10.10.10.0/24 is present at EDGE-MX, the virtual link from Part 4 is up, and show ospf route / show ip route ospf / get router info routing-table ospf all show a route to the expected destination on all three routers. Nothing is missing. Nothing is filtered. This rules out Part 2’s area-type filtering and Part 4’s summarization and redistribution entirely.

What’s left, per Part 10’s own closing point, is cost. Checking interface cost on all three routers (the same show ip ospf interface, get router info ospf interface, and show ospf interface detail commands from Part 11, just read for the cost field this time instead of the state field) turns up the actual mismatch. CORE-CSR and BRANCH-FGT both show a cost of 10 on the link that was upgraded to 10 Gbps between them last month, consistent with the auto-cost reference-bandwidth 100000 change that shipped with that upgrade, exactly the kind of change Part 7 described printing a one-time local warning about keeping the value consistent domain-wide. EDGE-MX still shows a cost of 1 on every interface, still running the default 100 Mbps reference bandwidth, because the change ticket for the 10G upgrade only listed “the Cisco and Fortinet boxes” and Junos’s half of the change was pushed to a follow-up ticket that never got scheduled.

Why this produces slowness instead of an outage

This is the payoff of Part 3’s warning landing here, at the end of the series, instead of staying abstract. CORE-CSR and BRANCH-FGT now agree with each other: their view of the domain treats the upgraded 10G link as ten times cheaper than a 1 Gbps link, which is correct relative to each other. EDGE-MX never got the memo, so its SPF tree still treats every link as roughly equal cost regardless of actual bandwidth. All three routers still calculate a loop-free shortest-path tree, because SPF doesn’t require domain-wide reference-bandwidth agreement to produce a mathematically valid result, it only requires it to produce a sensible one. EDGE-MX ends up preferring a path back toward Area 1 that CORE-CSR and BRANCH-FGT no longer consider the cheap one, and the backup traffic ends up asymmetric, one direction over the upgraded 10G path, the other direction over a leg that’s still, from EDGE-MX’s perspective, “just as good” as any other hop. Asymmetric paths aren’t inherently broken, but two paths with different actual capacity carrying two directions of the same conversation is precisely the kind of thing that shows up as inconsistent latency on a bulk transfer, worse on the nights when something else is also competing for the slower leg’s bandwidth, and invisible every other time, which is why the complaint was intermittent rather than constant.

The fix is the one-line change Part 9 already showed how to write: set protocols ospf reference-bandwidth 100g on EDGE-MX, matching the value CORE-CSR and BRANCH-FGT are already running. No adjacency resets, no area redesign, no redistribution change. Three routers agreeing on one number that has no protocol-level check behind it at all, which is the exact gap Part 3 opened this series’ cost discussion with and Part 7 confirmed by name: the warning fires once, locally, at configuration time, on whichever router the change was actually applied to, and never again.

That’s the series. Twelve parts, one lab, and the closing fault was never a broken adjacency at all, just three routers that each did exactly what they were told, individually correct, collectively inconsistent, and invisible to every check except the one that compares a plain number across vendors instead of inside any one of them.