BGP Deep Dive Part 11: Convergence, Stability, and Scale

The first ten parts of this series covered what BGP carries and how it decides what to do with it. This part covers something different: what happens to a BGP deployment under stress, whether that stress is a flapping link, a router reboot, a link failure that needs to be noticed faster than BGP’s own timers notice it, or a topology that has outgrown “each router installs exactly one best path.”

Route flap dampening: mostly a historical answer now

The oldest mechanism here is route flap dampening, and this site already has a two-part series on it: Part 1 covers the flapping problem and the exponential-decay penalty model, Part 2 covers RFC 7454’s actual guidance and where dampening still has a legitimate role. It’s worth restating the conclusion here rather than the mechanics: RFC 2439 dampening, penalizing a route each time it flaps and suppressing it once the penalty crosses a threshold, made sense on the slower convergence, smaller routing tables, and less path-diverse internet of the mid-1990s. RFC 7454, published in 2015 as BCP 194, recommends against enabling classic dampening on customer or peering sessions in most modern deployments, because the same aggressive suppression that quiets a genuinely flapping route also punishes a route that legitimately changed twice for two unrelated, entirely normal reasons, and modern convergence times and route-table churn patterns make that false-positive cost higher than it used to be. It’s included here for completeness, not as new content, precisely because the two-part series already did the job properly.

Graceful restart: separating the control plane from the forwarding plane

RFC 4724 solves a specific, narrow problem: a router’s BGP process can restart, whether from a software upgrade, a process crash, or a route processor failover on hardware that supports it, without the router’s forwarding plane losing its existing forwarding state. Without graceful restart, a BGP session bouncing means every neighbor immediately withdraws every route learned from that peer, which for a core router with hundreds of thousands of received routes and dozens of peers is a genuinely disruptive event, propagating withdrawals and triggering best-path recalculation across a wide blast radius for what might be a five-second software restart.

The mechanism is a new capability, the Graceful Restart Capability, negotiated in the OPEN message using the same capability framework from Part 2 and Part 10. A router that supports it tells its peers, in effect, “if my session with you drops, don’t immediately flush the routes you learned from me; hold them for a restart-time window instead, since I might come back and I’ll tell you when I’m done relearning everything.” The restarting router’s forwarding plane keeps using its last-known-good forwarding table through the outage. When the session re-establishes, the restarting router re-advertises everything it has, and marks the end of that re-advertisement with an End-of-RIB marker, an UPDATE message carrying no NLRI at all, whose entire purpose is to say “that’s everything, you can now safely purge anything you were holding for me that I didn’t just re-send.” Only after End-of-RIB does the neighbor prune stale routes and recompute best path against the fresh set.

This is a genuinely different kind of resilience than the dampening/BFD/ADD-PATH mechanisms in the rest of this part: graceful restart makes a planned or crash restart of the control plane invisible to the forwarding plane, rather than making failure detection faster.

BFD: making BGP’s own timers irrelevant

Left to its own defaults, BGP notices a dead peer only when its hold timer expires, commonly 90 or 180 seconds depending on vendor defaults, or somewhat faster if the underlying transport itself notices the link is down and tears down the TCP session immediately. For a directly connected eBGP peer over a point-to-point link, a physical link-down event is usually fast and reliable. For anything running over a switched fabric, an SD-WAN overlay, or a peer that’s more than one hop away, “the link is physically fine but the far end has stopped forwarding” is a real failure mode that BGP’s own timers are slow to notice.

RFC 5882 defines BFD as a generic, protocol-independent failure detection mechanism, and BGP is one of many protocols that can subscribe to it: a lightweight, sub-second heartbeat exchanged directly between two endpoints, independent of and much faster than BGP’s own hold-timer mechanism. When BFD detects that its peer has stopped responding, it signals every protocol registered against that session, BGP among them, and BGP tears the session down immediately rather than waiting out its own hold timer. In practice this is the single most impactful convergence-speed improvement available for BGP: the difference between “this network in this series’s own SD-WAN resilience series notices a dead path in under a second” and “this network notices a dead path in ninety seconds” is BFD, not anything BGP itself does differently. It’s the same mechanism this site has already covered for OSPF adjacencies, tuned the same way, for the same reason: routing protocol hold timers were never designed to be a failure-detection mechanism on their own, they were designed to be a backstop for when something faster didn’t already catch it.

ADD-PATH: escaping the one-best-path-per-neighbor constraint

Every mechanism covered so far in this series, from the best path algorithm in Part 4 to route reflection in Part 6, shares an assumption baked deep into BGP’s original design: for a given prefix, a router picks exactly one best path and advertises only that one path to each neighbor. This is efficient, and it’s also a real limitation. A route reflector, per Part 6’s coverage, only reflects its single best path to its clients, meaning a client can lose visibility into perfectly good alternate paths that the reflector itself didn’t happen to prefer, sometimes causing real problems: suboptimal routing when the reflector’s best path isn’t the best path for a particular client’s position in the topology, or slower reconvergence when a path fails, since the client never learned about the backup path that existed all along and now has to wait for a fresh advertisement.

RFC 7911 (ADD-PATH) removes the one-path-per-neighbor constraint by extending the NLRI encoding with a 4-byte Path Identifier, letting a router distinguish between multiple paths to the same prefix and advertise more than one of them to the same neighbor. Like graceful restart and multiprotocol support, this is capability-negotiated in the OPEN message, and the Send/Receive field in that capability controls direction: a router can support receiving multiple paths, sending multiple paths, or both. The payoff is exactly the two problems route reflection with a single best path can create: a client with visibility into more than one available path can make its own locally-optimal choice rather than inheriting the reflector’s, and can fail over to an already-known alternate path immediately rather than waiting for a new advertisement after the primary fails.

None of these four mechanisms are exotic. Real production networks combine all four: BFD for fast failure detection, graceful restart for planned maintenance, ADD-PATH on route reflectors serving topologically diverse clients, and RFC 7454’s guidance on dampening followed rather than classic dampening left on. The closing part of this series turns to a different kind of stress: BGP under deliberate attack, from real-world hijack incidents through RPKI, ROV, and BGPsec.