Arista (VMware) SD-WAN Deep Dive — Part 4: Topology Walkthroughs — MPLS-only meets Internet-only Across Continents

Series map. Part 4 of five.

1. Components, Gateways, and the Three Planes
2. Routing — Overlay, Underlay, BGP, and the Gateway as Route Reflector
3. The Data Plane — VCMP, DMPO, and Per-Flow Steering
4. Topology Walkthroughs — MPLS-only meets Internet-only Across Continents (this post)
5. Best Practice, Failure Modes, and a Design Checklist

This is the post the rest of the series exists to support. We take the GlobalCo scenario from Part 1, the routing model from Part 2, and the data-plane mechanics from Part 3, and we walk through five concrete flows — five flows that between them exercise every interesting case in the architecture.

The setup, restated

BritNet (UK national ISP) operates two Cloud Gateways, both in the UK: VCG-LON1 (Slough) and VCG-MAN1 (Manchester). Both are Internet-underlay only. Public IPv4 addresses, reachable from the public Internet, not attached to any MPLS VRF.

GlobalCo sites:

Site	Underlay 1	Underlay 2
HQ London	MPLS (BritNet)	DIA (BritNet)
Bristol	MPLS (BritNet)	Broadband
Newcastle	Broadband	—
New York	MPLS (international)	DIA (US local)
Chicago	MPLS only	—
Shanghai	DIA only	—

Each site has a VCE: VCE-HQ, VCE-BRS, VCE-NCL, VCE-NYC, VCE-CHI, VCE-SHA.

The MPLS network is a single L3VPN VRF, “GlobalCo-MPLS”, that includes HQ, Bristol, New York, and Chicago. The other sites are not on MPLS.

We’ll walk five flows:

1. Newcastle → HQ. Pure Internet-underlay Direct.
2. Bristol → HQ. Multi-underlay Direct with DMPO making the call.
3. Chicago → UK Cloud Gateway. The wall — and the first thing to fix.
4. Newcastle → Chicago via Partner Gateway. Cross-underlay relay.
5. Chicago → Shanghai via Partner Gateway. Two MPLS-only / Internet-only Edges with neither sharing an underlay — the headline.

For flows 4 and 5 we assume BritNet has stood up a Partner Gateway in Slough. It sits in BritNet’s MPLS PoP. It has a BGP peering into the GlobalCo-MPLS VRF on its MPLS-facing interface, and a public IPv4 on its Internet-facing interface. Call it PGW-SLOUGH.

Flow 1: Newcastle → HQ — pure Internet Direct

User in Newcastle opens a Sharepoint page hosted at HQ. LAN dst is 10.1.0.50 (a server in HQ).

Routing — what VCE-NCL knows:

• It has VCMP control tunnels up to VCG-LON1 and VCG-MAN1 via its broadband (the only underlay it has).
• The Gateways have reflected to it the overlay routes from every other Edge. So VCE-NCL has:
  • 10.1.0.0/16 via VCE-HQ (overlay, via Gateway)
  • Direct tunnel to VCE-HQ does not yet exist.

First packet:

LAN-side packet (src=10.4.0.10 dst=10.1.0.50) hits VCE-NCL. Business Policy classifies it as office productivity, normal priority, link selection “any underlay, prefer best DMPO”.

• Cloud VPN mode for this site is set to Direct-preferred, fall back to Gateway. Since this is the first packet to VCE-HQ, there is no Direct tunnel yet. The Edge sends the packet via the Gateway as the safety path, and in parallel initiates Direct tunnel setup to VCE-HQ.
• Direct tunnel setup: VCE-NCL learns from the Gateway that VCE-HQ has two public underlay endpoints (DIA public IP) and one MPLS underlay (private, not reachable from VCE-NCL’s broadband). The Gateway tells both Edges about each other’s reachable underlay set. VCE-NCL builds a VCMP tunnel from its broadband to VCE-HQ’s DIA. Mutual key exchange, handshake, tunnel is up in tens of milliseconds.

Steady state:

After a couple of round trips, the Direct tunnel is up. DMPO measures it for a few seconds (passive piggybacking on the already-flowing traffic plus initial synthetic probes). Once it’s confirmed healthy, the flow migrates from Gateway-relayed to Direct mid-flow. The LAN session sees a flow ID continuation, no reset, no observable hiccup.

Now every subsequent packet:

Newcastle LAN
  → VCE-NCL
  → VCMP/UDP-2426 over broadband
  → public Internet
  → VCE-HQ's DIA interface
  → VCE-HQ
  → HQ LAN

No Gateway in the data path. Two Gateways still in the control path — VCE-NCL and VCE-HQ both still receive route updates via VCG-LON1 and VCG-MAN1.

Failure case: if VCE-NCL’s broadband goes down, the Direct tunnel dies. Without a second underlay, the flow goes away. There is no magic — DMPO can only switch among underlays the Edge has.

Flow 2: Bristol → HQ — multi-underlay Direct

Same destination, different source.

Routing — what VCE-BRS knows:

VCE-BRS sits on MPLS + Broadband. After the same Gateway-mediated bootstrap as Newcastle, but with more underlay options, VCE-BRS ends up with four Direct VCMP tunnels to VCE-HQ:

#	Bristol underlay	HQ underlay
1	MPLS	MPLS
2	MPLS	DIA
3	Broadband	MPLS
4	Broadband	DIA

Wait — MPLS → DIA and Broadband → MPLS? Aren’t those impossible? They are intra-Edge underlays; tunnel 2 means “VCE-BRS sends VCMP out its MPLS interface, the packet traverses the MPLS network to a PE near HQ, then somehow reaches VCE-HQ’s DIA interface”. This isn’t what happens. VCMP tunnels are between reachable underlay-pair endpoints. So:

• Tunnel 1 (MPLS-MPLS): both Edges sit on the same MPLS VRF, so they have direct underlay reachability private-IP to private-IP across MPLS.
• Tunnel 4 (Broadband-DIA): both Edges have public IPs reachable from each other’s public-Internet underlay.
• Tunnels 2 and 3: don’t get built. There’s no IP-level path from Bristol’s MPLS-attached interface to HQ’s public DIA IP, and vice versa, because the MPLS VRF is private and doesn’t terminate publicly.

So Bristol ↔ HQ ends up with two Direct tunnels: MPLS-MPLS and Public-Public.

Per-flow choice:

For a Sharepoint flow with policy “prefer any in-SLA, MPLS-preferred for traffic to HQ”:

• DMPO state: MPLS tunnel = 8ms RTT, 1ms jitter, 0% loss. Public tunnel = 14ms RTT, 3ms jitter, 0.05% loss.
• Both in SLA. MPLS-preferred policy → MPLS wins.

For a Teams audio flow with policy “voice on best-jitter underlay”:

• MPLS has lower jitter — MPLS wins.

For a 50 GB file backup flow with policy “prefer broadband to spare MPLS”:

• Public wins. The MPLS link is not loaded by this flow.

This is the per-flow steering payoff: the same Edge pair, the same destination, but three different flows on three different physical paths — selected at the first packet, maintained dynamically, with mid-flow remediation if anything degrades. All four flows are inside the same logical Direct tunnel pair from a control-plane perspective.

Flow 3: Chicago → UK Cloud Gateway — the wall

Now the interesting one. We try to do for Chicago what we did for Newcastle.

Underlay reality:

• VCE-CHI has only an MPLS interface. It is BGP-peered with the local PE inside the GlobalCo-MPLS VRF. It can reach Bristol, HQ, and New York via MPLS. It can reach no public Internet endpoint.
• The UK Cloud Gateways VCG-LON1 and VCG-MAN1 have only public Internet IPs. They are not in any MPLS VRF.

Can VCE-CHI build a control tunnel to either Gateway? No. There is no IP-level path between Chicago’s MPLS interface and a UK Gateway’s public IP. The MPLS VRF terminates inside the GlobalCo customer’s PE-attached endpoints; the Gateway is not such an endpoint.

Result: VCE-CHI never registers with any Gateway. It can ping its MPLS neighbours all day. It can BGP with the PE. But from the Arista SD-WAN architecture’s point of view, VCE-CHI is not part of the overlay. Other Edges have no overlay route to Chicago. The Orchestrator shows VCE-CHI as offline.

This is the wall. It’s not a bug. It’s the architectural consequence of choosing “Cloud Gateway only, in the UK, Internet underlay only” when you have MPLS-only Edges in your fleet. The fix is to give VCE-CHI a path to something the Gateway can reach.

Option A: give Chicago Internet underlay. Stand up a DIA at the Chicago office. It’s a US$/£/€ decision. Works fine.

Option B: deploy a Partner Gateway inside the GlobalCo-MPLS VRF, in the UK, so Chicago can reach it over MPLS.

For the rest of the post we go with Option B — because it’s where the architecture gets clever, and because the question this series exists to answer is exactly “how do we do this without sticking DIA at every site?”.

Adding the Partner Gateway

BritNet stands up PGW-SLOUGH, a Partner Gateway VM in their Slough PoP. Two interfaces:

• MPLS-facing interface: attached to the GlobalCo-MPLS VRF on the local BritNet PE. eBGP peering with the PE inside the VRF. From the MPLS network’s perspective, PGW-SLOUGH is a CE belonging to GlobalCo with a single very large set of prefixes behind it (everything in the GlobalCo overlay).
• Internet-facing interface: a public IPv4, reachable from the public Internet just like a Cloud Gateway would be.

What changes in the routing tables:

On the MPLS side (PE-import):

• The PE receives, from PGW-SLOUGH, prefixes for every site in GlobalCo’s overlay — including Newcastle (which has no MPLS), Shanghai (which has no MPLS), and HQ/Bristol/NYC (which are reachable via the MPLS network anyway).
• The MPLS network redistributes those into the GlobalCo-MPLS VRF.
• VCE-CHI, peering BGP with its local PE, now sees routes for every GlobalCo overlay prefix via the VRF, with PGW-SLOUGH as the ultimate next hop.

On the SD-WAN overlay side:

• PGW-SLOUGH receives Edge advertisements like a normal Gateway, including from VCE-CHI (which can now reach it over MPLS — see below) and from VCE-NCL, VCE-SHA, etc. (which reach it over Internet).
• PGW-SLOUGH reflects those advertisements to all the other Edges it sees, including back to VCE-CHI.
• Every Edge in the tenant now has an overlay route per remote prefix, two via the Cloud Gateways and one via the Partner Gateway.

Crucially, VCE-CHI can now reach PGW-SLOUGH because PGW-SLOUGH is inside the MPLS VRF. VCE-CHI builds a VCMP control tunnel to PGW-SLOUGH over MPLS. It registers. It exchanges overlay routes. It is now a first-class member of the overlay. The wall is broken.

VCE-CHI cannot build a tunnel to either Cloud Gateway (still Internet-only, still no path) — and that’s fine. From its perspective, PGW-SLOUGH is its Gateway. The Cloud Gateways are visible to it only as advertised but unreachable peers, which the Edge filters out of its tunnel-build candidates.

Flow 4: Newcastle → Chicago via PGW-SLOUGH

User in Newcastle hits a database in Chicago. LAN dst is 10.6.0.40.

Routing — what VCE-NCL knows:

After PGW-SLOUGH is in place, VCE-NCL has overlay routes for 10.6.0.0/16:

• via Cloud Gateways (VCG-LON1 / VCG-MAN1) — but the Cloud Gateways can’t actually reach VCE-CHI either, so they don’t advertise 10.6.0.0/16 to anyone. So in practice this entry is not present.
• via PGW-SLOUGH (Partner Gateway) — yes.

So the only overlay route is via PGW-SLOUGH.

Can a Direct tunnel be built Newcastle ↔ Chicago? No. Newcastle has Internet only; Chicago has MPLS only. No shared underlay. No Direct candidate. This is one of the unavoidable Gateway-relayed flows.

Packet path:

Newcastle LAN
  → VCE-NCL
  → VCMP over broadband (Internet)
  → public Internet to PGW-SLOUGH public IP
  → PGW-SLOUGH (Internet-facing interface)
  → PGW-SLOUGH decapsulates VCMP, looks up dst (10.6.0.40) in its overlay table
  → next hop: VCE-CHI via MPLS-facing VCMP tunnel
  → PGW-SLOUGH re-encapsulates into VCMP for VCE-CHI
  → VCMP over PGW-SLOUGH's MPLS-facing interface
  → MPLS network (GlobalCo-MPLS VRF), routed normally as a private CE-to-CE flow
  → arrives at the PE attached to VCE-CHI's MPLS interface
  → VCE-CHI receives VCMP
  → decapsulates
  → original LAN packet onto Chicago LAN
  → 10.6.0.40

Two VCMP segments, one Partner Gateway in the middle. DMPO measures each leg independently — Newcastle ↔ PGW-SLOUGH over the public Internet, PGW-SLOUGH ↔ VCE-CHI over MPLS. Remediation (FEC, duplication) is per-leg.

The Cloud Gateways are not in the data path at all. They’re still in the control path for VCE-NCL (which is homed to all three: VCG-LON1, VCG-MAN1, and now PGW-SLOUGH). They have nothing to add to this flow because they have no path to Chicago. They sit out.

The asymmetry consequence: VCE-CHI’s reply takes the same Partner Gateway in reverse — PGW-SLOUGH sees both halves, so flow symmetry through it is enforced as in Part 3. Stateful middleboxes between the Edges (firewalls inside the LAN) are happy.

Flow 5: Chicago → Shanghai via PGW-SLOUGH — the headline

The flow this series exists to explain. Chicago has MPLS only; Shanghai has Internet only; they share no underlay.

Routing — what VCE-CHI knows:

• Overlay route for Shanghai LAN (10.7.0.0/16) via PGW-SLOUGH only. (The Cloud Gateways could advertise this — they have Shanghai on Internet — but they have no path to Chicago, so the Cloud-Gateway-mediated route to Shanghai is not usable from Chicago’s perspective. The Edge filters it out at selection time as “Gateway unreachable from me”.)
• No Direct candidate. No shared underlay with Shanghai.

Packet path:

Chicago LAN
  → VCE-CHI
  → VCMP over MPLS (private, inside GlobalCo-MPLS VRF)
  → MPLS network routes the VCMP packet as a normal CE-to-CE private flow
  → arrives at the PE near PGW-SLOUGH in Slough
  → PGW-SLOUGH (MPLS-facing interface)
  → PGW-SLOUGH decapsulates VCMP, looks up dst (10.7.0.40)
  → next hop: VCE-SHA via Internet-facing VCMP tunnel
  → PGW-SLOUGH re-encapsulates into VCMP for VCE-SHA
  → VCMP over PGW-SLOUGH's Internet-facing interface (public IP)
  → public Internet across Europe, Asia, etc., to VCE-SHA's public IP
  → VCE-SHA receives VCMP
  → decapsulates
  → original LAN packet onto Shanghai LAN
  → 10.7.0.40

Two VCMP segments, two underlays, one Partner Gateway. Chicago’s leg is over MPLS only; Shanghai’s leg is over public Internet only. Neither Edge sees the other’s underlay constraint. From Chicago’s perspective it’s sending a packet to a remote LAN over its only path — MPLS. From Shanghai’s perspective it’s receiving a packet from a remote LAN over its only path — Internet. PGW-SLOUGH bridges the underlay change.

This is the architectural pay-off. A box that speaks both underlays makes the two underlays equivalent at the overlay layer. Without it, this flow is impossible. With it, it’s a two-hop relay that DMPO can measure and remediate on each half.

The latency consequence is real. Chicago → UK over MPLS, then UK → Shanghai over public Internet, is a long way round for a flow whose endpoints are physically closer to each other than either is to the UK. Best practice (Part 5) is to consider two Partner Gateways — one in the UK serving the Atlantic, one in APAC serving the Pacific — to keep the relay near the optimal great-circle route. Cost trade-off.

Two Partner Gateways and choosing between them

Suppose BritNet stands up a second Partner Gateway in Singapore: PGW-SIN. MPLS-attached to GlobalCo’s APAC MPLS partner, public IP on the Internet side.

VCE-CHI now sees overlay routes via both Partner Gateways (assuming Chicago’s MPLS provider hands off to Singapore via a global MPLS partnership). VCE-SHA sees overlay routes via both. Some flows will prefer PGW-SLOUGH; some will prefer PGW-SIN. The decision is:

• Route preference — administratively set. You can mark PGW-SIN as preferred for APAC-destined prefixes and PGW-SLOUGH for EMEA-destined prefixes.
• DMPO link quality — if both Partner Gateways are reachable, DMPO picks the better leg at flow time.
• Business Policy — explicit per-application: “voice always via PGW-SIN for APAC flows”.

A Chicago → Shanghai flow with both Partner Gateways in play would normally land on PGW-SIN, because both Edge legs to PGW-SIN are shorter than the corresponding legs to PGW-SLOUGH. But you have to actually configure that preference; it doesn’t fall out automatically. DMPO knows about link quality, not about geography.

A Gateway fails mid-flow

VCG-LON1 falls over.

• Edges homed to both Cloud Gateways: see the VCMP control tunnel to LON1 drop; their existing routes via LON1 expire after the hold time; they continue with MAN1 routes only. No flows interrupted because routes via LON1 are duplicates of routes via MAN1.
• Edges with active Gateway-relayed flows through LON1: DMPO sees the tunnel drop. Flows migrate to MAN1’s tunnel. Sub-second hit if any.
• Partner Gateway PGW-SLOUGH continues to function — it’s independent.

Now PGW-SLOUGH fails.

• Chicago, which depends solely on PGW-SLOUGH, goes offline from the overlay. Routes via PGW-SLOUGH disappear from every other Edge after hold time.
• Active Chicago flows blackhole. There is no Cloud Gateway alternative for Chicago because the Cloud Gateways have no MPLS path.
• This is why a second Partner Gateway (PGW-SIN in our extended example, or a paired PGW in the same MPLS network) is non-negotiable for production design when you have MPLS-only sites. We hammer this in Part 5.

A note on the Cloud Gateways’ role here

You might be wondering, after walking through flows 4 and 5: what are the Cloud Gateways actually doing in this network? The answer is: they serve the Internet-underlay-only Edges (Newcastle, Shanghai) for Cloud-Gateway-mediated Internet breakout and as a secondary Gateway path for Edges that can reach them. They don’t carry Chicago at all. In a design with Partner Gateways that cover every underlay-tenant the customer needs, the Cloud Gateways are often relegated to internet breakout and a redundancy role for Internet-attached Edges.

In a Partner-Gateway-light design (just Cloud Gateways), every site must have Internet underlay. There’s no shortcut.

What’s next

Part 5 — the last one — wraps everything up: best-practice design rules, the failure modes that catch teams the first time they try this, segmentation and security, and a one-page checklist for designing a new Arista SD-WAN deployment without leaving the obvious holes.

If anything in the flow walkthroughs above doesn’t match what you’ve seen in production, I want to know. The MPLS-only / Internet-only relay case in particular has subtleties that differ by SP, by MPLS partner topology, and by what the Partner Gateway is doing on its BGP route maps — I’ve described the canonical case.