Cloud On-Ramp Part 1: The Architecture Decision and AWS Transit Gateway
Hub Placement Part 3 closed out that series with a placement principle dressed up as a throwaway line: “the underlay is the cloud provider’s SDN, and it has opinions.” That’s true, and it’s also where the series stopped — because where the hub goes is a placement question, and how it gets traffic into and out of the cloud fabric once it’s there is an entirely different, much more concrete engineering problem. This post and the next one are about that problem: the actual on-ramp mechanics of peering a Fortinet SD-WAN overlay into a hyperscaler’s transit infrastructure.
This part covers the architecture decision — why a transit-hub model rather than a peering mesh — and AWS Transit Gateway specifically: BGP-over-IPsec mechanics, ASN selection, and how to map the on-prem VRF segmentation built up across the MP-BGP and VRFs and Route Leaking Between VRFs posts onto TGW’s route-table model. Part 2 does the same for Azure Virtual WAN, then closes with a worked dual-cloud resilience design.
The architecture decision: why a transit hub, not a mesh
Before any cloud-specific detail, there’s a shape question that has to be settled, and it’s one this blog has effectively already answered — just in a different context. Hub Placement Part 1 spent considerable effort establishing why a hub-and-spoke overlay beats a full mesh: a hub is a routing anchor, a policy enforcement point, and a single place to get the FMG/FAZ relationship right, instead of N(N-1)/2 pairwise relationships that all have to be right independently.
The cloud on-ramp question is the same shape, one layer down: does each VPC/VNet peer directly with the on-prem overlay (a mesh), or does traffic transit through a single, cloud-native hub construct that all VPCs and the on-prem overlay attach to? AWS’s answer to “what should that construct be” is Transit Gateway (TGW) — a managed regional router that VPCs, VPNs, Direct Connect gateways, and (relevantly here) a FortiGate’s Site-to-Site VPN attachments all plug into.
Choosing TGW over a mesh of per-VPC VPN connections is, structurally, the exact same argument Part 1 made for hubs over meshes:
- One routing anchor instead of many. Route propagation is configured once, on the TGW’s route tables, instead of replicated across every VPC’s VPN gateway.
- One place to enforce the VRF/segmentation model, rather than hoping every VPC’s individual VPN connection was configured with matching, non-conflicting policy.
- A failure domain that scales sanely. Adding the fortieth VPC to the estate is a TGW attachment and a route-table association — not a fortieth independent VPN tunnel pair with its own BGP session to keep an eye on.
That’s the decision. Once it’s made, the rest of this post is “how do you actually wire a FortiGate SD-WAN hub into one.”
The AWS Transit Gateway model, in the terms that matter here
TGW has its own vocabulary, and getting it straight up front saves a lot of confusion later — particularly because some of these terms sound like FortiOS concepts and mean something adjacent-but-different:
| TGW term | What it actually is | FortiOS-world analogue |
|---|---|---|
| Transit Gateway | The regional hub construct itself — a managed router that attachments connect to | The cloud-resident hub FortiGate’s role, but implemented by AWS, not by a FortiGate |
| Attachment | A connection from something (a VPC, a VPN, a Direct Connect gateway, a peering TGW) to the Transit Gateway | An SD-WAN member, loosely — a thing that can send and receive traffic through the hub |
| VPN attachment | Specifically, a Site-to-Site VPN connection — this is what the FortiGate’s IPsec tunnels terminate into | The IPsec overlay member type from the zones config |
| Transit Gateway route table | A route table belonging to the TGW, separate from any VPC’s route table — this is the actual routing-domain separation mechanism | The closest cloud-native equivalent to a VRF |
| Association | Which route table an attachment uses to route its outbound traffic | Which VRF an interface is assigned to |
| Propagation | Which route table an attachment’s routes get advertised into | Route redistribution into a VRF’s BGP instance |
That last row is the one to sit with for a moment, because it’s where people most often get tripped up coming from a FortiOS background: association and propagation are independent settings on the same attachment. An attachment can propagate its routes into one route table while being associated with (and therefore routing its own traffic via) a different one. That’s not an edge case to memorize and forget — it’s the exact mechanism the VRF-mapping section below depends on.
BGP-over-IPsec: how the FortiGate actually peers with a TGW
A TGW VPN attachment isn’t a single tunnel — AWS provisions two IPsec tunnels per VPN connection, terminating on two different public IP addresses, specifically so that a single-tunnel failure (or a maintenance event on AWS’s side, which does happen and is exactly why there are two) doesn’t take the whole attachment down. Each tunnel gets its own BGP session. From the FortiGate’s side, that means:
- Two IPsec phase1/phase2 pairs, one per AWS tunnel endpoint
- Two BGP neighbor statements, one per tunnel’s inside-tunnel IP — not peering on loopbacks here, which is worth pausing on
- ECMP or active/passive preference between them, exactly the same active/standby-vs-active/active decision the Resilience series walked through for DC hub pairs — except now both “hubs” are tunnels into the same managed construct rather than two physical FortiGates
That last point about loopbacks deserves its own paragraph, because it inverts something this blog has spent a lot of words establishing as a best practice. SD-WAN Resilience Part 2 and Hub Placement Part 1 both make the case — correctly, for that context — that BGP should peer on stable loopback addresses rather than tunnel-interface IPs that move on every re-establishment. AWS’s side of a TGW VPN attachment does not support loopback peering: the BGP session runs over the tunnel’s own inside-tunnel /30, full stop, because that’s how AWS provisions the endpoint. The FortiGate has no choice in the matter on this leg.
This isn’t a contradiction of the loopback-peering principle so much as a reminder of why it was a principle in the first place: it solves a problem (tunnel re-establishment churning the BGP session) that exists when you control both ends of the tunnel and can choose to decouple the BGP session from the transport. When one end is a managed service that has already made that choice for you — and made it the other way — the right move is to peer the way the managed service expects, and let its own redundancy model (two tunnels, two independent BGP sessions, AWS-side automatic failover between them) do the job loopback peering would otherwise have done. Forcing loopback peering onto an endpoint that doesn’t support it isn’t more resilient; it’s just broken.
config vpn ipsec phase1-interface
edit "aws-tgw-tun1"
set interface "wan1"
set peertype any
set net-device disable
set proposal aes256-sha256
set dpd on-idle
set remote-gw <AWS tunnel 1 outside IP>
set psksecret <tunnel 1 PSK from AWS config file>
next
edit "aws-tgw-tun2"
set interface "wan1"
set peertype any
set net-device disable
set proposal aes256-sha256
set dpd on-idle
set remote-gw <AWS tunnel 2 outside IP>
set psksecret <tunnel 2 PSK from AWS config file>
next
end
config router bgp
set as 65010
config neighbor
edit "<tunnel 1 AWS-side inside IP>"
set remote-as 64512
set update-source "aws-tgw-tun1"
set soft-reconfiguration enable
next
edit "<tunnel 2 AWS-side inside IP>"
set remote-as 64512
set update-source "aws-tgw-tun2"
set soft-reconfiguration enable
next
end
end64512 is AWS’s default Amazon-side ASN for a Transit Gateway — it’s configurable at TGW creation time, but plenty of estates leave it at the default, which is fine as long as nothing else in the estate is already using it. That caveat is the entire content of the next section.
ASN selection: the mistake that’s invisible until it isn’t
Every BGP design this blog has walked through — MP-BGP and VRFs, the Resilience series’ loopback peering, BGP Route Dampening — has lived inside a single estate’s private-ASN space, where the only thing that mattered was internal consistency. Cloud on-ramp is the first time this blog’s BGP material has to reckon with a design that isn’t entirely under one administrator’s control, and ASN collisions are the most common, most quietly-dangerous way that surfaces.
The private ASN range is 64512–65534 (RFC 6996). AWS’s default Amazon-side TGW ASN — 64512 — sits right at the bottom of it. If your existing private WAN already uses 64512 anywhere (a remarkably common choice, precisely because it’s the first number in the range and the path of least resistance when someone needed “an ASN” five years ago and grabbed the first one that worked), and you peer a FortiGate configured with that same ASN against a TGW that’s also using it, you get a same-ASN adjacency. Depending on the exact BGP implementation details on both sides, that either silently works in a way that breaks loop detection, or silently doesn’t establish at all — and either failure mode produces symptoms (“routes aren’t propagating,” “routes are propagating but traffic loops”) that look nothing like “check your ASNs.”
The fix is unglamorous and entirely preventable with five minutes of inventory before the first tunnel goes up:
- Audit every private ASN already in use across the estate — on-prem BGP instances, any existing cloud VPN connections, any partner/MSSP peerings — before assigning one to a new TGW or to the FortiGate’s customer-gateway-side configuration.
- Treat the cloud on-ramp’s ASN choices as part of the same namespace as everything else, not a fresh slate just because it’s “in AWS now.” A TGW’s Amazon-side ASN and a FortiGate’s customer-gateway ASN both have to be unique within the reachable BGP topology, which — the moment an overlay connects on-prem to cloud — now includes both sides.
- When in doubt, change the TGW’s side, not the FortiGate’s. The FortiGate’s ASN is shared across every BGP relationship it has — the overlay to spokes, any DC peerings, the works. The TGW’s Amazon-side ASN is local to that one Transit Gateway and trivially reconfigurable at creation time (and reconfigurable, with more care, afterward). Moving the thing that has the smallest blast radius is the same instinct that made VRF-scoped route tables the right tool for segmentation in the section below — keep changes contained to where they’re cheapest to make.
Mapping on-prem VRFs onto TGW route tables
This is the part where the work already done on this blog pays off directly — because the problem of “I have multiple isolated routing domains on-prem and need that isolation to survive crossing into the cloud” is exactly the problem MP-BGP and VRFs and Route Leaking Between VRFs solved on the FortiGate side. The question is just: what’s the TGW-side equivalent of a VRF, and how do the two ends line up?
The answer is the TGW route table — a routing table that belongs to the Transit Gateway itself, independent of any VPC’s own route table, and which (as the vocabulary section above flagged) attachments are associated with for outbound routing and can independently propagate their routes into. Building a clean mapping looks like this:
On-prem (FortiGate) AWS (Transit Gateway)
------------------- ---------------------
VRF 0 (corporate) <-------> TGW route table "corp-rt"
- associated: corp VPN attachment,
corp VPC attachments
- propagates: corp VPC CIDRs,
on-prem corp prefixes
VRF 1 (guest) <-- X --> (no attachment — guest never crosses
to AWS at all; it breaks out locally,
per the LIB post)
VRF 2 (DMZ/partner) <-------> TGW route table "dmz-rt"
- associated: DMZ VPN attachment,
partner VPC attachments
- propagates: partner VPC CIDRs,
on-prem DMZ prefixes only
(NOT corp prefixes — this is the
cloud-side route leak to prevent)A few things about that mapping are worth making explicit, because they’re exactly the kind of detail that’s easy to get backwards under deadline pressure:
Each VRF that needs cloud reachability gets its own VPN attachment and its own TGW route table — not a shared attachment with route filtering bolted on afterward. This mirrors the on-prem lesson directly: Route Leaking Between VRFs makes the case that “no route exists to leak” is a structurally stronger guarantee than “a policy denies the route that exists.” The same logic applies to TGW route tables — associating the DMZ attachment with dmz-rt and only propagating DMZ-relevant prefixes into it means there is no corporate route sitting in a table the DMZ side could ever consult, accidentally or otherwise. A shared attachment with “allow DMZ subnets, deny corporate subnets” filtering is the procedural version of the same guarantee, and it has the same weakness: it has to be gotten right, and kept right, by someone, forever.
Some VRFs legitimately have no cloud-side counterpart at all — and that’s fine, and worth stating in the design rather than leaving as an absence someone has to notice. The guest VRF from the Local Internet Breakout post is the clean example: it never had a route to anything beyond the local breakout zone in the first place, so it has no business with a TGW attachment, a route table, or an ASN of its own. The absence isn’t a gap in the cloud on-ramp design; it’s the on-prem isolation guarantee continuing to do its job, undisturbed, into a part of the topology that simply doesn’t concern it.
BGP path attributes are now the mechanism that decides what each side learns — and they cross an administrative boundary. On-prem, route-maps and prefix-lists controlling what gets redistributed between VRFs are entirely under one team’s control. The moment a BGP session crosses into TGW territory, what gets propagated into a route table (an AWS-side setting) and what the FortiGate chooses to accept and redistribute (a FortiOS-side setting, via route-maps on the BGP neighbor) are configured in two different consoles, by — in larger organisations — two different teams, possibly on two different change-control cadences. Document both halves of every cross-boundary route policy in one place. The single most common cause of “routes I expected to see aren’t there” in a cloud on-ramp isn’t a misconfigured BGP session; it’s one side changing its half of the agreement without the other side knowing it happened.
Verifying the on-ramp
Once the tunnels are up and BGP has had a chance to converge, three checks confirm the design is doing what it’s supposed to:
diagnose vpn ike gateway list name aws-tgw-tun1
diagnose vpn ike gateway list name aws-tgw-tun2Confirms both phase1 SAs are established — the prerequisite for everything that follows. If only one tunnel is up, BGP will still work (that’s the point of having two), but it’s worth knowing which one is down and why before treating the on-ramp as healthy.
get router info bgp neighbors <AWS-side inside IP>Run against both neighbor addresses. Confirms the session is Established, and — more usefully — shows the prefix counts sent and received, which is the fastest way to spot a propagation mismatch (FortiGate sending prefixes AWS isn’t accepting into the route table, or vice versa) before it shows up as a connectivity ticket from someone trying to reach a VPC resource.
get router info routing-table bgp vrf <N>The same VRF-scoped routing-table check the LIB post closed with, run here for exactly the same reason: confirm that VRF 0 sees the corporate VPC prefixes, VRF 2 sees the DMZ/partner prefixes, and — just as importantly — that neither VRF’s table contains a single route it shouldn’t. On the AWS side, the TGW console’s route table view provides the mirror image of this check; reading both halves side by side is the only way to be confident the mapping is symmetric and the isolation guarantee actually survived the trip across the boundary.
What’s next
| Part | Topic |
|---|---|
| 1 (this post) | The architecture decision, AWS Transit Gateway, BGP-over-IPsec mechanics, ASN selection, mapping VRFs onto TGW route tables |
| 2 | Azure Virtual WAN — the virtual hub model compared with TGW, route propagation and labels, and a worked dual-cloud resilience design |
Azure’s Virtual WAN takes a recognisably similar shape — a managed hub construct that attachments plug into, with its own route-propagation model — but the vocabulary, the BGP mechanics, and (this is the part worth waiting for) the failure modes differ in ways that matter the moment an estate is on-ramped into both clouds at once, or into one cloud and one DC, and has to behave sensibly when either path degrades. That’s where Part 2 picks up.