Linux Networking from the Ground Up: Network Namespaces, veth Pairs, and Building a Multi-Router Lab on One Host

Why bother with namespaces

There is a class of network testing where standing up a full GNS3 topology, or spinning up containers, or asking IT for VMs, is comically excessive. You want three routers, two switches, and a handful of hosts. You want them to share a kernel because that is fast and free. You want to be able to tear it all down in a single command when you are finished.

Linux gives you the primitives to build exactly that. Network namespaces, veth pairs, and bridges are what every container runtime in the world is built on top of. With FRRouting layered in, you can run real BGP, OSPF, and IS-IS sessions between namespaces on a single laptop and treat the whole thing as a routing lab. This post walks through the primitives, builds a non-trivial topology, and shows how to layer NETEM on top so you can test convergence under real-world conditions.

This post assumes you are comfortable with ip, with basic IPv4 routing, and with the idea that BGP and OSPF involve neighbours, areas, and ASNs. It does not assume you have used ip netns before.

What a namespace actually is

The Linux kernel has had namespaces since 2.6. A network namespace is a separate copy of the kernel’s networking stack: its own interfaces, its own routing tables, its own iptables/nftables rules, its own sockets. From inside one namespace you cannot see another namespace’s interfaces, and traffic does not cross between them unless you wire it up explicitly.

Containers are essentially “a process running with its own set of namespaces”. Linux gives you UTS, PID, mount, IPC, user, cgroup, time, and network namespaces, and Docker or Podman give you a process running inside its own copy of all of them. We are only interested in the network one.

# Create a namespace
sudo ip netns add r1

# List existing ones
ip netns list

# Run a command inside it
sudo ip netns exec r1 ip addr show

# Get a shell inside it
sudo ip netns exec r1 bash

A fresh namespace contains only a down lo. No other interfaces, no routes, no DNS resolution. It is an island.

veth pairs: cabling between islands

A veth (virtual ethernet) pair is a kernel-level virtual cable. You create it as a pair of linked interfaces; whatever goes in one end comes out the other. The trick is that you can move one end of the pair into a different namespace, and now you have a “cable” between the two.

# Create the cable
sudo ip link add veth-r1 type veth peer name veth-host

# One end stays in the host namespace, the other goes into r1
sudo ip link set veth-r1 netns r1

# Bring both ends up
sudo ip netns exec r1 ip link set veth-r1 up
sudo ip link set veth-host up

# Address them
sudo ip netns exec r1 ip addr add 10.10.1.1/30 dev veth-r1
sudo ip addr add 10.10.1.2/30 dev veth-host

# Verify
sudo ip netns exec r1 ip route get 10.10.1.2
ping -c 2 10.10.1.1

That is the smallest possible lab — the host and one namespace, connected by a single point-to-point link. Everything else in this post is an elaboration of this pattern.

A multi-namespace topology

Real labs need more than two nodes. The cleanest way to build them is to give each router its own namespace, and to wire them together using veth pairs and bridges. A bridge is a virtual switch — it lives in the host namespace (or any namespace), and you connect veth ends to it the same way you would connect cables to a real switch.

The topology we will build:

         10.0.12.0/30           10.0.23.0/30
   r1 ------------------- r2 ------------------- r3
  AS65001    eBGP        AS65002     eBGP      AS65003
   |
   | 10.0.10.0/24 (LAN, hosted on a bridge)
   |
  h1

Three routers, each in its own AS, full mesh of eBGP between r1-r2 and r2-r3, with a host hanging off r1 to give us something to ping. The script:

#!/bin/bash
set -e

# Namespaces
for ns in r1 r2 r3 h1; do
    sudo ip netns add $ns
    sudo ip netns exec $ns ip link set lo up
done

# r1 -- r2 link
sudo ip link add r1-r2 type veth peer name r2-r1
sudo ip link set r1-r2 netns r1
sudo ip link set r2-r1 netns r2
sudo ip netns exec r1 ip addr add 10.0.12.1/30 dev r1-r2
sudo ip netns exec r2 ip addr add 10.0.12.2/30 dev r2-r1
sudo ip netns exec r1 ip link set r1-r2 up
sudo ip netns exec r2 ip link set r2-r1 up

# r2 -- r3 link
sudo ip link add r2-r3 type veth peer name r3-r2
sudo ip link set r2-r3 netns r2
sudo ip link set r3-r2 netns r3
sudo ip netns exec r2 ip addr add 10.0.23.1/30 dev r2-r3
sudo ip netns exec r3 ip addr add 10.0.23.2/30 dev r3-r2
sudo ip netns exec r2 ip link set r2-r3 up
sudo ip netns exec r3 ip link set r3-r2 up

# r1 LAN bridge with h1
sudo ip link add br-lan type bridge
sudo ip link set br-lan up

sudo ip link add r1-lan type veth peer name lan-r1
sudo ip link set r1-lan netns r1
sudo ip link set lan-r1 master br-lan
sudo ip link set lan-r1 up
sudo ip netns exec r1 ip link set r1-lan up
sudo ip netns exec r1 ip addr add 10.0.10.1/24 dev r1-lan

sudo ip link add h1-lan type veth peer name lan-h1
sudo ip link set h1-lan netns h1
sudo ip link set lan-h1 master br-lan
sudo ip link set lan-h1 up
sudo ip netns exec h1 ip link set h1-lan up
sudo ip netns exec h1 ip addr add 10.0.10.10/24 dev h1-lan
sudo ip netns exec h1 ip route add default via 10.0.10.1

# Loopbacks for router IDs and to give BGP something to advertise
sudo ip netns exec r1 ip addr add 1.1.1.1/32 dev lo
sudo ip netns exec r2 ip addr add 2.2.2.2/32 dev lo
sudo ip netns exec r3 ip addr add 3.3.3.3/32 dev lo

# Forwarding on the routers
for r in r1 r2 r3; do
    sudo ip netns exec $r sysctl -w net.ipv4.ip_forward=1 >/dev/null
done

At this point you have a fully cabled topology with no routing protocol running. From h1 you can ping r1’s LAN address, but nothing further. r1 cannot reach 3.3.3.3 yet.

Adding FRRouting

FRR (FRRouting) is the fork of Quagga that everyone actually uses. Install it on the host — it does not need to live inside the namespaces because we are going to run separate FRR instances per namespace, all driven from the same binary set.

sudo apt install frr frr-pythontools

The trick to running multiple FRR instances on one host is to give each one its own config directory and its own state. The cleanest way is to write a small wrapper that launches FRR daemons inside a namespace, pointing at a per-router config dir.

Create a directory tree:

sudo mkdir -p /etc/frr/r1 /etc/frr/r2 /etc/frr/r3

Write /etc/frr/r1/frr.conf:

frr defaults traditional
hostname r1
log file /tmp/r1-frr.log
!
interface lo
 ip address 1.1.1.1/32
!
router bgp 65001
 bgp router-id 1.1.1.1
 neighbor 10.0.12.2 remote-as 65002
 !
 address-family ipv4 unicast
  network 10.0.10.0/24
  network 1.1.1.1/32
  neighbor 10.0.12.2 activate
 exit-address-family
!
line vty
!

/etc/frr/r2/frr.conf:

frr defaults traditional
hostname r2
log file /tmp/r2-frr.log
!
interface lo
 ip address 2.2.2.2/32
!
router bgp 65002
 bgp router-id 2.2.2.2
 neighbor 10.0.12.1 remote-as 65001
 neighbor 10.0.23.2 remote-as 65003
 !
 address-family ipv4 unicast
  network 2.2.2.2/32
  neighbor 10.0.12.1 activate
  neighbor 10.0.23.2 activate
 exit-address-family
!
line vty
!

/etc/frr/r3/frr.conf:

frr defaults traditional
hostname r3
log file /tmp/r3-frr.log
!
interface lo
 ip address 3.3.3.3/32
!
router bgp 65003
 bgp router-id 3.3.3.3
 neighbor 10.0.23.1 remote-as 65002
 !
 address-family ipv4 unicast
  network 3.3.3.3/32
  neighbor 10.0.23.1 activate
 exit-address-family
!
line vty
!

Now launch FRR inside each namespace. The cleanest way is to use the namespaced version of frrinit.sh — recent FRR builds support -N <name> to give each instance a separate state dir. If your distro packaging does not, the fallback is to launch zebra and bgpd manually:

sudo ip netns exec r1 /usr/lib/frr/zebra -d -f /etc/frr/r1/frr.conf -i /tmp/r1-zebra.pid -z /tmp/r1-zserv.api
sudo ip netns exec r1 /usr/lib/frr/bgpd  -d -f /etc/frr/r1/frr.conf -i /tmp/r1-bgpd.pid  -z /tmp/r1-zserv.api

Repeat for r2 and r3, changing every reference. Within a few seconds the BGP sessions should come up.

To check, drop into the FRR shell inside a namespace:

sudo ip netns exec r1 vtysh -c "show bgp summary"
sudo ip netns exec r1 vtysh -c "show ip route"
sudo ip netns exec r3 vtysh -c "show ip route"

You should see routes for 10.0.10.0/24, 1.1.1.1/32, 2.2.2.2/32, and 3.3.3.3/32 propagating across the topology. From h1, ping 3.3.3.3 should work.

OSPF instead of BGP

Same scaffolding, different routing config. To run OSPF in area 0 across the same topology, replace the router bgp blocks with:

router ospf
 ospf router-id 1.1.1.1
 network 10.0.12.0/30 area 0
 network 1.1.1.1/32 area 0
 network 10.0.10.0/24 area 0

And launch ospfd in each namespace alongside zebra. The lab is otherwise identical. This is the pattern: the cabling and the namespaces are stable, and you swap the routing protocol on top.

Layering NETEM on the cables

This is where the lab gets useful. You can apply NETEM to any veth interface from inside its namespace, and the delay/loss/jitter will affect only that link.

Slow down the r1-r2 link to simulate a transatlantic path:

sudo ip netns exec r1 tc qdisc add dev r1-r2 root netem delay 80ms 5ms distribution normal
sudo ip netns exec r2 tc qdisc add dev r2-r1 root netem delay 80ms 5ms distribution normal

Now you can watch how BGP keepalives, OSPF Hello timers, and convergence behave under realistic latency. Add 1% loss and you can study what happens to BFD timers if you turn BFD on. Crank the loss up to 50% and you can watch the session flap.

Combine this with the prio band trick from the NETEM post and you can degrade specific traffic — for example, the BGP control plane only — without affecting data plane forwarding tests.

Persistence and cleanup

Namespaces and veth pairs do not survive a reboot. That is usually fine — labs are throwaway. If you want a particular topology to come up automatically, write a systemd unit that runs your build script:

# /etc/systemd/system/netlab.service
[Unit]
Description=Network namespace lab
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/netlab-up.sh
ExecStop=/usr/local/sbin/netlab-down.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The teardown script is mercifully short:

#!/bin/bash
for ns in r1 r2 r3 h1; do
    sudo ip netns del $ns 2>/dev/null || true
done
sudo ip link del br-lan 2>/dev/null || true
sudo pkill -f 'r[123]-zebra' || true
sudo pkill -f 'r[123]-bgpd'  || true
sudo rm -f /tmp/r[123]-*.pid /tmp/r[123]-*.api

Deleting a namespace automatically tears down anything inside it, including the veth ends — the kernel cleans up everything on namespace destruction. The bridge in the host namespace has to be removed explicitly.

Practical patterns

A few things this lab is unreasonably good at.

Reproducing a customer’s reported issue. Build the topology that mirrors theirs, apply NETEM to match the path characteristics, and run the same configuration. Most “intermittent” routing problems are reliably reproducible once you can shape the link.

Pre-change rehearsal. Before you change OSPF area boundaries or BGP communities in production, run the same change in a namespace lab with the same FRR version and watch what happens. The kernel module under FRR is the same code you run in production.

Convergence testing. Time how long it takes the topology to reconverge after a link drop:

sudo ip netns exec r1 ip link set r1-r2 down
# from h1 in another shell:
ping 3.3.3.3

You can graph this. You can compare BFD vs no BFD. You can measure the cost of a specific timer change. None of this is possible on a quiet production network without breaking it.

Teaching tool. A namespace lab is the cheapest way to demonstrate route reflection, confederations, route maps, AS-path manipulation, or community filtering to someone who has only ever read about them.

Where this stops being the right tool

Namespaces share a kernel. That means they share kernel features and kernel bugs. If your test depends on a specific platform’s microcode, hardware ASIC behaviour, or vendor-specific routing code, you need a different lab. For Cisco IOS, you still need IOL or VIRL or real hardware. For FortiGate, you still need a VM. Namespaces give you Linux behaviour, which is plenty for FRR-based labs but not for vendor parity testing.

The next two posts in this series cover firewalling — first migrating iptables to nftables without breaking production, and then SSH hardening. Both are things you can build into namespaces, but they are useful enough on their own that they deserve dedicated posts.