Network Emulation with NETEM: Simulating Latency, Loss, Jitter, and Bandwidth Constraints for Realistic Lab Testing

The problem with a perfect lab

Lab networks lie. Two virtual machines on the same hypervisor talk to each other over a path with sub-millisecond latency, no loss, line-rate throughput, and zero jitter. None of that resembles a real customer’s experience over a 4G uplink, a contended wifi network, or a transatlantic link routed through whatever the BGP fairy decided this morning.

NETEM is the Linux kernel’s network emulator and it lives inside the tc traffic-control subsystem. With one command you can turn an unrealistically clean lab link into a believable simulation of a 200 ms transatlantic path with 0.5% loss, or a satellite uplink with 600 ms latency and bursty jitter, or a mobile network that throws away every fifth packet for fun. If you have ever wondered why your VoIP application is fine in the office and unusable from a hotel room, NETEM is how you reproduce that without flying to a hotel.

This post is for engineers who already know what tc is for, want to use NETEM beyond tc qdisc add dev eth0 root netem delay 100ms, and need it to behave reliably enough to put numbers next to.

How NETEM fits into tc

Linux traffic control is built around queueing disciplines (qdiscs). Every interface has one attached as its root, and that qdisc decides what happens to packets between when the kernel hands them off and when the NIC actually transmits them. NETEM is one of those qdiscs. You attach it as the root, or as a child inside a classful qdisc when you want to apply it to only a subset of traffic.

The important practical facts:

• NETEM only affects egress on the interface it is attached to. To shape both directions of a flow you must attach it on both endpoints, or use a classful setup with an intermediate veth pair.
• NETEM does not generate traffic. It delays, drops, duplicates, reorders, or corrupts the traffic that is already passing through.
• The default Linux scheduler tick (HZ) and the timer resolution affect how precise small delays can be. Anything below a few milliseconds will be lumpy.
• NETEM sits below the application stack. The applications cannot tell — that is the entire point.

Make sure the kernel module is available:

modprobe sch_netem
lsmod | grep sch_netem

It ships with virtually every distro kernel.

Adding, changing, and removing rules

Three commands you will use constantly:

# Apply a 100 ms egress delay to eth0
sudo tc qdisc add dev eth0 root netem delay 100ms

# Change the existing rule (note: change, not add)
sudo tc qdisc change dev eth0 root netem delay 200ms

# Remove and revert to the default qdisc
sudo tc qdisc del dev eth0 root

# Inspect what is currently attached
tc qdisc show dev eth0

Get into the habit of always cleaning up. NETEM is sticky and it is embarrassing to spend an afternoon debugging why production is slow only to find a tc qdisc add ... delay 500ms you forgot about.

Latency and jitter

The simplest case is a fixed delay:

sudo tc qdisc add dev eth0 root netem delay 150ms

Every packet leaves 150 ms later than it would have. Real networks do not behave like this — the latency varies. Add jitter:

sudo tc qdisc add dev eth0 root netem delay 150ms 30ms

Now the delay is uniformly distributed between 120 and 180 ms. That is closer, but still not realistic. Real jitter is correlated — if the previous packet was delayed, the next one is more likely to be. Add a correlation coefficient:

sudo tc qdisc add dev eth0 root netem delay 150ms 30ms 25%

Each packet’s delay is 25% determined by the previous packet’s delay, 75% by a fresh draw. This produces noticeably more believable jitter patterns.

For really realistic emulation, swap the uniform distribution for one of the bundled distributions:

sudo tc qdisc add dev eth0 root netem delay 150ms 30ms distribution normal

The kernel ships with uniform, normal, pareto, and paretonormal distribution tables in /usr/lib/tc/ (path varies by distro). For a realistic transatlantic link, paretonormal produces a long-tailed distribution where most packets cluster around the mean but a small fraction take noticeably longer — which is what you actually see on the wire.

Packet loss

Three loss models are available, in increasing order of realism.

Random independent loss. Each packet is dropped with a fixed probability:

sudo tc qdisc add dev eth0 root netem loss 1%

Real networks rarely lose packets independently. They lose them in bursts when a buffer overflows or a wireless link fades. Add correlation:

sudo tc qdisc add dev eth0 root netem loss 1% 25%

Same idea as latency correlation: 25% of the loss decision comes from the previous packet’s outcome.

Gilbert-Elliott model. Two-state Markov model with a “good” state and a “bad” state. In the bad state loss is much higher. This is closer to how real wireless links behave — periods of fine, then a fade, then fine again:

sudo tc qdisc add dev eth0 root netem loss gemodel 1% 10% 70% 0.1%

The four numbers are: probability of moving from good to bad, probability of moving from bad to good, loss rate while in the bad state, loss rate while in the good state. Tuning these is fiddly but it produces dramatically more realistic results than uniform loss when you are testing a protocol’s recovery behaviour.

Burst loss. Drop a configurable run of consecutive packets when loss happens:

sudo tc qdisc add dev eth0 root netem loss random 1% burst 4

When a loss event triggers, drop up to four packets in a row. Useful for stress-testing TCP fast retransmit.

Duplication, corruption, reordering

These get less attention than latency and loss but matter for protocol testing.

Duplication sends a copy of some packets:

sudo tc qdisc add dev eth0 root netem duplicate 1%

Useful for testing applications that assume idempotency. It is also a quick way to discover which of your stateful middleboxes treat duplicate ACKs badly.

Corruption flips a single random bit in a percentage of packets:

sudo tc qdisc add dev eth0 root netem corrupt 0.5%

Most corrupted packets will be discarded by IP or TCP checksums and look like loss to the application. The interesting case is when the bit flip lands in a payload that is not protected by an application-layer checksum — you find out which protocols actually verify their data and which trust the network.

Reordering delays a fraction of packets so they arrive out of sequence:

sudo tc qdisc add dev eth0 root netem delay 10ms reorder 25% 50%

That says: 25% of packets are sent immediately (skipping the delay), with 50% correlation. The other 75% take the 10 ms delay. The result is that some packets overtake others. This is a good way to find application code that quietly assumes in-order delivery on a transport that does not guarantee it.

Bandwidth and rate limiting

NETEM has a rate parameter, but for serious bandwidth shaping you usually want to combine NETEM with tbf (token bucket filter) or htb (hierarchical token bucket). The simple case first:

sudo tc qdisc add dev eth0 root netem rate 1mbit

This caps the egress rate at 1 Mbit/s. Combine with delay and loss:

sudo tc qdisc add dev eth0 root netem rate 1mbit delay 200ms 20ms loss 0.5%

For more control — for example, if you want bursting — chain qdiscs:

sudo tc qdisc add dev eth0 root handle 1: tbf rate 5mbit burst 32kbit latency 400ms
sudo tc qdisc add dev eth0 parent 1:1 handle 10: netem delay 100ms 10ms loss 0.1%

Now traffic is shaped to 5 Mbit/s by tbf, then handed to NETEM for delay and loss. Order matters. If you put NETEM before tbf, the latency you applied gets eaten by the tbf queue when the link is saturated, which is rarely what you want.

Applying NETEM to only some traffic

Attaching NETEM as the root affects every packet on the interface. Frequently you want to degrade only a specific flow — for example, slow down traffic to a particular host while leaving everything else alone. The pattern is to use a classful qdisc as the root, classify traffic into bands, and attach NETEM only to the band you care about.

# Set up a prio root with three bands
sudo tc qdisc add dev eth0 root handle 1: prio

# Attach NETEM to the third band only
sudo tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 200ms loss 1%

# Filter: any traffic to 10.99.0.0/24 goes to band 3
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 1 \
    u32 match ip dst 10.99.0.0/24 flowid 1:3

Everything destined for 10.99.0.0/24 is now degraded; everything else is untouched. This is how you build a believable test harness for a single application without compromising your SSH session.

Bidirectional emulation

NETEM only delays egress. If you want to simulate a 200 ms round-trip path, you have two choices.

The straightforward one is to apply NETEM at both ends. Half the round-trip on each side. If you control both endpoints, this is fine.

The cleaner one is to use a Linux box as a bridge in the middle, with two interfaces, and apply NETEM separately on each direction. A common pattern is to set up an ifb (intermediate functional block) device to redirect ingress to a queue you can shape:

sudo modprobe ifb
sudo ip link add ifb0 type ifb
sudo ip link set ifb0 up

# Redirect ingress on eth0 into ifb0
sudo tc qdisc add dev eth0 ingress
sudo tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0 \
    action mirred egress redirect dev ifb0

# Now apply NETEM to ifb0 to shape "ingress"
sudo tc qdisc add dev ifb0 root netem delay 100ms

Combined with a NETEM qdisc on eth0’s egress, this gives you independent control over both directions on a single box.

Useful scenarios with numbers

Some recipes that map onto common real-world conditions.

Average transatlantic link, US to UK.

sudo tc qdisc add dev eth0 root netem delay 80ms 5ms distribution normal loss 0.05%

Satellite uplink (geostationary).

sudo tc qdisc add dev eth0 root netem delay 600ms 50ms distribution paretonormal loss 0.5%

Congested wifi.

sudo tc qdisc add dev eth0 root netem delay 30ms 20ms distribution paretonormal loss gemodel 0.5% 5% 30% 0.1%

4G mobile, light congestion.

sudo tc qdisc add dev eth0 root netem rate 5mbit delay 50ms 10ms distribution normal loss 0.2%

Lossy backhaul that pretends to be fine until it is not.

sudo tc qdisc add dev eth0 root netem delay 20ms loss gemodel 0.1% 1% 80% 0.01%

That last one is particularly useful for finding applications with unrealistic timeouts. The link looks clean almost all the time, then occasionally drops 80% of packets for a short window, and reveals every retry policy in the stack.

Verifying the emulation actually does what you think

Trust nothing. Always verify the shape with a measurement before drawing conclusions.

# Confirm latency and jitter
ping -c 100 -i 0.1 <peer>

# Confirm bandwidth
iperf3 -c <peer> -t 30

# Confirm loss with a longer ping run
ping -c 1000 -i 0.05 <peer> | tail -5

mtr is also worth running — it gives you per-hop loss and latency, which is useful for confirming that your NETEM is doing the work and not, say, an upstream switch that has chosen to start dropping frames.

A common mistake is to run NETEM on a virtio NIC inside a hypervisor and not realise that the hypervisor’s own software switch is also adding latency or doing batching. Always verify against ping, not against your assumptions.

Gotchas

A short list of things that have cost me hours.

• The kernel timer resolution puts a floor on how fine your delays can be. Below ~1 ms, expect noise. If you need microsecond precision you are in the wrong tool.
• NETEM’s limit parameter defaults to 1000 packets. Under high bandwidth and high latency this fills up and starts tail-dropping. Raise it: ... limit 100000.
• Some drivers do GSO/TSO offload, sending huge “packets” down to the NIC that NETEM treats as a single unit. If your shaping numbers look off, try ethtool -K eth0 gso off tso off gro off and see if behaviour changes.
• tc -s qdisc show dev eth0 shows packet and byte counters. Check them. If your filter is wrong, you will see zero packets in the band you thought you were shaping.
• NETEM does not survive a reboot unless you persist it (a systemd unit, an ifupdown hook, or a NetworkManager dispatcher script depending on your distro). Do not put it in /etc/rc.local and forget about it.

Where this fits

NETEM is the second half of a pair with tcpdump. tcpdump tells you what is happening on the wire; NETEM lets you make the wire behave however you want. Once you have both, a single Linux box becomes a serious lab tool — you can capture a real customer’s traffic profile, reproduce it on demand, and run your application or protocol against it as many times as you need to prove a fix works.

The next post in the series builds on this directly: with namespaces, veth pairs, and FRRouting, you can stand up an entire multi-router topology on one host and apply NETEM between any two nodes to simulate any path you like.