Why Your Traceroute Shows Packet Loss at Hop 2 and Why It Is Usually Fine

Run mtr against a public address from a home connection and you will probably see this. The second hop shows a quarter of a second of latency and some packet loss, and every hop after it is fine. It looks like a fault. It is almost never one, and you can prove that in a minute with tools you already have.

This came out of testing trippy for the Modern Linux Tools series, where all three traceroute tools gave me the same result and I wanted to know why. I ran everything below from a Debian 13 VM on my home network, towards 1.1.1.1. The addresses of my ISP’s routers are replaced with <isp-router> and <isp-upstream>.

What the tools show

mtr with 20 cycles:

HOST: modern-tools                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.11.1               0.0%    20    1.1   0.9   0.7   1.1   0.1
  2.|-- <isp-router>              10.0%    20  251.8 254.4 230.4 284.1  11.5
  3.|-- <isp-upstream>             0.0%    20   10.5  10.1   9.7  10.8   0.3
  4.|-- 141.101.71.166             0.0%    20   26.2  17.4   9.6 102.9  20.7
  5.|-- 141.101.71.174             0.0%    20   10.5  10.7  10.1  11.0   0.3
  6.|-- 141.101.71.211             0.0%    20   23.1  19.6  11.5  76.0  14.5
  7.|-- 1.1.1.1                    0.0%    20   11.7  11.4  10.6  11.9   0.3

traceroute on the same path:

 1  192.168.11.1  0.721 ms  0.651 ms  0.705 ms
 2  * * *
 3  <isp-upstream>  9.693 ms  9.627 ms  9.564 ms
 4  141.101.71.166  10.296 ms  10.209 ms  10.121 ms

trip in CSV mode, 20 cycles:

Hop,IPs,Loss%,Snt,Last,Avg,Best,Wrst
1,192.168.11.1,0.00,20,0.8,0.84,0.7,1.0
2,<isp-router>,45.00,20,264.4,256.64,245.2,269.9
3,<isp-upstream>,0.00,20,9.9,9.83,8.9,10.3

Three tools, one story. Hop 2 is slow and lossy, and traceroute does not get an answer from it at all. The loss figure changes between runs: 10% in mtr, 45% in trip, and a full line of asterisks in traceroute. The latency does not change. It sits at around 250 ms every time.

Why it cannot be a real delay

Look at hop 3. If packets really took 250 ms to get through hop 2, nothing beyond it could answer in under 250 ms. Hop 3 answers in 10 ms, and the destination answers in 11 ms. The traffic to the destination went through hop 2 and arrived quickly, so the 250 ms is not the time it took to forward the packets.

What is actually being measured

A traceroute works by sending packets with a time-to-live (TTL) that runs out at each router in turn. The router that sees TTL reach zero throws the packet away and sends back an ICMP “time exceeded” message. The tool times that reply.

Producing that reply is a different job from forwarding traffic. A router forwards packets in hardware, at line rate, on its fast path. An ICMP reply to a packet that expired on it is generated by its control-plane CPU, which is also busy with routing protocols, management and everything else. Routers commonly rate-limit or deprioritise that work, on purpose, to protect the CPU. So the reply can be late, or dropped, while the packets passing through the same router are not affected.

That is a theory until you test it. Here are three tests.

Test 1: ping the router directly

If the router is just slow to reply to ICMP, it will be slow when pinged directly too. Twenty pings to hop 2:

20 packets transmitted, 14 received, 30% packet loss, time 5760ms
rtt min/avg/max/mdev = 236.176/254.570/297.878/15.303 ms

The same 250 ms, and 30% loss. Now the router before it, and the destination:

$ ping -c 20 -i 0.3 -q 192.168.11.1
20 packets transmitted, 20 received, 0% packet loss, time 6072ms
rtt min/avg/max/mdev = 0.623/0.738/1.003/0.089 ms

$ ping -c 20 -i 0.3 -q 1.1.1.1
20 packets transmitted, 20 received, 0% packet loss, time 5719ms
rtt min/avg/max/mdev = 10.985/12.476/32.198/4.543 ms

Pinging the destination, the traffic that crosses hop 2, shows no loss and 12 ms. So the same router is slow when it is the target and fine when it is on the path.

Test 2: change the probe type

The router rate-limits ICMP. It may handle other traffic differently. trip can probe with TCP to port 443:

$ trip -p tcp -P 443 -m csv -C 20 -t 5 1.1.1.1
Hop,Loss%,Snt,Last,Avg,Best,Wrst
1,0.00,20,0.8,0.89,0.7,1.1
2,100.00,20,???,0.00,???,???
3,0.00,20,9.7,9.80,9.4,10.8
4,0.00,20,10.2,20.52,9.5,111.7

Now hop 2 does not answer at all, and hops 3 and 4 are clean. This is the same pattern again: the router does not reply to the probes that expire on it, and the path through it works. The TCP run also showed 15% loss on hop 5, which is a different router doing the same thing, and the next test is how to tell.

Test 3: look past the hop

This is the quickest test, and the one I use first. If a hop shows loss or latency and the hops after it are clean, the problem is not in the traffic. Real packet loss on a path shows up on the hop where it starts and on every hop after it. Real added latency does the same.

Hop 2 has loss and later hops have none. Hop 4 had a worst case of 102.9 ms and hop 5 had a worst case of 11.0 ms. Hop 6 had a worst case of 76.0 ms and the destination had 11.9 ms. Each of those is a router slow to answer a probe, not a slow link. The destination is what tells you whether the path as a whole is working.

The rules I use

1. Judge a path by its last hop. If the destination has no loss and sensible latency, the path is fine, whatever the middle hops say.
2. Loss or latency must carry on to count. A problem that starts at hop N is real only if hops N+1 and beyond show it too.
3. Asterisks mean “no reply”, not “no path”. A row of * * * followed by normal hops is a router that does not answer. Traceroute got to hop 3 through hop 2.
4. Test the destination directly. Use ping or a TCP connection to the real target and compare it with what the traceroute says.
5. Use the report modes. One pass of traceroute shows one sample. Twenty cycles of mtr or trip show a pattern, and a pattern is what separates a rate limit from a fault.

When hop 2 is the problem

It can be a real problem. If the loss starts at a hop and carries on to every later hop, or if the destination itself shows loss or high latency, the problem is on the path. Your own traffic would show it too, as slow downloads or dropped video calls. In that case the first hop to show the problem tells you where to look, and it is worth comparing a run over TCP with a run over ICMP, because a path problem will appear in both.

A traceroute is a way of asking each router a question, not a way of watching your traffic. Most of the time the answer to “why is hop 2 slow?” is “because it is busy answering you”.