The grep Benchmark Trap: Why Your Search Looks 100 Times Faster Than It Is

I was timing grep against ripgrep for the Modern Linux Tools series when grep reported searching a 105 MB file in 2 milliseconds. That is faster than the disk could even supply the data. It was wrong, and the reason is worth knowing, because it will quietly corrupt any benchmark of grep you write.

The test machine was a Debian 13 VM with GNU grep 3.11. The file is 109,500,000 bytes: 1.5 million lines of interface port1 set ip 10.0.0.1 255.255.255.0 allowaccess ping https ssh, repeated. Every line matches the pattern allowaccess.

The number that cannot be right

hyperfine is the tool I use for timing commands. It runs a command repeatedly and reports the mean:

$ hyperfine -N -w 2 -r 20 "grep allowaccess /tmp/big.txt"
  Time (mean ± σ):       2.0 ms ±   0.1 ms

Two milliseconds for 105 MB is 50 GB/s. That is not a plausible read speed, even from memory. And the same file by hand:

$ time (grep allowaccess /tmp/big.txt >/dev/null)
real	0m0.003s

$ time (grep allowaccess /tmp/big.txt | wc -l)
1500000
real	0m0.270s

With the output sent to /dev/null, grep takes 3 ms. With the output sent to a pipe, the identical search takes 270 ms. That is about 90 times slower, for the same command on the same file.

What grep is doing

GNU grep checks where its standard output points. When stdout is /dev/null, it knows nobody will see the matching lines, so the only thing left to report is the exit status: did anything match? That question is answered by the first matching line. grep stops there.

strace shows it. Counting system calls for the same search:

Output goes toread callswrite calls
/dev/null9none
A pipe1,12326,734

With /dev/null it reads a few blocks, finds a match and exits. With a pipe it reads the whole file, in 1,123 reads, and writes 1.5 million matching lines out in 26,734 writes.

This is a deliberate optimisation, and it is documented behaviour that has been in GNU grep for years. grep -q is the explicit version, and it is just as quick:

$ hyperfine -N -w 2 -r 20 "grep -q allowaccess /tmp/big.txt"
  Time (mean ± σ):       2.1 ms ±   0.3 ms

The shortcut is correct for what you asked. If the question is “does this file contain the word”, 2 ms is the true cost. It is only wrong when you use it as a measure of how long it takes to search the file.

Why it catches people

hyperfine sends the standard output of the command it is timing to /dev/null by default. Most of the time that is what you want, because you are measuring the command and not your terminal. For grep it triggers the shortcut, so the benchmark measures how long it takes to find the first match.

The mistake is easy to miss because the result looks like good news. A fast result gets a nod, and a slow one gets investigated. I only spotted mine because 2 ms was too fast to believe.

The fix

Send the output to something that is not /dev/null, so grep has to do the real work. A pipe to cat is the simplest:

$ hyperfine -w 2 -r 10 "grep allowaccess /tmp/big.txt | cat"
  Time (mean ± σ):     262.7 ms ±   8.1 ms

Note that I dropped -N here. That flag runs the command without a shell, and a pipe needs one, so the shell adds a few milliseconds to the figure. For comparing two tools through the same pipe it cancels out.

If your matches are few, the pipe barely matters and the full file is scanned either way. The trap affects the case where the pattern is found early in the file, and the result is wrong by the largest factor when most lines match, as they did here.

Does ripgrep do the same?

I ran rg through the same two tests:

Output goes torg mean time
/dev/null161.2 ms
A pipe to cat226.0 ms

rg does not stop at the first match when its output goes to /dev/null. It did the full scan both times. Its two figures differ because writing to a pipe has a cost, and the pipe version includes cat and the shell. This is a difference between the two tools that matters for benchmarks: a comparison of grep and rg with the default hyperfine settings would have said grep was about 80 times faster, when with a pipe the figures are 263 ms against 226 ms.

I do not know that every grep implementation does this. GNU grep 3.11 does, which is the one on Debian 13. Check strace -c on yours before you rely on it.

A checklist for benchmarking search tools

  1. Count the system calls. strace -c -e trace=read,write shows in seconds whether a tool read the whole file. If the number of reads is tiny, the tool stopped early.
  2. Check the output. Make sure each command in the comparison prints the same lines. Pipe both into wc -l or md5sum once, outside the timing.
  3. Use a pipe for timing. | cat works. If you want to remove the cost of the pipe from the picture, compare it with -c (count only), which makes both tools read everything and print one number.
  4. Warm the cache and repeat. One run reads from disk, and later runs read from memory. Use hyperfine’s warm-up option.
  5. Distrust a result that looks too good. If the speed is more than the disk or memory could possibly supply, something is being skipped.

Where the result is for the “does this file contain it” question, grep -q is the right tool and 2 ms is the right answer. Just do not put it in a benchmark table next to a tool that reads the whole file.