eBPF and bpftrace: Network Observability Without Touching the Packet Path

24 June 2026 Linux

Every tool covered in this series so far works by observing after the fact — ss reads kernel socket state, tshark processes packets that have already been captured, tc instruments a qdisc you have configured. eBPF is different. It lets you attach small programs directly to kernel events — a TCP connection opening, a socket buffer filling, a packet arriving at the XDP hook before it even reaches the network stack — and extract exactly the data you care about, with kernel-verified safety and near-zero overhead.

bpftrace is the high-level interface to eBPF for observability. Where writing raw eBPF requires C, libbpf, and a compilation toolchain, bpftrace gives you a scripting language modelled on awk that compiles to eBPF bytecode on the fly. A one-liner that traces every TCP connection establishment across the entire system is a single bpftrace command.

This post covers what eBPF is, why it matters for network engineering, and the practical bpftrace patterns that answer questions no other tool can.

What eBPF Actually Is

eBPF (extended Berkeley Packet Filter) is a kernel subsystem that allows user-supplied bytecode to run inside the kernel, attached to specific probe points. The kernel verifier checks the program before loading it — it must terminate, cannot access arbitrary memory, and cannot crash the kernel. If verification passes, the program is JIT-compiled and runs at near-native speed.

Probe points relevant to network work include:

Probe type	What it hooks	Example use
`kprobe` / `kretprobe`	Any kernel function entry/return	Hook `tcp_connect`, `tcp_close`, `inet_csk_accept`
`tracepoint`	Stable kernel tracepoints (preferred over kprobes)	`tracepoint:tcp:tcp_retransmit_skb`, `tracepoint:net:netif_receive_skb`
`xdp`	Packet arrival before the network stack	Per-packet filtering/counting at line rate
`socket filter`	Socket-level packet filtering	Per-socket observability
`tc` eBPF	Traffic control hook (ingress/egress)	Packet mangling or policy at the qdisc layer

The key distinction from tcpdump or tshark: those tools work by copying packet data to userspace. eBPF programs run in the kernel and can aggregate, filter, and summarise before anything reaches userspace — which is why they scale to line-rate on 100Gbps NICs where tcpdump would drop packets.

Installation

# bpftrace
apt install bpftrace          # Ubuntu 20.04+
dnf install bpftrace          # RHEL 8+ / Fedora

# bcc tools (a complementary set of pre-written eBPF programs)
apt install bpfcc-tools
dnf install bcc-tools

# Kernel headers (required for kprobes)
apt install linux-headers-$(uname -r)

Most bpftrace programs require root or CAP_BPF + CAP_PERFMON. Check your kernel version — eBPF networking features improve significantly from 4.x to 5.x to 6.x:

uname -r
bpftrace --version

bpftrace Language Basics

A bpftrace program has the structure:

probe { action }

Multiple probe/action blocks in one script. Variables prefixed with @ are maps (hash tables stored in kernel space). printf emits to stdout. @[key] = count() builds a frequency map.

List available probes

# All TCP-related tracepoints
bpftrace -l 'tracepoint:tcp:*'

# All kernel functions matching a pattern
bpftrace -l 'kprobe:tcp_*'

# Inspect tracepoint arguments
bpftrace -lv 'tracepoint:tcp:tcp_retransmit_skb'

Core Networking Patterns

Trace every new TCP connection (source, dest, port)

bpftrace -e '
tracepoint:sock:inet_sock_set_state
/args->newstate == 1/
{
  printf("%-6d %-20s %-20s %d -> %d\n",
    pid,
    comm,
    ntop(args->family, args->saddr),
    args->sport,
    args->dport);
}
'

ntop() converts a raw address to a printable string. args->newstate == 1 filters for TCP_ESTABLISHED (state 1 in the kernel enum). This catches every outbound and inbound connection establishment system-wide — no pcap, no promiscuous mode.

Count connections by destination port (live histogram)

bpftrace -e '
tracepoint:sock:inet_sock_set_state
/args->newstate == 1/
{
  @ports[args->dport] = count();
}

END { print(@ports); }
'

Run for 30 seconds then Ctrl-C. Output is a sorted frequency table of destination ports — useful for auditing what a host is connecting to without capturing a single packet.

TCP retransmit rate by remote address

bpftrace -e '
tracepoint:tcp:tcp_retransmit_skb
{
  @retransmits[ntop(AF_INET, args->saddr)] = count();
}

interval:s:5
{
  print(@retransmits);
  clear(@retransmits);
}
'

This emits a retransmit-per-remote-address table every 5 seconds. Addresses with climbing retransmit counts indicate a lossy path — cross-reference with ss -tni to confirm cwnd reduction on those flows.

Trace TCP state machine transitions

bpftrace -e '
tracepoint:sock:inet_sock_set_state
/args->protocol == IPPROTO_TCP/
{
  printf("%-20s %s -> %s\n",
    ntop(args->family, args->daddr),
    tcp_state[args->oldstate],
    tcp_state[args->newstate]);
}
' \
--include /usr/include/linux/tcp.h

For debugging CLOSE_WAIT accumulation or TIME_WAIT exhaustion in real time — you see every state transition as it happens, with the remote address.

Measure TCP connection latency (SYN to ESTABLISHED)

bpftrace -e '
tracepoint:sock:inet_sock_set_state
/args->newstate == 2/
{
  @start[args->skaddr] = nsecs;
}

tracepoint:sock:inet_sock_set_state
/args->newstate == 1 && @start[args->skaddr]/
{
  $latency_us = (nsecs - @start[args->skaddr]) / 1000;
  @latency_us = hist($latency_us);
  delete(@start[args->skaddr]);
}
'

hist() produces a power-of-2 histogram. This tells you the actual TCP handshake latency distribution across all connections on the host — not sampled, not averaged, every connection.

Count packets by protocol at the network layer

bpftrace -e '
tracepoint:net:netif_receive_skb
{
  @rx_packets = count();
}

tracepoint:net:net_dev_xmit
{
  @tx_packets = count();
}

interval:s:1
{
  printf("RX: %d  TX: %d\n", @rx_packets, @tx_packets);
  clear(@rx_packets);
  clear(@tx_packets);
}
'

Packet-per-second rate at the network driver level, before any tc qdisc processing.

Identify which processes are opening the most sockets

bpftrace -e '
tracepoint:syscalls:sys_enter_socket
{
  @sockets_by_proc[comm] = count();
}

interval:s:10
{
  print(@sockets_by_proc);
  clear(@sockets_by_proc);
}
'

Useful when you see an unexpected spike in ss output and want to know which process is responsible without filtering through lsof or /proc.

XDP: Per-Packet Observability at Line Rate

XDP (eXpress Data Path) is an eBPF hook that runs before the kernel allocates a socket buffer for the packet. It is the closest you can get to the wire in software. XDP programs can pass, drop, redirect, or modify packets — and because they run before the memory allocator, they can sustain line-rate on 10/25/100Gbps interfaces where the kernel stack would saturate.

For observability (not filtering), attach in XDP_FLAGS_SKB_MODE which does not require driver support:

bpftrace -e '
xdp:eth0
{
  @packets_by_src[ntop(AF_INET, ((struct iphdr *)skb->data)->saddr)] = count();
}

interval:s:5
{
  print(@packets_by_src);
  clear(@packets_by_src);
}
'

This counts inbound packets by source IP at the NIC, before routing, before iptables, before any userspace tool could observe them. DDoS traffic that is dropped by XDP never appears in ss, tshark, or /proc/net/ — but it shows up here.

bcc Tools for Network Work

The bcc (BPF Compiler Collection) package ships a set of pre-written, production-ready eBPF tools. The network-relevant ones:

Tool	What it does
`tcpconnect`	Logs every outbound TCP connection with PID and latency
`tcpaccept`	Logs every accepted inbound TCP connection
`tcpretrans`	Logs TCP retransmissions with state and remote address
`tcplife`	Per-connection lifetime, bytes transferred, duration
`tcptop`	Real-time TCP throughput by process and connection
`sofdsnoop`	Traces file descriptor passing over sockets (detect socket hijacking)
`netqtop`	Per-queue NIC statistics (useful with multi-queue NICs)

# Watch TCP connection lifetimes live
/usr/share/bcc/tools/tcplife

# Retransmit events with process info
/usr/share/bcc/tools/tcpretrans

# Per-connection throughput, refreshed every second
/usr/share/bcc/tools/tcptop

Practical Scenario: Diagnosing Intermittent Connection Drops

A service reports intermittent connection resets. ss shows no persistent anomalies. tshark captures nothing because the window is too narrow. eBPF can watch continuously:

bpftrace -e '
tracepoint:sock:inet_sock_set_state
/args->newstate == 7/
{
  printf("%s  %s:%d -> %s:%d  (pid %d / %s)\n",
    strftime("%H:%M:%S", nsecs),
    ntop(args->family, args->saddr), args->sport,
    ntop(args->family, args->daddr), args->dport,
    pid, comm);
}
' | tee /tmp/tcp-close-events.log

Leave this running. Every TCP close — clean shutdown or reset — is logged with timestamp, addresses, ports, and the process. When the next incident occurs, the log shows exactly which connections closed at that moment, and which process owned them.

Combine with the RST classification from rst-forensics: eBPF tells you when and from which process; rst-forensics tells you whether the RST came from the server, the client, or a mid-path device.

Connecting to the Series

eBPF operates at a lower level than every other tool in this series, which makes it complementary rather than a replacement:

ss -tni shows current socket state; eBPF shows state transitions as they happen
tshark -z expert flags retransmissions after capture; tcp_retransmit_skb tracepoint flags them in real time without capturing anything
tc instruments the qdisc layer; XDP eBPF instruments the layer below it, before the qdisc
ip -j route show shows the current routing table; kprobes on ip_route_input_slow show routing decisions per-packet as they are made

The jq post patterns apply directly to bpftrace output — pipe the JSON output format (bpftrace -f json) into jq for post-processing and aggregation.