Packet Flow Through the Kernel

The Packet Receive Path

When a network packet arrives at the NIC, it travels through several layers before reaching your application: DMA to ring buffer, interrupt, softirq, netfilter, socket buffer, and finally the read() call. Understanding this path is essential for network performance tuning.

• 1. NIC DMA — Packet is DMA'd into a ring buffer in RAM
• 2. Interrupt — NIC raises an interrupt to notify the CPU
• 3. NAPI — Kernel switches to polling mode under load
• 4. GRO — Generic Receive Offload merges small packets
• 5. Netfilter — iptables/nftables rules are applied
• 6. Socket — Packet is delivered to the application socket buffer
• 7. Application — recv()/read() copies data to userspace

RX and TX Rings

The NIC uses DMA ring buffers to transfer packets between device memory and main memory. The ring is a fixed-size circular buffer: the NIC writes incoming packets, and the kernel reads them. The ring size determines how many packets can be buffered before drops occur.

# View ring buffer sizes
ethtool -g eth0
# Ring parameters for eth0:
# Pre-set maximums:
# RX:     4096
# TX:     4096
# Current hardware settings:
# RX:     256
# TX:     256

# Increase ring buffers for high-throughput
ethtool -G eth0 rx 4096 tx 4096

# Check multi-queue support
ethtool -l eth0
# Combined:     8

NAPI Polling

NAPI (New API) prevents interrupt storms under high packet rates. Instead of generating an interrupt per packet, the NIC generates one interrupt, then the kernel polls for packets in a loop. This dramatically reduces overhead at high packet rates.

# View current coalescing settings
ethtool -c eth0
# rx-usecs: 3
# rx-frames: 0
# tx-usecs: 3
# tx-frames: 0
# adaptive-rx: on

# Disable adaptive and set fixed coalescing
ethtool -C eth0 adaptive-rx off adaptive-tx off
ethtool -C eth0 rx-usecs 50 rx-frames 25

# For low-latency workloads
ethtool -C eth0 rx-usecs 0 rx-frames 1

XDP: eBPF at the NIC

XDP (eXpress Data Path) runs eBPF programs at the earliest point in the receive path — before the kernel even allocates an sk_buff. This enables wire-speed packet processing for load balancers, firewalls, and DDoS mitigation.

# Check if NIC supports XDP native mode
ethtool -i eth0 | grep driver
# ixgbe (supports XDP)

# List attached XDP programs
ip link show eth0
# ... xdp qdisc noqueue ...

# Check XDP drop statistics
bpftool prog list | grep xdp

# Simple XDP drop program (bpftrace)
sudo bpftrace -e 'xdp { printf("packet on %s\n", comm); }'

The Transmit Path

The transmit path mirrors receive but in reverse. The application writes to a socket, the kernel builds an sk_buff, routes it, applies netfilter rules, queues it via the traffic control (qdisc) layer, and finally the NIC transmits via DMA.

# Check queueing discipline
tc qdisc show dev eth0
# qdisc fq 0: root refcnt 2 limit 10000p

# View transmission statistics
ip -s link show eth0
# TX: bytes packets errors dropped overrun mcast
# 1234567890 1234567      0     123      0      0

# Check for drops in the stack
cat /proc/net/softnet_stat
# Each line is a CPU: processed, dropped, time_squeeze

Performance Bottlenecks

Network performance issues usually fall into a few categories: interrupt saturation, softirq overload, socket buffer limits, or application read/write overhead. Identifying which layer is the bottleneck determines your tuning approach.

# Check for packet drops at each layer
cat /proc/net/softnet_stat  # Kernel drops
ethtool -S eth0 | grep drop  # NIC drops
ss -s                         # Socket statistics
nstat -a | grep Drop          # Stack drops

# Monitor interrupt distribution
cat /proc/interrupts | grep eth

# Check socket buffer limits
sysctl net.core.rmem_max net.core.wmem_max
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem