Stage 3 · Build
Networking Stack Deep Dive
Network Performance Tuning
Buffer sizing, RSS/RPS/RFS, XDP, and bypassing the kernel for wire-speed networking.
Buffer Sizing
Network buffer sizing directly impacts throughput and latency. Too small and packets are dropped during bursts. Too large and you get bufferbloat — high latency from packets sitting in oversized queues.
# Bandwidth-Delay Product for optimal buffer sizing
# BDP = bandwidth * RTT
# Example: 10 Gbps link, 1ms RTT = 1.25 MB
# Set TCP buffer sizes (min, default, max)
sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
# Socket buffer limits
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=2097152
# Network device backlog
sysctl -w net.core.netdev_max_backlog=30000For 10 Gbps with 0.1ms RTT, the optimal buffer is about 125 KB. Set tcp_rmem max to at least this value. The kernel auto-tunes within these bounds.
RSS, RPS, and RFS
RSS (Receive Side Scaling) distributes packet processing across multiple CPU cores using hardware queues. RPS (Receive Packet Steering) is a software fallback for NICs without multi-queue support. RFS (Receive Flow Steering) directs packets to the CPU running the application.
# Check RSS queues
ethtool -l eth0
# Combined: 8
# Set RSS to use all available CPUs
ethtool -L eth0 combined 8
# Enable RPS for software steering
echo "ff" > /sys/class/net/eth0/queues/rx-0/rps_cpus
# ff = use CPUs 0-7
# Enable RFS for application-aware steering
sysctl -w net.core.rps_sock_flow_entries=32768
echo 32768 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
# Check XPS (Transmit Packet Steering)
cat /sys/class/net/eth0/queues/tx-0/xps_cpus
# 0-7RSS distributes packets in hardware. RPS adds software distribution for NICs without multi-queue. RFS ensures packets arrive on the CPU where the application is running, reducing cross-CPU cache misses.
XDP Performance
XDP processes packets at the driver level, before the kernel allocates sk_buff. This enables millions of packets per second for simple operations like filtering, forwarding, or load balancing.
# Check XDP support
ethtool -i eth0 | grep driver
# ixgbe (supports native XDP)
# Load an XDP program (example: drop all)
sudo ip link set dev eth0 xdp obj drop.o sec xdp
# Measure XDP performance
sudo xdp-bench -i eth0 -n 10
# rx: 14.8M pps
# drop: 14.8M pps
# Remove XDP program
sudo ip link set dev eth0 xdp offXDP native mode processes 14-20M pps on modern hardware. Generic mode achieves 2-4M pps. For maximum performance, use native XDP with a supported driver.
Kernel Bypass (DPDK)
DPDK (Data Plane Development Kit) bypasses the kernel entirely, giving applications direct access to NIC hardware. This eliminates all kernel networking overhead for maximum throughput and minimum latency.
# Check DPDK support
lsmod | grep uio
# uio_pci_generic or vfio-pci
# Bind NIC to DPDK driver
sudo dpdk-devbind.py --bind=vfio-pci 0000:01:00.0
# Check DPDK stats
sudo dpdk-devbind.py --status
# DPDK performance: 100M+ pps
# Kernel networking: 5-20M pps
# XDP: 14-20M ppsDPDK is used for NFV, load balancers, and packet processing. It requires dedicated NICs and significant CPU resources. Use it only when kernel networking cannot meet your requirements.
TCP Offload Features
# Check offload features
ethtool -k eth0 | grep -E "tcp-segmentation|checksum|generic-segmentation"
# tcp-segmentation-offload: on
# tx-checksumming: on
# generic-segmentation-offload: on
# Enable/disable specific offloads
sudo ethtool -K eth0 tso on # TCP segmentation offload
sudo ethtool -K eth0 gro on # Generic receive offload
sudo ethtool -K eth0 lro off # Disable large receive offload
sudo ethtool -K eth0 tx-checksum-ipv4 on
# Check current GRO size
ethtool -g eth0TSO lets the NIC segment large TCP segments, reducing CPU work. GRO merges small incoming packets, reducing per-packet overhead. Both should generally be enabled.
Network Benchmarking
# iperf3 — throughput testing
iperf3 -s # Server
iperf3 -c <server> -t 10 # Client, 10 second test
iperf3 -c <server> -P 4 # 4 parallel streams
# netperf — latency under load
netperf -H <server> -t TCP_RR -l 10
# TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) to 10.0.0.2 (10.0.0.2)
# Socket Size Request Resp. Elapsed Trans.
# Send Recv Size Size Time Rate
# 16384 87380 64 64 10.00 25000.00
# nping — packet-level testing
sudo nping --rate 10000 -c 100000 <target>Use iperf3 for throughput, netperf for latency, and nping for packet-level testing. Always test from multiple source IPs to identify bottlenecks.
Single-stream TCP tests often underperform because they are limited by congestion window growth. Use iperf3 -P 4 or multiple netperf instances to saturate the link.
Measure baseline performance before any tuning. Use sar -n DEV 1, ss -s, and /proc/net/softnet_stat to identify the actual bottleneck. Tuning the wrong layer wastes time and may make things worse.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.