Stage 3 · Build
Networking Stack Deep Dive
TCP/IP Implementation
Connection establishment, congestion control (BBR, CUBIC), and tuning the Linux TCP stack.
TCP State Machine
TCP connections progress through a series of states: LISTEN, SYN_SENT, SYN_RECEIVED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, TIME_WAIT, CLOSE_WAIT, and CLOSED. Understanding these states is essential for diagnosing connection issues.
# Count connections by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
# 150 ESTAB
# 23 TIME-WAIT
# 5 LISTEN
# 2 CLOSE-WAIT
# Detailed connection info
ss -tan state established
# State Recv-Q Send-Q Local Address:Port Peer Address:Port
# ESTAB 0 0 10.0.0.1:22 10.0.0.2:54321TIME_WAIT is normal and expected for recently closed connections. CLOSE_WAIT indicates the remote side closed but the local application has not — this often signals a bug.
Connection Establishment
TCP uses a three-way handshake: SYN, SYN-ACK, ACK. The kernel handles this in the TCP code, and the socket API is notified via accept(). Understanding the handshake helps diagnose connection latency and failures.
# Measure connection establishment time
time curl -s -o /dev/null http://localhost:8080
# real 0m0.023s
# Trace the handshake with strace
strace -e connect curl -s http://localhost:8080
# Check SYN backlog
ss -ltn | head -5
# State Recv-Q Send-Q Local Address:Port
# LISTEN 128 128 0.0.0.0:80
# SYN cookies prevent SYN floods
sysctl net.ipv4.tcp_syncookies
# 1 = enabledThe Recv-Q column for LISTEN sockets shows the current SYN backlog. If it reaches Send-Q, new connections are dropped.
Congestion Control
Congestion control algorithms determine how TCP adapts to network conditions. Linux supports several algorithms: CUBIC (default), BBR, Reno, and others. BBR is Google's model-based algorithm that works particularly well on lossy or high-latency networks.
# Check available algorithms
sysctl net.ipv4.tcp_congestion_control
# net.ipv4.tcp_congestion_control = cubic
# List available algorithms
sysctl net.ipv4.tcp_available_congestion_control
# cubic reno
# Switch to BBR
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.core.default_qdisc=fq
# Make persistent
echo "net.ipv4.tcp_congestion_control = bbr" >> /etc/sysctl.conf
echo "net.core.default_qdisc = fq" >> /etc/sysctl.confBBR works best with fq qdisc. CUBIC is the default and works well for most scenarios. BBR excels on wireless, long-distance, and lossy links.
| Algorithm | Approach | Best For |
|---|---|---|
| CUBIC | Loss-based, window growth | Standard wired networks |
| BBR | Model-based, bandwidth/RTT | Lossy/high-latency links |
| Reno | Loss-based, additive increase | Legacy, rarely used |
| DCTCP | ECN-based | Data center networks |
TCP Tuning
Linux exposes dozens of TCP tuning parameters via sysctl. The most impactful are buffer sizes, keepalive settings, and queue limits.
# Increase TCP buffer sizes (min, default, max in bytes)
sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
# Increase socket buffer limits
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
# Increase backlog for busy servers
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
# Keepalive tuning (detect dead connections)
sysctl -w net.ipv4.tcp_keepalive_time=600
sysctl -w net.ipv4.tcp_keepalive_intvl=30
sysctl -w net.ipv4.tcp_keepalive_probes=3tcp_rmem and tcp_wmem control per-socket buffer sizes. The kernel auto-tunes within these bounds based on memory pressure and network conditions.
TCP Memory Management
The kernel manages TCP memory globally and per-socket. When memory pressure occurs, the kernel reduces buffer sizes and may drop packets. Understanding these limits prevents silent performance degradation.
# Global TCP memory usage
cat /proc/net/sockstat
# TCP: inuse 123 orphan 45 tw 67 alloc 89 mem 12
# TCP memory limits (pages)
sysctl net.ipv4.tcp_mem
# 188734 251646 377468
# Per-socket memory usage
ss -tm | head -10
# skmem:(r128,rb131072,t0,tb46080,f0,w0,o0,bl0,d0)The three values in tcp_mem are: low (pressure off), pressure (pressure on), and high (maximum). When memory exceeds the high threshold, new allocations fail.
TCP Diagnostics
# Retransmission statistics (indicates packet loss)
nstat -a | grep Retrans
# TcpRetransSegs 1234
# RTT statistics
ss -ti | head -5
# cubic wscale:7,7 rto:204 rtt:1.5/0.75 ato:40
# Check for zero-window events (receiver overwhelmed)
nstat -a | grep -i "zero"
# TcpExtTCPZeroWindowDrop 0
# Monitor connection rate
sar -n TCP 1 5
# active/s passive/s retrans/sHigh retransmission rates indicate network congestion or packet loss. Check with mtr or ping to identify where loss occurs.
ss is faster and provides more TCP internal details than netstat. Use ss -ti for TCP diagnostics, ss -s for summary statistics, and ss -m for memory usage.
Google reports BBR improves throughput by 2-25% on most networks. It is especially beneficial for long-distance links, mobile networks, and situations with non-congestion packet loss. Consider BBR as the default for production servers.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.