Stage 3 · Build
Storage Reliability & Performance
I/O Observability
Using iostat, pidstat, blktrace, and eBPF tools to isolate storage bottlenecks.
iostat Analysis
iostat is the primary tool for device-level I/O analysis. It shows throughput, IOPS, latency, and utilization for each block device. Understanding iostat output is the first step in diagnosing storage issues.
# Extended stats, 1-second intervals
iostat -xz 1
# Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
# sda 120.00 245.00 4800.00 9800.00 5.00 45.00 4.00 15.51 1.23 3.45 0.89 40.00 40.00 1.67 60.80
# nvme0n1 12000.00 8000.00 480000.00 320000.00 0.00 0.00 0.00 0.00 0.45 0.78 15.67 40.00 40.00 0.05 97.50
# Key metrics to watch:
# %util — Device utilization (close to 100% = saturated)
# r_await/w_await — Latency per request (lower is better)
# aqu-sz — Average queue depth (higher = more pressure)
# rkB/s/wkB/s — Throughput%util is misleading for multi-queue devices like NVMe. Use r_await and w_await for true latency assessment. NVMe drives can show 99% util with low latency.
pidstat I/O Tracking
# I/O per process, 1-second intervals
pidstat -d 1
# UID PID kB_rd/s kB_wr/s kB_ccwr/s Command
# 1000 1234 45678.00 12345.00 0.00 postgres
# 1000 1235 12345.00 45678.00 0.00 postgres
# I/O per thread
pidstat -dt -p 1234 1
# I/O with disk name
pidstat -d -p ALL 1 | sort -nrk 5 | head -10pidstat -d shows per-process I/O rates. Sort by kB_wr/s or kB_rd/s to find the heaviest I/O consumers.
blktrace Deep Dive
blktrace traces block layer events with per-request detail. It shows exactly when requests are queued, dispatched, completed, and merged. This is the deepest level of I/O analysis.
# Record block I/O events
sudo blktrace -d /dev/sda -w 10 -o trace
# Analyze with blkparse
sudo blkparse -i trace.blktrace.0 | head -30
# 8,0 1 1 0.000000000 1234 Q WS 23456789 + 8 [postgres]
# 8,0 1 2 0.000001234 1234 G WS 23456789 + 8 [postgres]
# 8,0 1 3 0.000002345 1234 I WS 23456789 + 8 [postgres]
# 8,0 1 4 0.000003456 1234 D WS 23456789 + 8 [postgres]
# 8,0 1 5 0.005678901 1234 C WS 23456789 + 8 [0]
# Event types:
# Q = Queued, G = Get request, I = Inserted
# D = Dispatched, C = Completed, M = Merged
# Generate summary
sudo blkparse -i trace.blktrace.0 -d trace.bin
sudo btt -i trace.bin
# D2C (/dev/sda) Q2C (/dev/sda)
# 0.12ms 0.45msblktrace shows the complete lifecycle of each I/O request. Q2C (queue to complete) is the total latency. D2C (dispatch to complete) is the device latency.
eBPF I/O Tools
# Trace block I/O with latency
sudo biolatency -D
# Disk = /dev/sda
# usecs : count distribution
# 0 -> 1 : 0 | |
# 2 -> 3 : 12 | |
# 4 -> 7 : 456 |*** |
# 8 -> 15 : 2345 |************* |
# 16 -> 31 : 12345 |************************************************|
# Trace I/O size distribution
sudo biosize
# Trace I/O per process
sudo biotop
# PID COMM D MAJ MIN DISK I/O Kbytes AVGms
# 1234 postgres R 8 0 sda 156 6240 1.23
# Trace block I/O by latency
sudo biolatency -D -QeBPF I/O tools provide real-time, per-process visibility without the overhead of blktrace. Use them for quick diagnosis.
iotop for Process I/O
# Interactive I/O monitor
sudo iotop -oP
# Total DISK READ: 12.34 M/s | Total DISK WRITE: 45.67 M/s
# PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
# 1234 be/4 postgres 5.67 M/s 12.34 M/s 0.00 % 85.23 % postgres
# Batch mode for logging
sudo iotop -botqq --iter=60 > iotop.log
# Only show processes doing I/O
sudo iotop -oiotop shows per-process I/O in real-time. The -o flag shows only processes actually doing I/O, reducing noise.
I/O Analysis Workflow
- 1. Start with iostat -xz 1 to identify the saturated device
- 2. Use pidstat -d 1 to find which process is causing the I/O
- 3. Use iotop for real-time process-level I/O visibility
- 4. Use biolatency to understand I/O latency distribution
- 5. Use blktrace for deep per-request analysis (last resort)
- 6. Check scheduler and queue depth for optimization opportunities
iostat -xz shows extended device statistics with inactive devices filtered out. Focus on %util, r_await, and w_await to identify bottlenecks quickly.
blktrace captures every block I/O event. For busy servers, limit capture time with -w 10 (10 seconds). Analyze with blkparse -d and btt for summaries.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.