Stage 5 · Platform
Stateful Storage Operations
Storage Performance
IOPS limits, filesystem choices, fio testing, and latency dashboards for persistent disks.
Performance Factors
Storage performance depends on: IOPS (input/output operations per second), throughput (MB/s), latency (ms), filesystem choice, and access pattern. Understanding these factors helps you choose the right storage class for each workload.
| Factor | Database Workload | Log Workload | Media Workload |
|---|---|---|---|
| IOPS | High (10,000+) | Low (1,000) | Low (1,000) |
| Throughput | Medium (500 MB/s) | High (500 MB/s) | Very High (1 GB/s) |
| Latency | Critical (<1ms) | Not critical | Not critical |
| Volume Type | io2 | st1 | sc1 |
IOPS and Throughput
IOPS measures how many read/write operations a volume can handle per second. Throughput measures data transfer rate. Some volume types (gp3, io2) let you provision IOPS and throughput independently. Others (gp2) tie IOPS to volume size.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: high-iops
provisioner: ebs.csi.aws.com
parameters:
type: io2
iops: "20000"
encrypted: "true"
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: balanced
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "5000"
throughput: "250"
encrypted: "true"
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: throughput-optimized
provisioner: ebs.csi.aws.com
parameters:
type: st1
encrypted: "true"io2 provides the highest IOPS (up to 64,000). gp3 provides a good balance (up to 16,000 IOPS, 1,000 MB/s). st1 provides high throughput (500 MB/s) at low cost for sequential workloads like logs.
Filesystem Choices
The filesystem affects performance and features. ext4 is the default — reliable and well-tested. xfs provides better performance for large files and parallel I/O. Choose based on your workload characteristics.
| Filesystem | Best For | Pros | Cons |
|---|---|---|---|
| ext4 | General purpose | Mature, reliable | 256TB limit, slower large files |
| xfs | Large files, databases | Better parallel I/O, no size limit | No shrink, can't convert from ext4 |
| btrfs | Snapshots, compression | Built-in snapshots, compression | Less mature for production |
fio Testing
fio (Flexible I/O Tester) measures storage performance. Run fio inside a pod to benchmark IOPS, throughput, and latency. Compare results against your workload requirements.
# Run fio benchmark pod
kubectl run fio-test --image=ubuntu:22.04 --rm -it -- /bin/bash
# Inside the pod:
apt-get update && apt-get install -y fio
# Random read IOPS test
fio --name=randread --ioengine=libaio --iodepth=32 \
--rw=randread --bs=4k --direct=1 --size=1G --numjobs=4 \
--runtime=60 --time_based --group_reporting --filename=/data/test
# Sequential write throughput test
fio --name=seqwrite --ioengine=libaio --iodepth=16 \
--rw=write --bs=128k --direct=1 --size=1G --numjobs=1 \
--runtime=60 --time_based --group_reporting --filename=/data/testrandread tests random read IOPS (important for databases). seqwrite tests sequential write throughput (important for logs). --direct=1 bypasses OS cache for accurate results. --runtime=60 runs for 60 seconds.
Performance Monitoring
Monitor storage performance with node-exporter and CSI driver metrics. Key metrics: disk I/O latency, IOPS, throughput, queue depth, and filesystem utilization. Alert on latency spikes and capacity thresholds.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: storage-alerts
spec:
groups:
- name: storage.rules
rules:
- alert: HighDiskLatency
expr: |
histogram_quantile(0.99,
rate(node_disk_read_time_seconds_total[5m])
/ rate(node_disk_reads_completed_total[5m])
) > 0.01
for: 5m
labels:
severity: warning
annotations:
summary: "High disk read latency on {{ $labels.instance }}"
- alert: DiskSpaceLow
expr: |
(node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.15
for: 10m
labels:
severity: critical
annotations:
summary: "Disk space below 15% on {{ $labels.instance }}"HighDiskLatency fires when p99 read latency exceeds 10ms. DiskSpaceLow fires when available space drops below 15%. These alerts catch storage issues before they impact applications.
Optimization Strategies
- Use io2 for databases requiring high IOPS (10,000+)
- Use gp3 for balanced workloads (configurable IOPS/throughput)
- Use st1 for sequential workloads (logs, analytics)
- Use XFS for large files and parallel I/O
- Enable read-ahead for sequential workloads
- Use separate volumes for data and logs
- Monitor IOPS and throughput limits to avoid throttling
Monitor volume utilization with node_filesystem_avail_bytes. If volumes are consistently >80% full, expand them. If IOPS/throughput are consistently low, downgrade to a cheaper volume type. Right-sizing saves costs without sacrificing performance.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.