Stage 5 · Platform
Production Troubleshooting & Operations
Resource Pressure
CPU throttling, memory pressure, ephemeral storage eviction, PSI metrics, and QoS classes.
CPU Throttling
CPU throttling happens when a container exceeds its CPU limit. Unlike memory, exceeding CPU limits doesn't kill the container — it slows it down. This is invisible in average CPU usage metrics but causes latency spikes.
# Check throttling metrics
kubectl exec -it prometheus -- promtool query instant \
'rate(container_cpu_cfs_throttled_periods_total{pod="my-pod"}[5m]) /
rate(container_cpu_cfs_periods_total{pod="my-pod"}[5m])'
# Check container CPU usage vs limits
kubectl top pod my-pod --containers
# Check cgroup CPU stats
kubectl exec my-pod -- cat /sys/fs/cgroup/cpu/cpu.statThrottling ratio = throttled periods / total periods. A ratio > 0.1 (10%) indicates significant throttling. Check if CPU requests are too low or limits are too restrictive.
Memory Pressure
Memory pressure occurs when a node runs low on memory. kubelet evicts pods to free memory. Eviction order: BestEffort first, then Burstable, then Guaranteed. Pods with lower memory usage relative to limits are evicted first.
# Check node memory conditions
kubectl describe node worker-1 | grep -A 5 Conditions
# Check pod memory usage
kubectl top pod --sort-by=memory
# Check memory requests vs actual usage
kubectl get pods -o json | jq '.items[] | {
name: .metadata.name,
memoryRequest: .spec.containers[0].resources.requests.memory,
memoryLimit: .spec.containers[0].resources.limits.memory
}'MemoryPressure condition is True when the node is low on memory. kubelet evicts pods to free memory. Check which pods are using the most memory and whether their limits are appropriate.
Ephemeral Storage Eviction
Ephemeral storage (emptyDir, container logs, writable layers) has limits. When usage exceeds the limit, kubelet evicts the pod. Default limit: node capacity - reserved. Monitor /var/log and emptyDir usage.
apiVersion: v1
kind: Pod
metadata:
name: storage-heavy
spec:
containers:
- name: app
resources:
limits:
ephemeral-storage: "2Gi" # Limit writable layer
volumeMounts:
- name: cache
mountPath: /cache
volumes:
- name: cache
emptyDir:
sizeLimit: 1Gi # Limit emptyDir sizeephemeral-storage limits the total writable layer + logs + emptyDir mounts. If the pod exceeds this limit, kubelet evicts it. Set limits proportional to the application's storage needs.
PSI Metrics
PSI (Pressure Stall Information) measures resource pressure. It reports the percentage of time tasks are stalled waiting for a resource. PSI is more accurate than utilization metrics for detecting pressure.
# CPU pressure
kubectl exec my-pod -- cat /proc/pressure/cpu
# Memory pressure
kubectl exec my-pod -- cat /proc/pressure/memory
# I/O pressure
kubectl exec my-pod -- cat /proc/pressure/io
# Output format:
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0PSI shows: some (at least one task stalled) and full (all tasks stalled). avg10/60/300 are 10s/60s/5min averages. Any non-zero value indicates pressure. full > 0 means all tasks are stalled.
QoS and Eviction
QoS classes determine eviction priority. BestEffort pods are evicted first, then Burstable, then Guaranteed. Set requests equal to limits for Guaranteed QoS to maximize survival under pressure.
# BestEffort — evicted first
# No requests or limits
# Burstable — evicted second
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "512Mi"
# Guaranteed — evicted last (safest)
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "500m" # Must equal request
memory: "512Mi" # Must equal requestGuaranteed QoS: all containers have requests equal to limits for both CPU and memory. This gives the highest survival priority. Burstable: at least one container has requests different from limits. BestEffort: no requests or limits.
Right-Sizing Workloads
Right-sizing sets requests and limits to actual usage. Over-provisioning wastes resources. Under-provisioning causes throttling and eviction. Use VPA recommendations and monitoring data to right-size.
# Check actual usage vs requests
kubectl top pod --sort-by=cpu
# VPA recommendations (if VPA is installed)
kubectl describe vpa my-app-vpa
# Manual analysis — find pods using less than 50% of requests
kubectl get pods -o json | jq '.items[] |
select(.spec.containers[0].resources.requests.cpu != null) |
{
name: .metadata.name,
cpuRequest: .spec.containers[0].resources.requests.cpu,
memoryRequest: .spec.containers[0].resources.requests.memory
}'If actual usage is consistently < 50% of requests, the pod is over-provisioned. If usage is consistently > 80% of limits, the pod is under-provisioned. Right-size requests to actual usage + 20% buffer.
Monitor: node_memory_available, node_fs_available, pod_cpu_usage vs limits, pod_memory_usage vs limits, and container_cpu_cfs_throttled_periods. Set alerts for when any metric crosses 80% threshold.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.