Stage 6 · Operate
Capacity & Performance Engineering
Capacity Planning
Growth models, headroom targets, quota limits, and seasonality forecasting — staying ahead of demand.
Capacity Planning Basics
Capacity planning answers one question: when will we run out of resources? It combines current usage, growth rate, and resource limits to predict when you need to scale. Do this proactively, not reactively.
Capacity planning is not a one-time activity. Traffic grows, features change, and user behavior evolves. Revisit your capacity plan quarterly at minimum, and after any major launch.
Growth Models
Model your growth based on historical data and business projections. Linear growth is easiest to predict. Exponential growth requires more attention. Step-function growth (product launches) requires planning spikes.
growth_models:
linear:
description: "Steady, predictable growth"
formula: "capacity_needed = current + (growth_rate * months)"
example:
current_rps: 1000
growth_rate: 100 # 100 new RPS per month
months: 6
projected: 1600 # RPS needed in 6 months
exponential:
description: "Accelerating growth"
formula: "capacity_needed = current * (1 + growth_rate)^months"
example:
current_rps: 1000
growth_rate: 0.1 # 10% monthly growth
months: 6
projected: 1772 # RPS needed in 6 months
step_function:
description: "Sudden jumps at known dates"
triggers:
- "Product launch: 3x traffic spike"
- "Marketing campaign: 2x traffic spike"
- "Seasonal event: 5x traffic spike"
planning: "Pre-scale 1 week before expected spike"
historical:
description: "Based on past patterns"
data_source: "Last 12 months of traffic data"
approach: "Fit curve to historical data, extrapolate"
tool: "Prometheus query + spreadsheet"Headroom Targets
Headroom is the gap between your current usage and your resource limits. It absorbs traffic spikes and prevents outages. Set headroom targets based on your traffic variability and growth rate.
headroom_targets:
cpu:
target: "30-50% headroom"
rationale: "CPU spikes are sudden and short-lived"
alert: "Alert when headroom < 20%"
action: "Scale up or optimize"
memory:
target: "20-30% headroom"
rationale: "Memory grows gradually, OOM kills are sudden"
alert: "Alert when headroom < 15%"
action: "Scale up or investigate leaks"
disk:
target: "30% headroom"
rationale: "Disk fills up slowly, then suddenly"
alert: "Alert when headroom < 25%"
action: "Clean up or expand"
connections:
target: "40% headroom"
rationale: "Connection pools exhaust during traffic spikes"
alert: "Alert when headroom < 30%"
action: "Increase pool size or optimize queries"
network:
target: "50% headroom"
rationale: "Network saturation causes cascading failures"
alert: "Alert when headroom < 40%"
action: "Scale up or optimize"Quota Management
Quotas limit resource consumption per team or service. They prevent runaway costs and resource exhaustion. Automate quota management to avoid manual approval processes that slow down development.
apiVersion: v1
kind: ResourceQuota
metadata:
name: production-quota
namespace: production
spec:
hard:
requests.cpu: "32"
requests.memory: "64Gi"
limits.cpu: "64"
limits.memory: "128Gi"
pods: "100"
services: "20"
persistentvolumeclaims: "10"
---
# Automatically increase quota when usage approaches limit
apiVersion: v1
kind: ConfigMap
metadata:
name: quota-controller
namespace: kube-system
data:
config.yaml: |
quotas:
- namespace: production
threshold: 0.8 # Increase when usage > 80%
increase:
requests.cpu: "8"
requests.memory: "16Gi"
max:
requests.cpu: "128"
requests.memory: "256Gi"
notify: "#capacity-alerts"Seasonality Forecasting
Many services have seasonal patterns. E-commerce peaks on Black Friday. Streaming peaks in the evening. B2B peaks on weekday mornings. Identify your seasonal patterns and pre-scale before they hit.
seasonal_scaling:
patterns:
weekday_peak:
schedule: "0 8 * * 1-5" # Weekdays 8 AM
scale_to: 2.0 # 2x normal capacity
duration: "10h"
evening_peak:
schedule: "0 18 * * *" # Every day 6 PM
scale_to: 1.5
duration: "6h"
weekend_valley:
schedule: "0 0 * * 0,6" # Weekends midnight
scale_to: 0.5 # 50% normal capacity
duration: "24h"
events:
black_friday:
date: "2024-11-29"
pre_scale_days: 7
scale_to: 5.0
scale_down_days: 3
holiday_sale:
date: "2024-12-15"
pre_scale_days: 3
scale_to: 3.0
scale_down_days: 2Capacity Dashboard
Build a capacity dashboard that shows current utilization, headroom, growth trends, and projected exhaustion dates. This dashboard is your early warning system for capacity issues.
dashboard_panels:
- title: "Current Utilization"
metrics:
- "CPU: {{ cluster:cpu:utilization:ratio }}"
- "Memory: {{ cluster:memory:utilization:ratio }}"
- "Disk: {{ cluster:disk:utilization:ratio }}"
- "Connections: {{ cluster:connections:utilization:ratio }}"
- title: "Headroom Remaining"
metrics:
- "CPU headroom: {{ 1 - cluster:cpu:utilization:ratio }}"
- "Memory headroom: {{ 1 - cluster:memory:utilization:ratio }}"
- title: "Days Until Capacity"
query: "headroom / daily_growth_rate"
alert: "When < 30 days"
- title: "Growth Trend (6 months)"
type: "timeseries"
metrics:
- "cluster:cpu:utilization:daily"
- "cluster:memory:utilization:daily"
- title: "Seasonal Pattern"
type: "heatmap"
metric: "cluster:traffic:hourly:daily"Capacity planning is not just technical. It requires input from product, marketing, and sales. Know about upcoming launches, campaigns, and events. Capacity surprises come from business decisions, not technical failures.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.