Stage 4 · Provision
Designing for High Availability
Runbook Automation
Turning incident response into executable code that runs consistently every time.
What Are Runbooks?
A runbook is a documented procedure for responding to a specific type of incident. It contains step-by-step instructions that any on-call engineer can follow, regardless of their familiarity with the system. Runbooks reduce mean time to recovery (MTTR) by eliminating guesswork during incidents.
Runbook Structure
# Runbook: High API Error Rate
## Alert
- Alert Name: HighErrorRate
- Threshold: > 5% error rate for 5 minutes
- Severity: P1
## Triage Steps
1. Check Grafana dashboard: /d/api-overview
2. Identify which endpoints are failing
3. Check upstream dependencies
## Resolution Steps
1. If database is slow:
- Check connection pool: kubectl exec -it db-pod -- psql -c "SELECT count(*) FROM pg_stat_activity"
- Scale connection pool: kubectl scale deployment api --replicas=5
2. If cache is down:
- Verify Redis: redis-cli -h redis.internal ping
- Restart Redis pod: kubectl rollout restart deployment redis
3. If dependency is down:
- Enable circuit breaker: curl -X POST http://api/config/circuit-breaker?enable=true
- Serve degraded responses
## Escalation
- If not resolved in 15 minutes: Page SRE team lead
- If data loss suspected: Page database teamEvery runbook should have: the alert that triggered it, triage steps to diagnose, resolution steps to fix, and escalation paths if the fix fails.
Write runbooks assuming the reader has zero context. At 3 AM during an outage, the on-call engineer may be disoriented. Every step should be explicit, with exact commands to copy and paste.
Automation Tools
Runbook automation transforms manual steps into automated workflows. Tools like Rundeck, StackStorm, and PagerDuty Automate can execute runbook steps programmatically, reducing human error and response time.
| Tool | Type | Strength |
|---|---|---|
| Rundeck | Job scheduler | Web UI, access control, audit logging |
| StackStorm | Event-driven automation | Triggers, sensors, actions |
| PagerDuty Automate | Incident response | Integrated with alerting |
| AWS Systems Manager | Cloud automation | Runbook execution on EC2/ECS |
| Custom scripts | Shell/Python | Full control, versioned |
Executable Runbooks
The most effective runbooks are executable — they are code that can be run directly. Write runbooks as scripts with parameterized inputs, logging, and rollback steps. Version control them alongside your application code.
#!/usr/bin/env bash
set -euo pipefail
SERVICE=$1
REPLICAS=$2
echo "[RUNBOOK] Scaling $SERVICE to $REPLICAS replicas"
# Pre-check
CURRENT=$(kubectl get deployment $SERVICE -o jsonpath='{.spec.replicas}')
echo "[INFO] Current replicas: $CURRENT"
# Execute
kubectl scale deployment $SERVICE --replicas=$REPLICAS
# Verify
echo "[WAIT] Waiting for rollout..."
kubectl rollout status deployment $SERVICE --timeout=120s
echo "[SUCCESS] $SERVICE scaled from $CURRENT to $REPLICAS"This script automates scaling a deployment. It logs every step, verifies the result, and can be called from any alerting system.
Integration with Alerting
Runbooks should be directly linked from alert definitions. When an engineer receives an alert, they should have a one-click link to the relevant runbook. PagerDuty, OpsGenie, and Grafana all support runbook URLs in alert annotations.
Runbooks make incidents shorter and less painful. They do not prevent incidents. The goal is to combine runbooks (reactive) with chaos engineering (proactive) to continuously improve reliability.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.