Runbook Automation

# Runbook: Service Name - Procedure Name

## Metadata
- Service: user-api
- Trigger: PagerDuty alert "High Error Rate"
- Owner: platform-team
- Last tested: 2024-01-15
- SLA impact: High — user-facing API

## Prerequisites
- kubectl access to production cluster
- PagerDuty acknowledgment of alert
- Access to Grafana dashboard: link

## Diagnosis
1. Check current error rate in Grafana: link
2. Check recent deployments: kubectl rollout history deployment/user-api -n production
3. Check pod logs: kubectl logs -l app=user-api -n production --tail=100

## Remediation Steps
### Option A: Restart pods (transient errors)
```
kubectl rollout restart deployment/user-api -n production
```

### Option B: Rollback (recent deployment)
```
kubectl rollout undo deployment/user-api -n production
```

### Option C: Scale up (capacity issue)
```
kubectl scale deployment/user-api --replicas=10 -n production
```

## Verification
- [ ] Error rate returns to normal within 5 minutes
- [ ] Latency p99 returns to baseline
- [ ] No customer reports after remediation

## Escalation
If remediation fails after 15 minutes, escalate to SRE lead.
Contact: #sre-escalation on Slack

automation_levels:
  level_0_documentation:
    description: "Human reads and executes manually"
    example: "Step-by-step wiki page"
    risk: "Low"
    frequency: "Rare operations"

  level_1_guided:
    description: "Script prompts for confirmation at each step"
    example: "Interactive shell script with read prompts"
    risk: "Low"
    frequency: "Weekly operations"

  level_2_semi_automated:
    description: "Script executes all steps, human approves start"
    example: "Ansible playbook with --check mode"
    risk: "Medium"
    frequency: "Daily operations"

  level_3_fully_automated:
    description: "Triggered by alert, no human in loop"
    example: "Auto-scaling, auto-restart controllers"
    risk: "High"
    frequency: "Continuous operations"

  level_4_autonomous:
    description: "Self-healing, learns from past incidents"
    example: "ML-based anomaly detection and response"
    risk: "Very high"
    frequency: "Rare — requires proven track record"

---
# playbooks/renew-certificate.yml
- name: Renew TLS Certificate
  hosts: localhost
  connection: local
  gather_facts: false

  vars:
    domain: "api.example.com"
    cert_path: "/etc/ssl/certs/{{ domain }}.pem"
    key_path: "/etc/ssl/private/{{ domain }}.key"
    renew_days_before: 30

  tasks:
    - name: Check certificate expiry
      command: |
        openssl x509 -enddate -noout -in {{ cert_path }}
      register: cert_expiry

    - name: Calculate days until expiry
      set_fact:
        days_remaining: >
          {{ ((cert_expiry.stdout | regex_search('\d{4}.*')) |
             to_datetime('%b %d %H:%M:%S %Y %Z') |
             ansible_date_time.epoch) | int }}

    - name: Renew certificate if expiring soon
      when: days_remaining|int < renew_days_before|int
      block:
        - name: Request new certificate
          command: |
            certbot renew --cert-name {{ domain }} --non-interactive
          register: cert_result

        - name: Reload nginx
          service:
            name: nginx
            state: reloaded

        - name: Send notification
          slack:
            token: "{{ slack_token }}"
            channel: "#ops-notifications"
            msg: "Certificate renewed for {{ domain }}"

safety_guards:
  rate_limit:
    max_actions_per_hour: 3
    cooldown_period: 300s

  blast_radius:
    max_pods_restarted: 10
    max_traffic_shift: 20%
    max_services_affected: 1

  approval_gate:
    required_for:
      - "database operations"
      - "certificate changes"
      - "dns modifications"
    approvers:
      - "sre-lead"
      - "on-call-engineer"

  auto_rollback:
    enabled: true
    trigger: "error_rate > 5% for 2m"
    action: "undo last change"

#!/usr/bin/env bash
# Test runbook automation in staging

set -euo pipefail

ENVIRONMENT="staging"
RUNBOOK="renew-certificate"

echo "=== Testing runbook: $RUNBOOK ==="
echo "Environment: $ENVIRONMENT"
echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Run in dry-run mode first
echo "--- Dry run ---"
ansible-playbook "playbooks/${RUNBOOK}.yml" \
  -e "environment=$ENVIRONMENT" \
  --check --diff

# Ask for confirmation
read -p "Proceed with actual run? (yes/no): " confirm
if [[ "$confirm" != "yes" ]]; then
  echo "Aborted."
  exit 0
fi

# Run for real
echo "--- Actual run ---"
ansible-playbook "playbooks/${RUNBOOK}.yml" \
  -e "environment=$ENVIRONMENT"

# Verify result
echo "--- Verification ---"
echo "Runbook test completed at $(date -u +%Y-%m-%dT%H:%M:%SZ)"