Runbook Quality

# Runbook: [Service] - [Issue]

## Metadata
- Service: [name]
- Alert: [alert name]
- Owner: [team]
- Last tested: [date]
- SLA impact: [High/Medium/Low]

## Prerequisites
- [ ] kubectl access to [cluster]
- [ ] PagerDuty acknowledgment
- [ ] Dashboard link: [URL]

## Diagnosis Steps
### Step 1: Check current state
```
kubectl get pods -l app=[service] -o wide
kubectl top pods -l app=[service]
```
**Expected:** All pods Running, CPU < 80%, Memory < 80%
**If not:** Go to Remediation Option A

### Step 2: Check recent changes
```
kubectl rollout history deployment/[service]
kubectl get events -n [namespace] --sort-by=.lastTimestamp | tail -20
```
**Expected:** No recent deployments, no unusual events
**If recent deploy:** Go to Remediation Option B

### Step 3: Check logs
```
kubectl logs -l app=[service] --tail=100 --prefix
```
**Look for:** ERROR, WARN, exception stack traces
**If errors found:** Go to Remediation Option C

## Remediation Options
### Option A: Restart pods
```
kubectl rollout restart deployment/[service] -n [namespace]
```
**Verify:** Error rate drops within 5 minutes

### Option B: Rollback
```
kubectl rollout undo deployment/[service] -n [namespace]
```
**Verify:** Error rate drops within 5 minutes

### Option C: Scale up
```
kubectl scale deployment/[service] --replicas=[N] -n [namespace]
```
**Verify:** New pods become Ready within 2 minutes

## Verification
- [ ] Error rate < 1% in Grafana
- [ ] Latency p99 < [threshold]
- [ ] All pods Running and Ready
- [ ] No new alerts firing

## Escalation
If no resolution after 15 minutes:
1. Escalate to [team lead]
2. Post in #sre-escalation
3. Update status page

# Pod status and resource usage
kubectl get pods -l app=SERVICE -n NAMESPACE -o wide
kubectl top pods -l app=SERVICE -n NAMESPACE

# Logs with timestamps and context
kubectl logs -l app=SERVICE -n NAMESPACE --tail=100 --prefix --timestamps
kubectl logs -l app=SERVICE -n NAMESPACE --since=30m | grep -i error

# Events (recent activity)
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp | tail -20
kubectl get events -n NAMESPACE --field-selector type=Warning

# Deployment status
kubectl rollout status deployment/SERVICE -n NAMESPACE
kubectl rollout history deployment/SERVICE -n NAMESPACE

# Restart pods
kubectl rollout restart deployment/SERVICE -n NAMESPACE

# Scale deployment
kubectl scale deployment/SERVICE --replicas=N -n NAMESPACE

# Rollback
kubectl rollout undo deployment/SERVICE -n NAMESPACE
kubectl rollout undo deployment/SERVICE --to-revision=N -n NAMESPACE

# Exec into pod for debugging
kubectl exec -it deployment/SERVICE -n NAMESPACE -- /bin/sh

# Port forward for local debugging
kubectl port-forward svc/SERVICE 8080:8080 -n NAMESPACE

# Check configmaps and secrets
kubectl get configmap -n NAMESPACE
kubectl describe configmap CONFIG -n NAMESPACE

rollback_checklist:
  before_rollback:
    - "Confirm the issue is user-impacting"
    - "Check recent deployments: kubectl rollout history"
    - "Verify rollback target is safe"
    - "Notify the team in war room"

  during_rollback:
    - "Execute: kubectl rollout undo deployment/SERVICE"
    - "Monitor rollout status: kubectl rollout status"
    - "Watch error rate in Grafana"

  after_rollback:
    - "Verify error rate returns to baseline"
    - "Verify latency returns to baseline"
    - "Check for any new issues"
    - "Update status page if applicable"
    - "Notify team of resolution"

  verification_queries:
    error_rate: |
      sum(rate(http_request_duration_seconds_count{code=~"5.."}[5m]))
      /
      sum(rate(http_request_duration_seconds_count[5m]))
    latency_p99: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
      )

verification_queries:
  high_error_rate:
    query: |
      sum(rate(http_request_duration_seconds_count{code=~"5.."}[5m]))
      /
      sum(rate(http_request_duration_seconds_count[5m]))
    expected: "< 0.01 (less than 1%)"

  high_latency:
    query: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket{code!~"5.."}[5m])) by (le)
      )
    expected: "< 0.2 (less than 200ms)"

  pod_health:
    query: |
      sum(kube_pod_status_ready{namespace="NAMESPACE", condition="true"})
      /
      sum(kube_pod_status_status{namespace="NAMESPACE"})
    expected: "= 1.0 (100% pods ready)"

  disk_space:
    query: |
      node_filesystem_avail_bytes{mountpoint="/"}
      /
      node_filesystem_size_bytes{mountpoint="/"}
    expected: "> 0.2 (more than 20% free)"

  certificate_expiry:
    query: |
      (probe_ssl_earliest_cert_expiry - time()) / 86400
    expected: "> 30 (more than 30 days)"

automation_hooks:
  diagnose:
    script: "scripts/diagnose-service.sh"
    args:
      - "--service=SERVICE"
      - "--namespace=NAMESPACE"
      - "--output=json"
    output:
      pods_ready: ".pods.ready"
      cpu_usage: ".resources.cpu"
      memory_usage: ".resources.memory"
      recent_deploy: ".deployment.latest"

  remediate_restart:
    script: "scripts/restart-deployment.sh"
    args:
      - "--service=SERVICE"
      - "--namespace=NAMESPACE"
      - "--verify=true"
    safety:
      max_restarts_per_hour: 3
      requires_approval: false

  remediate_rollback:
    script: "scripts/rollback-deployment.sh"
    args:
      - "--service=SERVICE"
      - "--namespace=NAMESPACE"
      - "--verify=true"
    safety:
      requires_approval: true
      approvers: ["sre-lead"]

  verify:
    script: "scripts/verify-service-health.sh"
    args:
      - "--service=SERVICE"
      - "--namespace=NAMESPACE"
      - "--thresholds=error_rate:0.01,latency_p99:0.2"