The 3am Page That Taught Me Alerts Need Runbooks, Not Just Thresholds

The page said: "PodCrashLoopBackOff — payments-worker — namespace: prod." That's it. Correct alert, correct threshold, correct routing — it woke me up exactly when it should have. And I still spent the first 15 minutes of a 3am page doing something no on-call engineer should ever have to do: figuring out from scratch what a crash-looping payments-worker pod even means for the business, and what I was allowed to safely do about it.

What actually happened

A payments-worker pod had started crash-looping after a dependency (an internal fraud-check service) started returning malformed responses under load. The worker's client library didn't handle the malformed response gracefully — it panicked, Kubernetes restarted it, it panicked again. Textbook crash loop. The fix, in hindsight, took about 90 seconds: scale down the worker to stop hammering the broken dependency, page the fraud-check team, and let payments queue safely rather than process incorrectly.

But at 3am, half-awake, alone, with no context, here's what actually happened instead: I spent 10 minutes confirming it wasn't safe to just restart-and-hope (it wasn't — it would keep crash-looping). Then 15 minutes reading through the worker's code trying to understand what it does and whether scaling it to zero was safe (is it processing anything that can't be re-queued? I genuinely didn't know). Then I paged the fraud-check team's on-call, who confirmed the malformed-response issue in about 2 minutes because they already knew about it internally. Total time to actual resolution: 40 minutes. Total time that mattered: about 90 seconds, once someone with the right context was involved.

flowchart LR
    A[fraud-check service] -->|malformed response under load| B[payments-worker]
    B -->|panics, restarts| B
    B -.->|PodCrashLoopBackOff| C[Kubernetes]
    C -->|alert| D[PagerDuty: SRE on-call]
    D -.->|"no runbook: has to page fraud-check team from scratch"| E[fraud-check on-call]
    E -->|"knew the fix in 2 min"| F[Scale payments-worker to 0]

The real cost of a missing runbook isn't the outage — it's the exhaustion

Payments queued safely the whole time; nothing was actually lost. The 40 minutes didn't cost the business much. But it cost me: I was awake, stressed, and context-switching through unfamiliar code for 40 minutes at 3am for a problem that had a known, 90-second fix — known by someone else, on another team, who wasn't the one who got paged. That mismatch — the person who gets paged isn't always the person who has the answer — is exactly what a runbook is supposed to bridge.

What a runbook actually needs to contain (and what it doesn't)

I used to think runbooks were bloated wiki pages nobody reads. They are, when they try to explain the whole system. A useful runbook answers exactly one question fast: given this specific alert, what do I do in the next 2 minutes? Everything else is noise.

1. **Is this safe to ignore/wait on, or does it need action now?** — a one-line severity/impact statement, written for someone with zero context.
2. **The single safest first action** — not a menu of options. One default move that's almost always correct (in our case: scale the worker down, don't try to fix the root cause live).
3. **Who actually owns the root cause**, by name or team/Slack channel — not "the payments team" in the abstract, but the specific on-call rotation to page.
4. **What NOT to do** — the thing that seems reasonable but makes it worse (in our case: don't just restart the pod repeatedly — it re-triggers the same crash against the same broken dependency).
5. **How to confirm it's actually resolved** — a specific metric or log line to check, not "it looks fine now."

## PodCrashLoopBackOff: payments-worker

**Impact:** Payments queue safely during this — no data loss, but
processing is delayed. Not an emergency, but resolve within 30 min
to avoid queue backlog alerts.

**First action:** Scale payments-worker to 0 replicas immediately.
  kubectl scale deploy/payments-worker -n prod --replicas=0
This stops it from hammering whatever it's failing against.

**Root cause owner:** fraud-check team (#fraud-check-oncall).
Page them — this crash loop is almost always caused by their
service returning malformed responses under load.

**Do NOT:** repeatedly restart/delete the pod. It will crash-loop
again immediately against the same broken dependency and wastes
time.

**Resolution check:** scale payments-worker back to its normal
replica count and confirm error rate on `payments_worker_errors_total`
drops to baseline within 5 minutes.

The actual fix wasn't just documentation — it was automation

Writing the runbook stopped the 40-minute version of this incident. But the deeper fix was recognizing that "scale the worker down when this specific crash loop appears" is a deterministic, safe, well-understood action — which means it doesn't need a human awake at 3am to execute it at all. That's the exact class of problem I've since started encoding as automated, guarded remediation instead of a runbook a human has to read and follow manually: match the known failure signature, run the known-safe first action automatically, and only escalate to a human with full context if the automated fix doesn't resolve it. A good runbook is the right first step. An automated one is the actual goal.