Interactive postmortem
Connection pool exhaustion: a 2am death spiral
One slow downstream query turned into a full outage in six minutes — not because of the slowness itself, but because of what a fixed-size connection pool does when every connection is stuck waiting and nothing has a timeout. Scroll to walk through it phase by phase, same incident as the full write-up on Field Notes, told here as a timeline instead of an article.
Incident timeline
T+0s
Slow reporting query starts arriving at normal traffic volume.
T+15s
Each slow request holds a connection 8–12s instead of <50ms — no query timeout to stop it.
T+45s
All 20 pool connections are checked out and stuck on the slow query.
T+46s
An unrelated fast endpoint tries to check out a connection. Pool is empty. It blocks.
T+50s
Every request, on every endpoint, now queues behind the exhausted pool.
T+2min
Upstream callers see timeouts and retry — adding more load to an already-saturated pool.
T+4min
On-call kills the stuck reporting queries directly at the database and pages the query owner.
T+6min
Pool drains, latency recovers. Health checks had stayed green the entire time.
01 · Detection
Health checks were green. Almost nothing else was.
The service's process never crashed. CPU and memory stayed normal, and its liveness probe kept returning 200 the entire time — because the process really was alive. What alerted the team wasn't a crash signal, it was p99 latency across every endpoint jumping from single-digit milliseconds to the full length of client timeouts, all at once, across unrelated routes that shared nothing but a database connection pool.
That "every endpoint at once" shape is itself a diagnostic clue — a single broken endpoint slowing down doesn't explain a totally unrelated health-check-adjacent route timing out too. Something shared was the bottleneck, not any one handler's logic.
02 · Triage
Ruling out the database, then finding the real chokepoint
First check was always going to be "is the database itself down?" It wasn't — query latency on the database side was normal for the vast majority of queries. That ruled out an infra-level outage and pointed at something in front of the database: the connection pool.
SHOW pg_stat_activity-equivalent tooling confirmed it: all 20 connections in the pool were checked out, and the queries behind most of them were a single rarely-hit reporting query that should take under 50ms, now running 8–12 seconds each.
03 · Mitigation
Stop the bleeding first, understand it fully second
The fastest lever available during the incident wasn't a code deploy — it was killing the stuck queries directly at the database, which immediately freed every connection they were holding. That single action recovered the pool and brought latency back to normal within about two minutes of being applied.
Restarting the service would not have fixed this — a fresh process would have opened a fresh pool and hit the exact same slow query within seconds. Killing the actual offending queries, not the process around them, was the correct mitigation.
04 · Root cause
One slow query wasn't the root cause — three missing bounds were
The reporting query going slow was the trigger (a migration the night before had silently dropped an index it depended on). But a single slow query is normally a contained, minor problem. It became a total outage because nothing in the request path had a bound on it:
- No query timeout — the slow query ran with
context.Background(), so it could hold its connection indefinitely. - No connection check-out timeout — requests waiting for a connection queued forever instead of failing fast.
- No retry backoff — upstream callers retried immediately on timeout, adding load to an already-drowning pool instead of backing off.
The pool size of 20 was never the problem — it was a perfectly reasonable number for the service's real traffic. Doubling it would have only delayed the same collapse to a slightly higher load.
05 · Follow-ups
What shipped afterward, and what didn't
Three changes went out in the days following, each closing one of the three missing bounds:
- 01 A hard 500ms timeout on every database query, no exceptions — enforced via a lint rule that flags any `context.Background()` near a DB call.
- 02A 200ms check-out timeout on the pool itself, so requests fail fast with a clear "pool saturated" error under real saturation instead of queueing.
- 03 Exponential backoff with jitter added to the retry logic in the two internal clients that called this service, so a timeout no longer triggers an immediate retry storm.
What didn't change: the pool size. It was correct before the incident and it's correct after — the fix was bounding behavior, not adding capacity.