Backend

Connection Pool Exhaustion: A Case Study in a 2am Death Spiral

One slow downstream dependency turned into a full outage in six minutes — not because of the slowness itself, but because of what a fixed-size connection pool does when every connection is stuck waiting.

January 27, 20268 min read

connection poolingdatabasescascading failuretimeouts

The API service itself never crashed. Its process stayed healthy, its CPU and memory were normal, its health check kept returning 200. And yet for six minutes it served almost nothing but timeouts, taking every downstream service that called it down with it. The cause was a connection pool doing exactly what it was configured to do — which was the actual problem.

The setup: a completely ordinary connection pool

The service held a database connection pool of 20 connections — a reasonable, unremarkable number for its traffic. Each request checked out a connection, ran a query, returned it. Nothing unusual. The database itself was healthy for the whole incident — this wasn't a database outage.

Gothe-pool-configuration-(this-is-the-actual-root-cause,-in-one-line).go

4 linesLn 1, Col 1Go

Every query ran with `context.Background()` — no deadline. A connection checked out for a query had no upper bound on how long it could be held.

The trigger: one slow query, not a crash

A single, rarely-hit reporting query — normally under 50ms — started taking 8 to 12 seconds because of an unrelated migration that had silently dropped an index the night before (a separate mistake, but not the one that caused the outage). 8-12 seconds is slow, but it's not a crash. On its own, this should have been a minor, contained latency blip on one endpoint.

The spiral: how a slow query becomes a total outage in six minutes

T+0s: The slow reporting query starts arriving at normal traffic volume — a few requests per second hit that endpoint.
T+15s: Each of those requests holds a connection from the pool for 8-12 seconds instead of the usual <50ms. With no query timeout, nothing kills them early.
T+45s: Enough concurrent slow requests have accumulated that all 20 connections in the pool are now checked out and busy, every one of them stuck in the slow query.
T+46s: A completely unrelated, normally-fast endpoint (say, GET /health-details or a simple user lookup) tries to check out a connection. The pool is empty. It blocks, waiting for one to free up.
T+50s onward: Every incoming request across the entire service — regardless of which endpoint, regardless of whether it needs the slow query at all — now queues waiting for a connection. Request latency for the *entire service* jumps from milliseconds to the full length of the client's own request timeout.
T+2min: Upstream callers, seeing this service time out, start retrying — which sends *more* concurrent requests at an already-saturated pool, making the queue longer, not shorter.
T+6min: The service is functionally down. Health checks still pass (the process is alive, not crashed) but nearly 100% of real requests are timing out.

How one slow query saturates a shared, unbounded pool

Rendering diagram…

The pool size was never the problem

20 connections was a perfectly reasonable number for the service's actual query volume and normal latency. The failure mode isn't "the pool is too small" — it's that a *fixed-size* pool with *no per-query timeout* has no way to shed a slow, misbehaving query. One bad query can monopolize the entire pool indefinitely, and every unrelated request pays for it.

The two changes that actually fixed this

We didn't increase the pool size — that just delays the same collapse to a slightly higher traffic level. The real fixes were about bounding how long any single query can hold a connection, and giving requests a way to fail fast instead of queueing forever.

Gofix-1:-every-query-gets-a-hard-timeout,-no-exceptions.go

4 linesLn 1, Col 1Go

Now the slow reporting query fails after 500ms instead of holding a connection for 8-12 seconds. The connection returns to the pool almost immediately, even when the query itself is broken.

Gofix-2:-bound-how-long-a-request-waits-for-a-connection-at-all.go

11 linesLn 1, Col 1Go

Under saturation, requests now fail in 200ms with a clear "pool saturated" error instead of queueing for 30+ seconds. Failing fast means upstream retries hit a fast failure instead of piling onto an already-drowning pool — it breaks the retry amplification loop.

The real lesson: unbounded anything is a single point of failure

The specific bug (a dropped index causing one slow query) was almost incidental. The reason it became a total outage instead of a minor blip was that *nothing in the system had a bound*: no query timeout, no connection check-out timeout, and client retries with no backoff. Any one of those three bounds, on its own, would have contained the incident to the single slow endpoint. Missing all three is what turned it into a cascading, service-wide failure.

A concrete audit for your own services

Grep your codebase for context.Background() or context.TODO() anywhere near a database call, an HTTP client call, or anything that acquires a shared, finite resource (connection pool, semaphore, worker queue slot). Each one is a potential connection-pool-exhaustion incident waiting for a slow dependency to trigger it.