Circuit Breakers

Why Retries Weren't Enough

Timeouts and bounded retries solved one class of pain in Fieldwork: requests no longer hung forever. They did not solve the next class: what happens when a dependency is slow for minutes, not milliseconds. Our membership service became the first example. The task API checks whether the actor is a workspace admin or contributor before it mutates a project. When the membership service started failing under load, the task API kept doing exactly what we told it to do: open connections, wait, retry a little, then fail. Multiplied by enough concurrent requests, that behavior turned one unhealthy dependency into broader saturation.

The alternative we rejected was "let autoscaling absorb it." Autoscaling can help capacity shortages, but it does nothing when the dependency is failing because of a bad query plan, a locked table, or an exhausted connection pool. In those cases more callers are not demand; they are pressure. The circuit breaker decision was less about clever distributed systems theory and more about operational honesty: if a dependency is already shouting that it cannot serve traffic, stop forwarding even more traffic to it.

What We Wrapped With a Breaker

We did not wrap the whole process in one giant breaker. That makes metrics ambiguous and failure scopes too broad. Instead, Fieldwork puts breakers around clearly remote, clearly optional-to-defer interactions: outbound calls from the task service to memberships, from the gateway to internal HTTP services, and from async workers to notification providers. We explicitly did not use a breaker around ordinary in-process validation or the primary Postgres write path because breaking those would only hide local problems behind another layer of state.

The membership check is a good example because it is required before many task mutations. If that client starts timing out at a high rate, we would rather return a 503 immediately with a dependency-unavailable error than spend three seconds learning the same answer one hundred times. That early refusal also becomes a cleaner operational signal. Instead of a blurred mix of deadline exceeded, cancelled contexts, and random upstream 502s, we can see the breaker trip and know exactly which dependency crossed the threshold.

Half-Open Is a Probe, Not a Ramp

The subtle part of circuit breakers is not opening. Anybody can flip from closed to open after enough failures. The subtle part is half-open. If you reopen the floodgates the moment the timeout window passes, you have built a timed pause, not a breaker. Fieldwork uses a small number of probe requests when half-open because the point is to test recovery, not to declare victory on a timer. That matters most for shared dependencies like membership reads where hundreds of callers may be queued behind the same service health change.

We also rejected the idea that the breaker alone should decide fallback behavior. Some endpoints can degrade. For example, a workspace activity feed can return task rows without enriched assignee display names if a profile service is unavailable. But permission checks cannot degrade that way. If the membership authorizer is down, guessing would be a security bug. So the breaker only answers whether to call; the handler or service still decides whether to fail closed, serve stale data, or skip an optional enrichment.

Where We Chose Not to Use Breakers

Knowing where not to use a pattern mattered as much as using it. We chose not to put breakers around the main Postgres repository methods in the request path. The database is not an optional downstream for the task service; it is the service. If Postgres is struggling, a breaker can reduce pressure, but it can also mask deeper pool sizing or query design problems while adding one more state machine to explain. For that layer we preferred deadlines, pool limits, and careful query work first.

We also avoided one shared global breaker for "all outbound HTTP." Fieldwork has different failure consequences for profile lookups, permission checks, webhook deliveries, and analytics fan-out. Sharing a breaker would let one sick integration penalize an unrelated one. So the final decision was specific: breaker per remote dependency, half-open with probe limits, fail closed for security-sensitive paths, and no pretending that a breaker is a substitute for fixing the real bottleneck underneath.

• Use breakers on clearly remote dependencies that can drag the caller down with them.
• Trip on enough real evidence to avoid flapping, but keep the threshold low enough to stop useless traffic early.
• Treat half-open as a controlled probe window, not an automatic all-clear.
• Decide fallback behavior at the business boundary, because not every dependency is safe to degrade around.
• Do not use breakers to paper over local database, query, or pool design problems.