Timeouts, Context Deadlines & Retries

The Deadline Starts at the Edge

Fieldwork's first version had the usual beginner mistake: every downstream client owned its own timeout, and each timeout looked reasonable in isolation. The API gateway gave handlers 10 seconds, the membership service used a 5 second HTTP client, and the task repository used a 3 second query context. On paper that sounds defensive. In practice it meant one request could spend 5 seconds waiting on memberships, then another 3 on Postgres, then a little more on audit logging, and only fail after the user's patience was already gone. We had timeouts everywhere and no real deadline anywhere.

We rejected the common alternative of "just tune each dependency separately" because it optimizes each hop instead of the request. Fieldwork is a tenant-scoped workspace API. A single POST that creates a task can validate membership, load the project, insert the task row, and enqueue an outbox event for notifications. If every hop resets its own clock, the system behaves like it has infinite time as long as each layer is locally polite. That is exactly how overloaded services pile up work they can no longer finish.

Context Is the Budget Carrier

Once we decided the deadline starts at the edge, the next decision was whether to pass raw timeout values through our function signatures or let context carry the budget. We rejected explicit timeout integers everywhere because they create two channels of truth: a context that can be cancelled, and a duration argument that may or may not match it. That makes reviews worse. You end up asking whether 750 milliseconds in the repository is a product decision, a defensive default, or a stale number copied from last month.

In Fieldwork, the task service needs membership authorization before it mutates anything. That means a request like create task must call the membership client and the project repository under the same budget. The service does not get to decide that auth is allowed 2 seconds and storage is allowed 2 more. It only gets to decide call order and whether an optional step is worth attempting with the time left. That is why context ended up as part of every repository and downstream client method in the course project.

Retries Need a Safety Contract

After deadlines were fixed, the tempting next move was to add retries everywhere. We did not. A retry is not a reliability feature by itself; it is a multiplication factor on whatever behavior already exists. If the original call is safe and likely failed due to a transient network problem, a retry is useful. If the original call already mutated state and only the response was lost, a retry can duplicate work. In a task-tracking system, that means duplicate tasks, duplicate membership invitations, or duplicated notification events.

So Fieldwork adopted a narrower rule: only idempotent reads are retried automatically, and writes are retried only when we have an explicit idempotency mechanism. For HTTP that means GET and safe DELETE-style operations with stable resource identity can retry. For task creation, we require an idempotency key tied to tenant_id, workspace_id, actor, and request body hash before the gateway is allowed to replay it. For the outbox publisher, the database primary key on the event row makes retries safe because repeated publish attempts do not create new application records.

What We Actually Standardized

The final shape was intentionally boring. The gateway creates the parent deadline. Every service, repository, and client method accepts context.Context first and treats cancellation as a normal control signal, not an exceptional crash. Retries are opt-in at the call site, never implicit in a generic transport wrapper. And every retry policy is shorter than the request budget it consumes. We preferred that explicitness over more automation because hidden resilience logic is how teams convince themselves the system is safer while it is actually harder to reason about.

• Set one request budget at the edge, then propagate it through Gin handlers, pgx queries, HTTP clients, and Kafka producers.
• Use context to carry cancellation and deadlines; do not add parallel timeout arguments except for clearly local background jobs.
• Retry only transient failures, and only for operations with a defined idempotency story.
• Never let retries outlive the parent deadline; a late success after the caller gave up is not a success.
• Treat deadline exceeded as backpressure working, not as a signal to blindly raise every timeout value.