Stage 7 · Master
Background Jobs & Async Work
Graceful Worker Shutdown
A deploy should move work forward, not kill it halfway through and hope retries hide the damage.
Why Hard Stops Are Expensive
A worker receiving SIGTERM during a deploy is not an edge case. It is normal platform behavior. If the process exits immediately, any in-flight job is left in an ambiguous state: maybe the webhook was sent but the completion update was not written, maybe half the export file was streamed, maybe the outbox event was published and the offset not committed. The queue will retry it is not a real answer unless the job is designed to tolerate duplicate execution and the worker releases or expires its lease correctly. Fieldwork treats graceful shutdown as part of queue correctness, not a polish step after the queue works.
The rejected approach was to let Kubernetes kill the worker and rely on lease timeouts later. That eventually recovers, but it turns every deployment into avoidable latency spikes and duplicate-work opportunities. A clean shutdown should stop claiming new jobs immediately, let active workers finish within a deadline, and make unfinished work visible for retry if the deadline is exceeded. Anything less is hoping the queue semantics compensate for process behavior they did not need to see.
If a job system promises at-least-once execution, the worker still has a responsibility to avoid creating unnecessary duplicates during normal operations like deploys.
The Shutdown Protocol
Fieldwork's shutdown sequence is intentionally boring. First, the process marks itself draining so the fetch loop stops claiming new jobs. Second, it cancels intake context so any polling sleep wakes immediately. Third, it waits for current workers to finish up to a configured timeout. Fourth, if the timeout expires, it releases remaining leases back to the queue or lets short leases expire so another worker can recover them. We rejected close everything instantly because it loses information about what was actually running. We also rejected wait forever because deploys need an upper bound. Draining is useful only if there is still a deadline.
- Stop claiming new jobs before doing anything else.
- Give in-flight work a deadline to finish; do not wait forever.
- Make lease recovery explicit when the deadline expires.
Leases, Deadlines, and In-Flight Work
Graceful shutdown only works if job execution itself is deadline-aware. A worker that ignores context.Context cannot be drained predictably. Fieldwork jobs are written to accept a context derived from the worker process, and long-running jobs periodically checkpoint or validate that their lease is still valid. That matters for uploads, exports, and any job that might outlive the deployment's termination grace period. We rejected giant monolithic jobs that do thirty minutes of work in one transaction precisely because they make graceful shutdown impossible. If a task is that large, it should be chunked into smaller jobs or heartbeat its lease.
| Shutdown behavior | Result | Assessment |
|---|---|---|
| Immediate exit on SIGTERM | Fast deploys, ambiguous jobs | Rejected |
| Drain with no deadline | Fewer duplicates, unbounded rollout time | Rejected |
| Drain with timeout and lease recovery | Predictable rollout and recoverable unfinished work | Chosen |
Deployment Contracts Matter Too
Application code cannot solve this alone. The worker deployment needs a sensible termination grace period, readiness should flip quickly when draining begins, and lease timeouts in the queue must be longer than normal job latency but shorter than a truly dead worker. That coordination is why graceful shutdown belongs in the same design conversation as job execution. We rejected the habit of tuning Kubernetes and worker code independently. If the platform kills the container after 10 seconds but the worker expects 30, the nicer shutdown code is mostly fiction.
A worker that only proves graceful shutdown during real deploys is untested. Send SIGTERM in local and staging runs while long jobs are active and verify lease recovery behaves the way the design claims.
The Decision
Fieldwork workers shut down by entering a draining state, stopping job intake, waiting briefly for in-flight work to finish, and then explicitly releasing or timing out unfinished leases. We rejected hard process exits and indefinite draining because both turn ordinary deploys into reliability hazards. The decision is not glamorous, but it is concrete: a worker is part of the delivery contract of a job, so shutdown behavior has to be engineered with the same care as claiming, retries, and idempotency.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.