Job Queues, Retries & Scheduling

Why Postgres Won First

A dedicated queueing system is eventually a reasonable choice. It was not the first one for Fieldwork. The background work in this system is tightly coupled to Postgres rows we already own: tasks changing status, exports being requested, sessions expiring, outbox events waiting to publish. Starting with Kafka for asynchronous jobs would have split the source of truth too early. Starting with Redis lists would have given us fast enqueue and weak introspection. We chose a Postgres-backed queue because the project already depends on Postgres, jobs often originate from the same transaction as the domain write, and operational simplicity mattered more than theoretical throughput at this stage.

The rejected argument was queues should always be in queueing infrastructure. That's true when scale or fan-out demands it. But for the first generation of Fieldwork jobs, the concrete cost of introducing another system was higher than the benefit. We would have added another persistence model, another operational surface, and more dual-write edges before we had enough workload diversity to justify them. Postgres was already where the authoritative state changed, so it became the first durable queue too.

Claiming Jobs with Leases

A queue table without leasing is just a race condition with SQL around it. Multiple workers must be able to look for ready work without claiming the same row. The design Fieldwork chose is a jobs table with status, run_at, attempt, max_attempts, locked_at, locked_by, and last_error. Workers claim jobs in a transaction using FOR UPDATE SKIP LOCKED, flip them to running, stamp a lease owner, and commit. That gives us horizontal workers without a distributed lock service. We rejected select then update because it loses races under concurrency and creates duplicate execution exactly when the queue is busiest.

• Keep queue rows tenant-aware so noisy tenants can be identified and, if necessary, isolated later.
• Separate queued, running, completed, and failed states instead of inferring everything from timestamps.
• Record locked_by and locked_at so stuck jobs are explainable and recoverable.

Retries That Back Off Instead of Thrash

The easiest retry policy is if err != nil { queue again now }, which is another way of saying turn a temporary outage into a denial-of-service against yourself. Fieldwork uses exponential backoff with jitter and a hard attempt ceiling. Network-flavored jobs — webhook delivery, object-storage callbacks, Kafka publish relay later on — often fail in waves. Immediate retries stack those waves into the same outage window. Backoff deliberately moves pressure away from the failing dependency. We rejected manual per-job retry sleeps inside worker code because that ties up worker slots doing nothing. Retry timing belongs in the queue row's next run_at, not in a sleeping goroutine.

Recurring Work without Cron Fragility

Recurring work such as session cleanup, attachment quarantine scans, or weekly workspace summary rollups should use the same queue, not a parallel universe of unmanaged cron scripts. Fieldwork's scheduler writes future queue rows with a recurrence key and next-run timestamp. That preserves one operational model: all work is inspectable in the jobs table, all retries use the same semantics, and all execution goes through the same worker fleet. We rejected host-level cron because it fragments observability and makes deployments surprising. A container restart should not erase the platform's idea of when work is due next.

The Decision

Fieldwork starts background execution with a Postgres-backed job queue. Jobs are claimed with leases using FOR UPDATE SKIP LOCKED, retried by moving their run_at into the future with exponential backoff, and scheduled recurring work by inserting future-dated rows into the same table. We rejected separate cron paths, immediate retry loops, and premature queue infrastructure because the concrete cost was more operational surface without a matching need. The design is intentionally plain: durable rows, explicit state transitions, inspectable SQL, and failure behavior that slows down instead of thrashing harder.