The Outbox Pattern

Why Dual Writes Fail

The outbox pattern exists because the straightforward code is wrong. Imagine the task service updates a task row in Postgres, commits, then publishes task.status_changed to Kafka. What happens if the database commit succeeds and the Kafka publish fails? Now the source of truth changed and no consumer heard about it. Reverse the order and the opposite failure appears: Kafka publishes an event for a task update that never committed. That is the classic dual-write problem. It shows up the moment one state change has to be durable in more than one system.

We rejected two bad answers. The first was publish inline and retry really hard. Retries reduce probability, not category. The second was distributed transactions. They solve a narrower problem with a much wider operational cost and do not fit the systems we plan to run here. The outbox pattern is the deliberately boring middle path: make one durable write transaction, then relay the resulting event record asynchronously.

Writing the Outbox Row

In Fieldwork's planned design, any write that should emit an integration event inserts an outbox row before the transaction commits. The row records tenant ID, aggregate type, aggregate ID, event type, payload, status, and created time. It is not a generic audit log. It exists specifically so a relay worker can later publish to Kafka without rereading business tables and trying to reconstruct intent. We rejected trigger-based serialization of table changes because it makes event design implicit and tied to low-level row deltas instead of the domain facts we actually want to expose.

• Store the event payload the relay will actually publish; do not make the relay reverse-engineer domain state.
• Keep outbox rows tenant-aware for debugging and throttling.
• Treat outbox rows as integration intent, not as a replacement for all internal audit history.

Relaying from Postgres to Kafka

A separate relay worker scans pending outbox rows, claims a batch with FOR UPDATE SKIP LOCKED, publishes them to Kafka, and marks them sent. That worker is allowed to retry because consumers are already designed for at-least-once delivery. We rejected deleting rows immediately after publish with no sent marker because operational visibility matters. A sent timestamp tells us what is stuck, what is slow, and what is safe to archive later. The relay is a background worker like any other job processor, which is one reason the earlier worker-pool and retry lessons mattered before this one.

Retries, Cleanup, and Operational Trade-Offs

The outbox pattern moves complexity rather than erasing it, which is fine as long as we move it somewhere more controllable. Relay failures need retry policy. Old sent rows need archival or deletion. Monitoring needs to alert when pending outbox age grows beyond expectation. We rejected the argument that this is too much machinery because the alternative machinery is hidden in inconsistent state reconciliation after failed dual writes. At least the outbox complexity is explicit, queryable, and testable.

The Decision

Fieldwork plans to publish integration events with the outbox pattern: write the business change and an outbox row in one Postgres transaction, then relay pending rows to Kafka from a separate worker. We rejected direct dual writes because their failure modes silently split the truth between systems. The outbox does add machinery, but it puts that machinery in a place we can inspect and reason about. That is the decisive advantage: the system can prove what it intended to publish even when the broker is unavailable at exactly the wrong moment.