The Outbox Pattern

Why Dual Writes Fail

The moment a service must persist one business change in two different systems, it faces the dual-write problem. Suppose billing-service marks an invoice as paid in Postgres and then tries to publish invoice.paid to Kafka. If the database commit succeeds but the broker publish fails, the source of truth changed and downstream consumers never hear about it. If the publish happens first and the database commit fails afterward, consumers hear about a payment that never actually committed. Either order produces a correctness gap.

Retries reduce the probability of the gap but do not remove the category of failure. Distributed transactions are usually too operationally heavy for this style of system and still do not simplify the mental model for beginners. The outbox pattern exists because it converts one impossible 'commit to two systems at once' problem into two honest steps: commit the domain change and event intent together in one database transaction, then relay that durable intent to the broker asynchronously.

The Outbox Table Records Intent

An outbox table is not an audit log and not a trigger-generated copy of low-level table changes. It is a deliberate integration queue stored in the same database as the transactional data. Each row should identify the tenant boundary, the aggregate that changed, the event type, the serialized payload, the publication status, attempt counters, and timestamps that let operators answer what is pending or stuck.

CREATE TABLE outbox_events (
    id uuid PRIMARY KEY,
    tenant_id uuid NOT NULL,
    aggregate_type text NOT NULL,
    aggregate_id uuid NOT NULL,
    event_type text NOT NULL,
    payload jsonb NOT NULL,
    status text NOT NULL CHECK (status IN ('pending', 'publishing', 'sent', 'failed')),
    attempt_count integer NOT NULL DEFAULT 0,
    locked_at timestamptz,
    locked_by text,
    last_error text,
    created_at timestamptz NOT NULL DEFAULT NOW(),
    updated_at timestamptz NOT NULL DEFAULT NOW(),
    published_at timestamptz
);

CREATE INDEX idx_outbox_pending_created_at
    ON outbox_events (status, created_at, id)
    WHERE status IN ('pending', 'publishing');

Writing Domain State and Outbox Together

The critical rule is simple: if a business write should emit an integration event, the service writes the domain mutation and the outbox row inside one database transaction. This lesson uses billing-service as the concrete example because invoice.paid is easy to reason about, but the same pattern belongs in identity-service and community-service for their own events. The source service decides that a business fact occurred; the outbox makes that decision durable before the transaction commits.

Relaying from Postgres to Kafka

A relay process runs separately from the request path. Its job is to fetch pending outbox rows, claim them safely so multiple workers do not publish the same row simultaneously, publish them to Kafka, and then mark them sent. FOR UPDATE SKIP LOCKED is a common and practical claiming strategy because it allows multiple relay workers to process the outbox concurrently without blocking one another on already-claimed rows.

Retries, Recovery, and Operations

Outbox shifts complexity into a place that is explicit and inspectable. That is a feature, not a drawback. Operators can now ask: Which tenant's events are backing up? How old is the oldest pending row? Which relay worker crashed while holding rows in publishing? Those are answerable SQL questions. The alternative—silent event loss during a direct dual write—is operationally simpler only until the first inconsistency investigation begins.

• Monitor the age of the oldest pending outbox row; growing age indicates broker or relay trouble even when request paths are still succeeding.
• Reset stale publishing rows to pending after a timeout so a crashed relay cannot strand them forever.
• Archive or prune old sent rows on a retention policy once they are no longer needed for audit or replay diagnostics.
• Keep tenant_id on every outbox row so backpressure, throttling, and incident analysis can remain tenant-aware.

The Decision

Meridian's future event publication path should use the outbox pattern in each transactional service. identity-service, community-service, and billing-service will record domain changes and event intent in one database transaction, then relay pending rows to Kafka asynchronously. That design does not yet exist in meridian, but it is the correct specification for avoiding silent event loss once Chapter 11 moves from scaffolding to implementation.