Producing and Consuming Events Idempotently

Define the Delivery Guarantee First

Event systems are easier to reason about once the delivery guarantee is stated honestly. At-most-once means a message may be lost but will not be retried. At-least-once means a message will be retried when necessary, so duplicates are possible. Exactly-once is often misunderstood: even if a broker offers strong guarantees within its own log, the moment a consumer updates a database, sends an email, or calls another API, the end-to-end system still has to deal with retries and replay. For that reason, production backends usually choose at-least-once delivery and make consumers idempotent.

Producer Identity Makes Idempotency Possible

A consumer can only deduplicate an event that has stable identity. That means the producer must emit an event_id, identify which aggregate changed, include the tenant boundary explicitly, and attach an ordering signal such as an aggregate version when consumers care about stale-vs-newer updates. Without those fields, the consumer is left guessing whether two messages describe the same fact or merely look similar.

Consumer Deduplication Must Be Durable

In-memory deduplication is not real deduplication because it disappears on restart—the exact moment replay becomes most likely. A correct consumer records handled event IDs in durable storage that lives next to the projection or side effect it is protecting. In Meridian's planned architecture, the reporting-service should maintain a processed_events table keyed by consumer name and event ID so a partition replay after a crash becomes a no-op instead of a duplicate analytics row.

CREATE TABLE processed_events (
    consumer_name text NOT NULL,
    tenant_id uuid NOT NULL,
    event_id uuid NOT NULL,
    aggregate_type text NOT NULL,
    aggregate_id uuid NOT NULL,
    aggregate_version bigint NOT NULL,
    processed_at timestamptz NOT NULL DEFAULT NOW(),
    PRIMARY KEY (consumer_name, event_id)
);

CREATE INDEX idx_processed_events_tenant_processed_at
    ON processed_events (tenant_id, processed_at DESC);

The Consumer Transaction Boundary

Idempotency depends on ordering the consumer's own steps correctly. The safe sequence is: begin a database transaction, try to insert the processed-event marker, apply the projection or side effect only if that insert succeeds, commit the database transaction, and only then commit the broker offset. If the process crashes after the database commit but before the offset commit, Kafka may redeliver the message, but the consumer's database already knows the event was handled.

Ordering, Retries, and Poison Messages

Idempotency does not remove operational failure modes; it makes them survivable. Temporary failures should be retried with backoff. Malformed events or schema mismatches should be dead-lettered instead of blocking a partition forever. Ordering-sensitive consumers must compare aggregate versions so that an older replay cannot overwrite a newer projection. If a consumer later triggers an external side effect—such as exporting a report—the same event ID should become the idempotency key for that external system whenever possible.

• Duplicate event ID: treat as success and advance after the no-op path completes.
• Older aggregate version than the projection already stores: ignore the stale event without corrupting the read model.
• Temporary downstream outage: retry with bounded backoff instead of committing the offset immediately.
• Malformed payload: move the message to a dead-letter path and alert, because infinite redelivery cannot repair bad data.

The Decision

Meridian's future event consumers should assume at-least-once delivery and be engineered so replay is harmless. That requires stable producer identity, durable processed_events state inside the consumer's own database, transactional dedup-plus-projection writes, and version guards for ordering-sensitive read models. None of this is implemented in meridian yet; the lesson defines the contract that the planned reporting-service and any later consumers must satisfy if they are to remain correct under retries and recovery.