Producing & Consuming Events Idempotently

Stop Chasing Exactly-Once

Exactly-once is one of those phrases that sounds like architecture maturity until you ask what it means at the boundary that matters. Kafka can offer strong guarantees inside its own log mechanics, but the moment a consumer updates Postgres, calls an email provider, or writes to Redis, the problem is bigger than broker semantics. Fieldwork therefore plans around at-least-once delivery and makes handlers idempotent. That is not settling. It is choosing a guarantee we can actually keep end-to-end.

The rejected design was psychological more than technical: we'll make duplicates impossible. In practice, duplicates appear because producers retry, consumers crash after side effects but before offset commit, or operators replay topics during recovery. Pretending those are exceptional makes the system brittle. Designing for idempotency makes them ordinary.

Producer Identity and Ordering

Idempotent consumers start with stable producer identity. Every planned Fieldwork event includes a globally unique event_id, the aggregate identity it belongs to, and a monotonic version for that aggregate. That version is not there for decoration. Consumers that maintain a projection of task state need to know whether they are applying event 9 after event 8 or receiving an old replay after event 10 already won. We rejected payloads that only contained task updated with no version or event ID because they force consumers to guess whether they have seen the fact before.

• Give every event a stable unique ID.
• Include aggregate version when consumers need ordering or projection safety.
• Use partition keys that keep one aggregate's event stream together when ordering matters.

Consumer-Side Deduplication

The simplest useful consumer pattern is a processed-events table in the consumer's own database. On message receipt, the consumer opens a transaction, attempts to insert event_id, applies its state change if the insert succeeds, and commits. If the insert hits a uniqueness violation, the event was already processed and the handler exits cleanly. That is the concrete design Fieldwork plans for audit projections, activity feeds, and notification bookkeeping. We rejected in-memory dedup caches because process restarts erase them, and deduplication is valuable precisely when processes restart.

Retries, Poison Messages, and Side Effects

Idempotency does not eliminate bad messages. Consumers still need a policy for malformed payloads, schema mismatches, and downstream side effects that fail persistently. Fieldwork's planned model is to separate this event has already been handled from this event cannot currently be handled. Temporary failures get retried with backoff. Truly poison events go to a dead-letter path with enough metadata to inspect and replay after a fix. External side effects like email sending also need their own idempotency keys when possible. We rejected the lazy answer of just don't commit the offset and let Kafka redeliver forever because that turns one poison message into a stuck partition.

The Decision

Fieldwork's planned eventing model embraces at-least-once delivery and makes consumers responsible for durable deduplication. Producers emit stable event IDs, aggregate versions, and meaningful keys; consumers write processed-event markers in the same transaction as their side effects. We rejected the fantasy of exactly-once end-to-end because it breaks the moment side effects leave the broker. The real decision is more practical and more reliable: duplicates will exist, so the system should know how to absorb them.