Idempotent Consumers

Why Idempotency Matters

In distributed systems, messages can be delivered more than once. Network retries, consumer crashes, and at-least-once delivery guarantees all cause duplicate processing. An idempotent consumer produces the same result whether a message is processed once or ten times.

Deduplication Keys

Every message must carry a unique identifier. The consumer tracks which identifiers have been processed and skips duplicates. The deduplication key should be the business-level unique identifier, not the message ID.

import redis
import hashlib

r = redis.Redis()

def process_payment(event):
    # Use business-level dedup key
    dedup_key = f"payment:{event['idempotency_key']}"
    
    # Atomic check-and-set (SET NX = not exists)
    if not r.set(dedup_key, "processing", nx=True, ex=3600):
        log.info(f"Duplicate event: {event['idempotency_key']}")
        return  # Already processed or in progress
    
    try:
        result = charge_payment(event)
        r.set(dedup_key, f"completed:{result['charge_id']}", ex=86400)
        return result
    except Exception as e:
        r.delete(dedup_key)  # Allow retry
        raise

Transactional Outbox Pattern

The transactional outbox pattern ensures reliable event publishing. Instead of publishing events directly to Kafka, write them to an outbox table in the same database transaction as the business operation. A separate process polls the outbox and publishes to Kafka.

BEGIN;

-- Business operation
INSERT INTO orders (id, user_id, amount) 
VALUES ('order_123', 'user_456', 99.99);

-- Outbox event (same transaction)
INSERT INTO outbox (id, topic, payload, created_at)
VALUES ('event_789', 'order-events', '{"order_id":"order_123","action":"created"}', NOW());

COMMIT;

-- Separate process polls outbox and publishes to Kafka:
-- SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100
-- Publish each row to Kafka, then UPDATE outbox SET published = true

Exactly-Once Boundaries

Exactly-once semantics exist at specific boundaries: Kafka producer to Kafka (idempotent producer), Kafka consumer to database (transactional processing). Across boundaries (Kafka to external API), you must use idempotent operations.

Exactly-once available:
  Producer ──► Kafka     (idempotent producer, enable.idempotence=true)
  Kafka ──► Consumer DB  (consume in transaction with DB write)

Not exactly-once:
  Kafka ──► External API  (cannot guarantee idempotency of external API)
  Kafka ──► Multiple DBs  (two-phase commit required)

Solution: Make the external call idempotent using dedup keys.

Idempotent Operation Design

• Natural idempotency — PUT operations are naturally idempotent (set X to Y).
• Idempotency keys — Client provides a unique key for each operation.
• Database constraints — UNIQUE constraints prevent duplicate inserts.
• Conditional writes — Use IF NOT EXISTS or version checks.
• Semantic idempotency — Design operations to produce the same result on retry.

Testing Idempotency

1. Send the same message twice and verify the result is identical.
2. Simulate consumer crashes during processing and verify no duplicates.
3. Test with concurrent consumers processing the same message.
4. Verify deduplication keys expire after a reasonable period.