Event Streams & Queues

Streams vs Queues

Event streams and message queues both decouple producers from consumers, but they serve different purposes. Streams are append-only logs that multiple consumers can read independently. Queues deliver each message to exactly one consumer. Streams preserve history; queues are ephemeral.

Kafka Fundamentals

Kafka is a distributed event streaming platform. Data is organized into topics, which are split into partitions. Each partition is an ordered, immutable log. Consumers read from partitions at their own pace, tracking their position with offsets.

Topic: user-events (3 partitions)

Partition 0: [e1, e4, e7, e10, ...]  offset 0,1,2,3
Partition 1: [e2, e5, e8, e11, ...]  offset 0,1,2,3
Partition 2: [e3, e6, e9, e12, ...]  offset 0,1,2,3

Consumer Group "analytics":
  Consumer 1 reads Partition 0 (offset 3)
  Consumer 2 reads Partition 1 (offset 3)
  Consumer 3 reads Partition 2 (offset 2)

Each consumer processes independently, at its own pace.

RabbitMQ

RabbitMQ is a traditional message broker that implements AMQP. It supports complex routing, message acknowledgments, and priority queues. RabbitMQ is better than Kafka for task distribution where each message should be processed exactly once.

• Exchanges route messages to queues based on rules (direct, topic, fanout).
• Consumers acknowledge messages after processing.
• Dead-letter queues capture failed messages.
• Priority queues ensure important messages are processed first.

Amazon SQS

SQS is a fully managed message queue. It supports standard queues (at-least-once delivery, best-effort ordering) and FIFO queues (exactly-once, strict ordering). SQS scales automatically and requires zero operational overhead.

Exactly-Once Semantics

Exactly-once delivery is the holy grail of messaging. In practice, true exactly-once delivery between independent systems is impossible. Instead, achieve effectively-once semantics through idempotent consumers and deduplication.

import redis
import hashlib

r = redis.Redis()

def process_event(event):
    # Create deduplication key
    dedup_key = f"processed:{hashlib.sha256(event['id'].encode()).hexdigest()}"
    
    # Check if already processed (SET NX = not exists)
    if not r.set(dedup_key, "1", nx=True, ex=86400):
        return  # Already processed — skip
    
    try:
        # Process the event
        handle_payment(event)
    except Exception as e:
        # Delete dedup key to allow retry
        r.delete(dedup_key)
        raise

Choosing a Messaging System