Async Observability

Tracing message hops, queue depth, consumer lag, and poison-message handling with OpenTelemetry.

6 min readSystem Design for SREProvision

Observability Challenges in Async Systems

Async systems are harder to observe than sync systems. Request traces break at queue boundaries. You cannot follow a single request through the entire pipeline. Instead, you must trace individual hops and correlate them using message IDs and trace context.

Message Tracing

Propagate trace context through messages by injecting OpenTelemetry trace headers into message attributes. Each consumer extracts the context and creates a child span. This connects the producer span with the consumer span into a single trace.

Trace context propagation in Kafka


from opentelemetry import trace
from opentelemetry.propagate import inject

tracer = trace.get_tracer("order-service")

def produce_order(event):
    with tracer.start_as_current_span("produce-order") as span:
        # Inject trace context into message headers
        headers = {}
        inject(headers)  # Adds traceparent header
        
        producer.send('orders', value=event, headers=headers)

def consume_order(message):
    # Extract trace context from message headers
    from opentelemetry.propagate import extract
    context = extract(dict(message.headers))
    
    with tracer.start_as_current_span("consume-order", context=context) as span:
        process_order(message.value)

The inject/extract functions propagate trace context through Kafka headers. This connects producer and consumer spans into a single distributed trace.

Queue Metrics

Queue depth — Number of messages in the queue. Growing = backpressure.
Queue age — Age of the oldest unprocessed message. Indicates processing delay.
Message rate — Messages produced/consumed per second.
Message size — Average message size. Large messages cause slow processing.
Dead-letter rate — Messages moving to DLQ per second.

Consumer Metrics

Consumer lag alert


# Alert when consumer lag exceeds 10K messages
kafka_consumer_lag > 10000

# Alert when consumer lag is growing for 5 minutes
deriv(kafka_consumer_lag[5m]) > 0

# Alert when consumer processing rate drops
rate(kafka_consumer_messages_processed_total[5m]) < 100

Consumer lag is the most critical async metric. High lag means messages are delayed. Growing lag means the consumer is falling behind.

Poison Message Detection

A poison message causes every consumer attempt to fail. Without detection, poison messages block the queue. Monitor for messages that fail repeatedly and automatically move them to a dead-letter queue after a configured threshold.

Poison message detection


def handle_message(message):
    failure_count = get_failure_count(message.id)
    
    if failure_count >= 3:
        log.error(f"Poison message detected: {message.id}")
        move_to_dlq(message)
        alert("poison_message", message_id=message.id)
        return
    
    try:
        process(message)
        clear_failure_count(message.id)
    except Exception as e:
        increment_failure_count(message.id)
        log.warning(f"Processing failed ({failure_count + 1}/3): {e}")

Track failure count per message. After 3 failures, treat as poison and move to DLQ. This prevents poison messages from blocking the queue.

Async System Dashboards

Queue health — Depth, age, production/consumption rates.
Consumer health — Lag, processing rate, error rate.
End-to-end — Time from message production to processing completion.
DLQ — Messages arriving in DLQ, reasons for failure.

Trace async flows end-to-end

Use OpenTelemetry context propagation to trace messages from producer through queue to consumer. This gives you end-to-end visibility into async pipelines that is impossible to achieve with logs alone.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 4 · Provision

Asynchronous Systems