Stage 7 · Master
Phase 17 — Observability
Distributed Tracing
Stand up Jaeger, watch one HTTP request's spans arrive as a real trace, then carry that same trace through Part 16's Kafka pipeline — across an outbox write, a relay publish, and an asynchronous consumer — so one payment.completed event is visible in Jaeger as a single, continuous trace, not two disconnected ones.
Every Span This Module Has Created So Far Has Had Nowhere to Go Until Now
The previous lesson's otlptracegrpc exporter has been pointed at jaeger:4317 since it was written, but this module has not yet defined what jaeger actually is. This lesson adds it to the compose stack — using Jaeger's own native OTLP ingestion, so no separate OpenTelemetry Collector process sits between this service and the trace backend.
jaeger:
image: jaegertracing/all-in-one:1.60
restart: unless-stopped
environment:
COLLECTOR_OTLP_ENABLED: "true"
ports:
- "16686:16686" # web UI
- "4317:4317" # OTLP gRPC, matches pkg/otelinit's collectorAddr
networks:
- hoa-net
One POST to /api/v1/organizations Produces a Small Tree of Spans, Not One Flat Line
middleware.Tracing starts the root span for the request itself. OrganizationService.Create's span attribute (Lesson 4) attaches to that same span rather than creating a new one, since Create does not yet start its own child span — a deliberate choice, since Create's own work is a single repository call with no internal sub-steps worth breaking out separately. A future service method with multiple genuinely distinct steps (a cache lookup, then a database fallback, say) would start its own child spans via tracer.Start(ctx, "step-name") for each.
The Trace Doesn't Have to End When the HTTP Response Does
Part 16's Relay publishes payment.completed asynchronously, seconds or minutes after the HTTP request that created the outbox row has already returned a response. Without deliberate propagation, that gap breaks the trace in two: one trace for the HTTP request, an entirely separate, disconnected one (or none at all) for the eventual Kafka publish and consumption. Kafka message headers ([]kafka.Header) can carry a serialized trace context exactly the way an HTTP header does — this lesson threads Part 16's own outbox and relay code with that propagation, so one payment.completed event is traceable end to end as a single trace in Jaeger: the HTTP request that wrote it, the relay that published it, and the consumer that turned it into a notification job.
payment_outbox Gains a trace_context Column
ALTER TABLE payment_outbox ADD COLUMN trace_context TEXT;ALTER TABLE payment_outbox DROP COLUMN trace_context; func (w *OutboxWriter) WritePaymentCompleted(ctx context.Context, tx pgx.Tx, orgID, paymentID uuid.UUID, payload CompletedPayload) error {
// envelope marshalling from the Kafka producers lesson is unchanged
+ // A database row cannot carry HTTP-style headers, so the propagator
+ // writes into a plain map and that map is stored as a column.
+ carrier := propagation.MapCarrier{}
+ otel.GetTextMapPropagator().Inject(ctx, carrier)
+ traceContext, err := json.Marshal(carrier)
+ if err != nil {
+ return err
+ }
+
_, err = tx.Exec(ctx, `
- INSERT INTO payment_outbox (org_id, aggregate_id, topic, event_type, event_version, partition_key, payload)
- VALUES ($1, $2, $3, $4, $5, $6, $7)`,
- orgID, paymentID, "payment.completed", "payment.completed", 1, paymentID.String(), body,
+ INSERT INTO payment_outbox (org_id, aggregate_id, topic, event_type, event_version, partition_key, payload, trace_context)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8)`,
+ orgID, paymentID, "payment.completed", "payment.completed", 1, paymentID.String(), body, traceContext,
)
return err
}
Relay Injects Into Kafka Headers; the Consumer Extracts and Continues the Trace
for _, row := range rows {
var carrier propagation.MapCarrier
if err := json.Unmarshal(row.TraceContext, &carrier); err != nil {
carrier = propagation.MapCarrier{} // a row written before this lesson has no trace_context; publish untraced rather than fail
}
ctx := otel.GetTextMapPropagator().Extract(ctx, carrier)
ctx, span := tracer.Start(ctx, "relay.publish")
headers := make([]kafkago.Header, 0, len(carrier))
for k, v := range carrier {
headers = append(headers, kafkago.Header{Key: k, Value: []byte(v)})
}
err := writer.WriteMessages(ctx, kafkago.Message{
Key: []byte(row.PartitionKey),
Value: row.Payload,
Headers: headers,
})
span.End()
if err != nil {
return err
}
}
A row from before this lesson shipped simply publishes without a parent trace — the consumer below still works identically, it just starts a fresh trace instead of continuing one, exactly like Lesson 4's handling of a missing HTTP traceparent header.
carrier := propagation.MapCarrier{}
for _, h := range msg.Headers {
carrier[h.Key] = string(h.Value)
}
ctx := otel.GetTextMapPropagator().Extract(ctx, carrier)
ctx, span := tracer.Start(ctx, "consumer.HandlePaymentCompleted")
err := processWithRetry(ctx, handler, msg.Topic, msg.Value)
span.End()
if err != nil {
return err
}
The span above wraps processWithRetry entirely, including the case where errPoisonEvent is returned on the very first attempt. This is deliberate: a trace showing a span that failed instantly with a poison-event error is exactly the diagnostic signal an operator wants when investigating why one particular payment's notification never arrived — losing that span because the event turned out to be malformed would hide the one piece of evidence explaining the failure.
Watch One Trace Span the HTTP Request, the Relay, and the Consumer in Jaeger's Own UI
Applied exercise
Confirm a pre-migration outbox row degrades gracefully instead of breaking the relay
relay.go's fallback for a row with no trace_context (json.Unmarshal failing on a NULL column) is supposed to publish untraced rather than crash the whole relay loop. Prove that specifically.
- Manually insert one row directly into payment_outbox with trace_context left NULL, bypassing OutboxWriter entirely (simulating a row written before migration 0057 was ever applied).
- Run the relay and confirm it publishes that row successfully rather than erroring out or skipping the whole batch.
- Confirm in Jaeger (or by the absence of a matching trace) that this specific event started a fresh, unlinked trace rather than causing a visible failure anywhere in the pipeline.
- Explain in one paragraph why carrier = propagation.MapCarrier{} (an empty carrier) is the correct fallback value, rather than skipping the row or returning an error from the relay loop.
Deliverable
Confirmation the row was published successfully, plus the one-paragraph explanation.
Completion checks
- The pre-existing NULL-trace_context row is published without error and without stalling any other row in the same batch.
- The explanation correctly identifies that an empty carrier degrades to 'start a new trace,' which is a strictly safer failure mode than either skipping a real payment event or crashing the relay over a missing observability detail.
Why does payment_outbox need its own trace_context column instead of the relay simply calling otel.GetTextMapPropagator().Inject at publish time, using whatever context the relay's own goroutine happens to be running in?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.