Distributed Tracing Basics

One Request, Four Services, One Trace

Fieldwork already had request IDs before tracing. They helped inside one service, but they were a weak substitute once the request crossed multiple boundaries. A create-task request might enter through the API gateway, hit the task service, call memberships for authorization, write to Postgres, and then publish a task.created event through the outbox flow that eventually wakes a notification worker. A request ID logged in one place did not automatically follow that path. We could manually correlate logs, but every incident felt like reconstructing a crime scene from partial timestamps.

That is why we chose tracing even before the system was huge. The alternative was to keep improving log correlation rules, but that keeps humans in the loop for a job tooling should do. A trace gives us a parent-child timeline across services: which hop started first, where latency accumulated, and whether the expensive part was permission lookup, database work, or downstream event publishing. In other words, tracing turned "the request was slow somewhere" into a concrete graph.

Propagation Has to Survive HTTP and Kafka

HTTP propagation is the easy part because libraries already understand W3C trace context. The harder part in Fieldwork was the async path. When the task service writes an outbox event and a worker later publishes it to Kafka, the trace cannot stop at the database transaction if we want the eventual notification span connected to the original task creation. We rejected the idea of treating async work as a brand-new trace because it hides causality exactly where production systems become hardest to reason about.

The answer was to persist trace headers alongside the outbox payload and copy them into Kafka message headers on publish. Consumers then extract the remote span context and continue the trace with a linked or child span, depending on the workflow. That design is a little more ceremony, but it paid for itself the first time we had to explain why notification delivery lagged long after the original HTTP request completed.

Spans Need Domain Names, Not Framework Noise

Another early mistake was naming spans after implementation details that meant little during investigation. Seeing middleware-3 or http-handler told us almost nothing. We renamed spans around domain work: task.create, membership.authorize-create-task, postgres.insert-task, outbox.enqueue-task-created, notifications.consume-task-created. Those names are longer than generic middleware spans, but they are the names an operator or reviewer already uses when talking about the system. That makes the trace readable without opening source first.

We also kept attribute discipline. tenant.id, workspace.id, project.id, task.id after creation, dependency.name, retry.attempt, and outbox.event_type were useful. Full task descriptions were not. Just like metrics, tracing punishes undisciplined cardinality and sensitive-data leakage. The advantage is that good attributes let you slice traces by meaningful production questions without turning the trace store into a second logging system.

What Tracing Told Us That Logs Didn't

The first real payoff came from a slow create-task path. Metrics showed p95 latency rising. Logs proved memberships and Postgres were both involved somewhere. The trace showed the actual shape: membership authorization was fast, the insert was fine, but the outbox enqueue span waited on a saturated connection because the transaction held one too long while serializing a large payload. That is a specific, fixable story. Without the trace, we would have kept arguing about which subsystem felt suspicious.

So the decision here was not "adopt OpenTelemetry because modern systems should have tracing." The decision was to model one request's causal path across gateway, internal services, database-backed async boundaries, and workers. That required propagation discipline, domain span names, and careful attributes. Once those were in place, traces stopped being pretty screenshots and became a practical debugging tool for tenant-scoped production requests.

• Start traces at the first reliable boundary, usually the gateway or HTTP ingress middleware.
• Reuse request context everywhere so HTTP and database spans remain attached to the same trace.
• Persist and forward trace headers through outbox and Kafka boundaries instead of starting disconnected traces.
• Name spans after domain operations and dependency edges, not generic framework stages.
• Keep attributes useful and bounded; traces are not a place to dump arbitrary payload data.