Stage 7 · Master
Phase 17 — Observability
OpenTelemetry
Give organization-service a TracerProvider, a propagator, and a request-scoped span — the plumbing every later tracing lesson in this module depends on — built by hand, not by importing a framework's pre-wired middleware.
hoa_organization_postgres_pool_acquired_connections Tells You Pressure Exists, Not Which Request Felt It
This module's Prometheus lesson can tell an operator the pool was near its ceiling at 09:14. It cannot tell them which specific request stalled on Acquire, how long that one request's database query itself took once it got a connection, or whether the delay originated in this service at all versus a slow call it made outward. A metric answers 'is something wrong, in aggregate.' A trace answers 'what exactly happened, to this one request, across every component it touched' — a fundamentally different, complementary signal this module has not introduced until now.
pkg/otelinit: One Constructor, Reused By Every Service That Adopts It
// Package otelinit builds one TracerProvider per service and sets it as
// the global default, alongside a W3C-standard propagator, so every
// service adopting OpenTelemetry does it identically instead of each
// reinventing exporter configuration.
package otelinit
import (
"context"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
// Shutdown flushes any spans still queued in the batch processor and must
// be called during graceful shutdown, before the process exits — spans
// created in the final seconds before shutdown are otherwise silently lost.
type Shutdown func(context.Context) error
// Init points the OTLP exporter at collectorAddr (this module's Jaeger
// container, listening for OTLP directly — no separate collector process
// needed). A 5-second connect/export timeout bounds the worst case: if the
// exporter is unreachable, spans are dropped after this timeout rather
// than blocking application goroutines indefinitely.
func Init(ctx context.Context, service, collectorAddr string) (Shutdown, error) {
exporter, err := otlptracegrpc.New(ctx,
otlptracegrpc.WithEndpoint(collectorAddr),
otlptracegrpc.WithInsecure(),
otlptracegrpc.WithTimeout(5*time.Second),
)
if err != nil {
return nil, err
}
res, err := resource.New(ctx, resource.WithAttributes(semconv.ServiceName(service)))
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(res),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, // W3C traceparent — what this module propagates everywhere
propagation.Baggage{},
))
return tp.Shutdown, nil
}
middleware.Tracing Extracts an Incoming Context and Starts One Span Per Request
A pre-built framework middleware would work, but this course has built every other piece of cross-cutting behavior — request ids, metrics, error envelopes — by hand, specifically so each lesson can show exactly what the wrapper would otherwise hide. Tracing gets the same treatment: this middleware is a dozen lines using otel's own extraction and span APIs directly.
package middleware
import (
"github.com/gin-gonic/gin"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/trace"
)
var tracer = otel.Tracer("organization")
// Tracing extracts any incoming W3C traceparent header via the global
// propagator (set once in pkg/otelinit.Init) and starts a span as its
// child. A request with no traceparent header at all — the common case
// for any request entering this platform from outside — extracts an
// empty span context, and Start simply begins a brand new trace rather
// than failing; there is no such thing as a malformed trace here, only
// the presence or absence of a parent to attach to.
func Tracing() gin.HandlerFunc {
return func(c *gin.Context) {
ctx := otel.GetTextMapPropagator().Extract(
c.Request.Context(),
propagation.HeaderCarrier(c.Request.Header),
)
ctx, span := tracer.Start(ctx, c.FullPath(),
trace.WithSpanKind(trace.SpanKindServer),
)
defer span.End()
c.Request = c.Request.WithContext(ctx)
c.Next()
if c.Writer.Status() >= 500 {
span.SetStatus(codes.Error, "request failed")
}
}
}
func New(orgHandler *handler.OrganizationHandler, m *metrics.Metrics, logger zerolog.Logger) *gin.Engine {
engine := gin.New()
engine.Use(middleware.Tracing())
engine.Use(middleware.RequestID(logger))
engine.Use(gin.Recovery())
engine.Use(middleware.ErrorHandler())
engine.Use(m.Middleware())
// routes unchanged below this line
Tracing runs first so RequestID can bind the active trace id into the context-scoped logger. Both still wrap Recovery and ErrorHandler, so failed and recovered requests finish the same server span with the final response status.
trace_id Joins request_id in the Contract This Module's First Lesson Defined
Lesson 1 listed trace_id as a required field this lesson would add. ForRequest gains a second parameter so every log line written inside a traced request carries both ids, letting a support engineer move from a log line straight to the exact trace in Jaeger without a separate lookup step.
func ForRequest(base zerolog.Logger, requestID, traceID string) zerolog.Logger {
ctx := base.With()
if requestID != "" {
ctx = ctx.Str("request_id", requestID)
}
if traceID != "" {
ctx = ctx.Str("trace_id", traceID)
}
return ctx.Logger()
}
id := c.GetHeader("X-Request-ID")
if id == "" {
id = uuid.NewString()
}
c.Header("X-Request-ID", id)
ctx := context.WithValue(c.Request.Context(), requestIDKey, id)
traceID := trace.SpanContextFromContext(c.Request.Context()).TraceID().String()
ctx = logging.NewContext(ctx, logging.ForRequest(logger, id, traceID))
c.Request = c.Request.WithContext(ctx)
This reads trace.SpanContextFromContext(c.Request.Context()) — the context Tracing's own middleware already populated one line earlier in router.go's Use chain — never the base context main started with.
main.go Gains Init at Startup and Shutdown Before the Process Exits
shutdownTracing, err := otelinit.Init(context.Background(), "organization", "jaeger:4317")
if err != nil {
logger.Error().Err(err).Msg("otel init failed")
os.Exit(1)
}
defer func() {
if err := shutdownTracing(context.Background()); err != nil {
logger.Error().Err(err).Msg("otel shutdown failed")
}
}()
// pool, repo, svc, handlers, router, internal server — all unchanged
// below this line
org_id Is a Span Attribute Here — the Opposite Rule From Metrics
This module's Prometheus lesson explicitly forbids labeling any metric by org_id, since Prometheus keeps one time series per distinct label combination forever. A trace's attributes carry no such cost — Jaeger indexes and stores each trace independently, so a span attribute unique to one request costs nothing extra for the next request's span. OrganizationService.Create (modified again, this time to record a span attribute rather than a log field) demonstrates the contrast directly.
span := trace.SpanFromContext(ctx)
span.SetAttributes(attribute.String("org_id", org.ID.String()))
Confirm a Span Exists and Carries Both the org_id Attribute and This Request's trace_id
Applied exercise
Prove a request with no incoming traceparent header still gets a valid trace
Tracing's extraction step is supposed to handle a missing header gracefully by starting a fresh trace, not by erroring. Prove that specifically, since almost every request this service receives today has no incoming traceparent at all.
- Send a normal curl request to any endpoint with no manually-added headers (the default case, since nothing upstream of this service currently sets traceparent).
- Confirm from the logged trace_id that a real, non-zero trace id was generated despite no header being present.
- Now manually add a deliberately malformed header, -H 'traceparent: not-a-real-value', and confirm the request still succeeds and still logs a valid trace_id rather than failing the request.
- Explain in one sentence why Extract must never turn a malformed or absent header into a request-level error.
Deliverable
The two trace_id values (no header, malformed header) plus the one-sentence explanation.
Completion checks
- Both requests succeed with HTTP 2xx and both produce a valid, non-empty trace_id.
- The explanation correctly identifies that tracing is an observability concern layered on top of the request, not a precondition for serving it.
Why does Tracing extract the incoming context before starting a new span, rather than always starting a brand-new trace for every request?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.