Stage 7 · Master
Observability & Incident Tools
OpenTelemetry for Python
Auto-instrumentation, custom spans, trace propagation, and exporting to Jaeger.
OpenTelemetry Overview
OpenTelemetry (OTel) is the standard for observability instrumentation. It provides a single API for traces, metrics, and logs. Instrument once, export to any backend (Jaeger, Prometheus, Datadog, etc.).
| Signal | What It Tracks | Backend Example |
|---|---|---|
| Traces | Request flow across services | Jaeger, Zipkin, Tempo |
| Metrics | Numeric time series | Prometheus, Datadog |
| Logs | Discrete events | Loki, Elasticsearch |
# Core OTel SDK
pip install opentelemetry-api opentelemetry-sdk
# Auto-instrumentation for FastAPI
pip install opentelemetry-instrumentation-fastapi
pip install opentelemetry-exporter-otlp
# Auto-instrumentation for requests/httpx
pip install opentelemetry-instrumentation-requests
pip install opentelemetry-instrumentation-httpxInstall the core SDK plus instrumentation libraries for your framework. Auto-instrumentation wraps common libraries automatically without code changes.
Auto-Instrumentation
# main.py
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
app = FastAPI()
# Auto-instrument FastAPI — traces all requests
FastAPIInstrumentor.instrument_app(app)
# Auto-instrument requests library — traces outgoing HTTP calls
RequestsInstrumentor().instrument()
@app.get("/health")
def health():
return {"status": "healthy"}
@app.get("/api/data")
def get_data():
# This call will be traced automatically
import requests
resp = requests.get("http://other-service:8080/data")
return resp.json()Auto-instrumentation wraps framework entry points and creates spans for each request. Outgoing HTTP calls are also traced. No code changes required.
# Set these environment variables before running the app:
# OTEL_SERVICE_NAME=my-api
# OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
# OTEL_EXPORTER_OTLP_PROTOCOL=grpc
# OTEL_TRACES_EXPORTER=otlp
# OTEL_METRICS_EXPORTER=otlp
# OTEL_LOGS_EXPORTER=otlp
# Or configure programmatically:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://jaeger:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)Environment variables are the recommended way to configure OTel. They work with auto-instrumentation and do not require code changes. Use programmatic configuration when you need custom behavior.
Custom Spans
from opentelemetry import trace
import time
# Get a tracer for your module
tracer = trace.get_tracer(__name__)
def process_order(order_id: str) -> dict:
"""Process an order with custom spans."""
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
# Child span for validation
with tracer.start_as_current_span("validate_order") as child_span:
child_span.set_attribute("order.validation", "schema")
validate_order(order_id)
child_span.add_event("validation_passed")
# Child span for database write
with tracer.start_as_current_span("save_to_db") as db_span:
db_span.set_attribute("db.system", "postgresql")
db_span.set_attribute("db.operation", "insert")
save_order(order_id)
db_span.add_event("order_saved", {"order_id": order_id})
span.set_attribute("order.status", "processed")
return {"status": "ok", "order_id": order_id}
def validate_order(order_id: str):
time.sleep(0.01) # Simulate work
def save_order(order_id: str):
time.sleep(0.02) # Simulate DB writeCustom spans wrap specific operations. Attributes provide context (order ID, database system). Events mark points in time (validation passed, order saved).
Context Propagation
from opentelemetry import trace
from opentelemetry.propagate import inject, extract
import httpx
tracer = trace.get_tracer(__name__)
def call_service_b(url: str):
"""Call service B with trace context propagation."""
with tracer.start_as_current_span("call_service_b") as span:
# Inject trace context into headers
headers = {}
inject(headers)
resp = httpx.get(url, headers=headers)
span.set_attribute("http.status_code", resp.status_code)
return resp.json()
# On the receiving end (Service B):
def handle_request(request):
"""Extract trace context from incoming request."""
# Extract trace context from headers
ctx = extract(request.headers)
with tracer.start_as_current_span("handle_request", context=ctx) as span:
span.set_attribute("http.method", request.method)
return {"status": "ok"}Context propagation links spans across services. inject() adds trace headers to outgoing requests. extract() reads them from incoming requests. This creates a distributed trace.
Metrics with OTel
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
# Create a meter
reader = PeriodicExportingMetricReader(exporter=OTLPMetricExporter())
provider = MeterProvider(metric_readers=[reader])
meter = provider.get_meter("my-app", "1.0.0")
# Define instruments
request_counter = meter.create_counter(
"http.requests",
description="Number of HTTP requests",
unit="1",
)
request_duration = meter.create_histogram(
"http.request.duration",
description="HTTP request duration",
unit="ms",
)
active_requests = meter.create_up_down_counter(
"http.active_requests",
description="Number of active requests",
)
# Use in code
request_counter.add(1, {"method": "GET", "status": "200"})
request_duration.record(250, {"method": "GET"})
active_requests.add(1)
# ... handle request ...
active_requests.add(-1)OTel metrics work like Prometheus metrics but use the OTel SDK. Export to any backend. The MeterProvider manages metric collection and export.
Exporting to Jaeger
services:
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686" # UI
- "14250:14250" # gRPC
otel-collector:
image: otel/opentelemetry-collector:latest
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
my-api:
image: my-api:latest
environment:
- OTEL_SERVICE_NAME=my-api
- OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
- OTEL_EXPORTER_OTLP_PROTOCOL=grpc
- OTEL_TRACES_EXPORTER=otlp
depends_on:
- otel-collectorThe OTel Collector receives traces from your application and forwards them to Jaeger. This decouples your application from the trace backend. You can switch backends without changing code.
OTel lets you instrument once and export to any backend. If you switch from Jaeger to Datadog, you only change the exporter configuration, not the instrumentation code.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.