Stage 5 · Platform
Observability & Autoscaling
OpenTelemetry Tracing
Collectors, auto-instrumentation, tail sampling, and trace correlation in Kubernetes.
OpenTelemetry Overview
OpenTelemetry (OTel) is the standard for observability instrumentation. It provides APIs and SDKs for traces, metrics, and logs. The OTel Collector receives telemetry data, processes it, and exports it to backends (Jaeger, Prometheus, Grafana Tempo).
| Signal | Purpose | Backend |
|---|---|---|
| Traces | Request flow across services | Jaeger, Tempo, Zipkin |
| Metrics | Numeric measurements | Prometheus, Mimir, Datadog |
| Logs | Structured event records | Loki, Elasticsearch, Splunk |
OTel Collector
The OTel Collector is a vendor-agnostic proxy for telemetry data. It receives data from instrumented applications (receivers), processes it (processors), and exports it to backends (exporters). Deploy it as a DaemonSet or Deployment.
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
namespace: observability
data:
config.yaml: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 8192
memory_limiter:
check_interval: 1s
limit_mib: 2048
spike_limit_mib: 512
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
- name: error-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow-traces
type: latency
latency: {threshold_ms: 1000}
exporters:
otlp/jaeger:
endpoint: jaeger-collector:4317
tls:
insecure: true
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [otlp/jaeger]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]The OTel Collector receives OTLP traces and metrics. tail_sampling keeps all error traces and traces slower than 1s. batch groups spans for efficient export. memory_limiter prevents OOM.
Auto-Instrumentation
Auto-instrumentation injects OTel SDKs into pods without code changes. The operator injects init containers and sidecars that instrument HTTP clients, database drivers, and gRPC. This works for Java, Python, Node.js, Go, and .NET.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: production
spec:
template:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
instrumentation.opentelemetry.io/inject-python: "true"
sidecar.istio.io/inject: "true"
spec:
containers:
- name: java-app
image: my-java-app:1.0
- name: python-worker
image: my-python-worker:1.0
---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-instrumentation
namespace: production
spec:
exporter:
endpoint: otel-collector.observability:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: 0.1 # Sample 10% of traces
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latestThe Instrumentation resource defines OTel configuration. The annotation injects the appropriate language SDK. sampler argument controls the head-based sampling ratio. propagators define trace context format.
Head-based sampling decides at trace creation whether to sample (configurable, may miss errors). Tail-based sampling decides after seeing the full trace (catches all errors, but requires state). Use head-based for cost control, tail-based for completeness.
Tail Sampling
Tail sampling keeps traces based on their characteristics: errors, slow requests, specific operations. This provides complete visibility into problematic traces while reducing volume. The OTel Collector's tail_sampling processor handles this.
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
# Always keep error traces
- name: errors
type: status_code
status_code: {status_codes: [ERROR]}
# Keep slow traces (>1s)
- name: slow
type: latency
latency: {threshold_ms: 1000}
# Keep traces touching specific services
- name: critical-service
type: string_attribute
string_attribute: {key: service.name, values: ["payment-api", "auth-service"]}
# Sample 1% of everything else
- name: probabilistic
type: probabilistic
probabilistic: {sampling_percentage: 1}
# Composite: combine policies
- name: composite
type: composite
composite:
max_total_spans_per_second: 5000
policy_order: [errors, slow, critical-service, probabilistic]
rate_allocation:
- policy: errors
percent: 30
- policy: slow
percent: 20
- policy: critical-service
percent: 20
- policy: probabilistic
percent: 30The composite policy combines multiple policies with rate allocation. 30% of capacity for errors, 20% for slow traces, 20% for critical services, 30% for probabilistic sampling. This ensures important traces are always captured.
Trace-Metrics Correlation
Exemplars link traces to metrics. When you see a spike in latency, you can click through to a specific trace that caused it. This provides immediate root cause analysis without switching between tools.
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: main
spec:
enableFeatures:
- exemplar-storage
retention: 15d
externalLabels:
cluster: production
remoteWrite:
- url: "https://mimir.example.com/api/v1/push"
sendExemplars: trueenableFeatures: exemplar-storage stores trace IDs with metrics. sendExemplars forwards exemplars to remote storage. In Grafana, you can click an exemplar on a metric panel to open the trace in Jaeger/Tempo.
Backend with Jaeger
Jaeger is a distributed tracing backend. It stores traces, provides a UI for trace analysis, and supports dependency graphs. Deploy Jaeger with the Jaeger Operator for automatic collector and query deployment.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger
namespace: observability
spec:
strategy: production
collector:
replicas: 3
resources:
requests:
cpu: "500m"
memory: "512Mi"
storage:
type: elasticsearch
options:
es:
server-urls: https://elasticsearch:9200
index-prefix: jaeger
esIndexCleaner:
enabled: true
numberOfDays: 14
schedule: "55 23 * * *"
query:
replicas: 2production strategy separates collector and query into different deployments. esIndexCleaner deletes traces older than 14 days. This keeps storage costs manageable while retaining enough history for debugging.
Ensure trace context propagates across services. Istio adds trace headers automatically. For non-Istio services, use W3C Trace Context (traceparent header). The OTel SDK propagates context by default when auto-instrumented.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.