Stage 4 · Provision
Design Case Studies
Design a Metrics Pipeline
Scraping, remote_write, aggregation, cardinality limits, and long-term storage with Thanos.
Requirements
A metrics pipeline collects, stores, aggregates, and serves time-series metrics from hundreds of services. It must handle millions of time series, provide sub-second query latency, and retain data for months. The pipeline itself must be highly available.
Metrics Collection
Applications
│
├── Prometheus (pull model)
│ ├── Scrapes /metrics endpoint every 15s
│ ├── Stores locally (2h block)
│ └── remote_write to long-term storage
│
├── OpenTelemetry Collector (push model)
│ ├── Receives metrics via OTLP
│ ├── Processes (filter, transform, sample)
│ └── Exports to Prometheus / Thanos
│
└── Node Exporter / cAdvisor
├── System metrics (CPU, memory, disk)
└── Container metrics (Kubernetes)Prometheus scrapes application metrics. OpenTelemetry Collector receives push-based metrics. Both write to long-term storage for historical queries.
Time-Series Storage
| Storage | Type | Retention | Scale |
|---|---|---|---|
| Prometheus | Local TSDB | Hours-days | Single node |
| Thanos | Object storage | Months-years | Distributed |
| Cortex | Object storage | Months-years | Distributed |
| Mimir | Object storage | Months-years | Distributed |
| InfluxDB | Built-in | Configurable | Cluster |
Aggregation Layers
Aggregation reduces data volume by pre-computing summaries. Instead of querying raw data points, query pre-aggregated averages, percentiles, and counts. Record rules in Prometheus pre-aggregate data at write time.
groups:
- name: api Aggregation
interval: 30s
rules:
- record: api:request_rate:rate5m
expr: sum(rate(http_requests_total[5m])) by (service)
- record: api:error_rate:rate5m
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
- record: api:latency:p99:rate5m
expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))Recording rules pre-compute expensive queries. Instead of calculating request rate on every dashboard refresh, the pre-aggregated value is ready instantly.
Cardinality Management
Cardinality is the number of unique time series. High-cardinality labels (user_id, request_id) create millions of time series that overwhelm storage and query. Control cardinality by limiting label values.
- Never use user_id as a label — Use request-level metrics instead.
- Limit label values — Enumerate known values, reject unknowns.
- Drop high-cardinality labels — Filter at the collector level.
- Alert on cardinality growth — Track series count per metric.
Long-Term Storage with Thanos
Prometheus instances
│
├── remote_write ──► Thanos Sidecar ──► Object Storage (S3)
│ │
│ ├── Thanos Store Gateway
│ │ └── Serves historical data
│ │
│ └── Thanos Query
│ └── Federates across all stores
│
└── Thanos Compactor
└── Downsamples and compacts old dataThanos extends Prometheus with long-term object storage, global query federation, downsampling, and compaction. It provides months of retention at low cost.
Prometheus alone handles most workloads for months. When you need long-term retention (>15 days) or global query federation, add Thanos or Mimir. Do not over-engineer the metrics pipeline from day one.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.