Stage 5 · Platform
Observability & Autoscaling
Grafana SLO Dashboards
Golden signals, RED/USE panels, burn-rate views, and error budget reporting.
Golden Signals
The four golden signals (from Google SRE) are: Latency (time to serve a request), Traffic (demand on the system), Errors (rate of failed requests), and Saturation (how full the system is). Monitor these for every service to quickly detect and diagnose issues.
| Signal | Metric | What It Tells You |
|---|---|---|
| Latency | request_duration_seconds | How fast responses are served |
| Traffic | requests_total | How much demand exists |
| Errors | http_requests_total{code=~'5..'} | What fraction of requests fail |
| Saturation | cpu_usage, memory_usage | How close to capacity |
RED Method
RED (Rate, Errors, Duration) focuses on request-driven services. It's simpler than the four golden signals and maps directly to Prometheus metrics. Rate = requests per second, Errors = error rate, Duration = request latency.
# Rate (requests per second)
# Panel: Requests/sec
expr: sum(rate(http_requests_total[5m])) by (service)
# Errors (error rate %)
# Panel: Error Rate
expr: |
sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
# Duration (p99 latency)
# Panel: p99 Latency
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)These three queries give you a complete view of service health. Rate shows demand, Errors shows failures, Duration shows performance. Combine them in a single dashboard with one panel per service.
USE Method
USE (Utilization, Saturation, Errors) focuses on infrastructure: nodes, disks, networks, and CPUs. Utilization = % of capacity used, Saturation = queue depth, Errors = hardware/OS errors. USE is for infrastructure; RED is for services.
# CPU Utilization
expr: 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (node)
# Memory Utilization
expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
# Disk Saturation (I/O queue depth)
expr: node_disk_io_time_weighted_seconds_total
# Network Errors
expr: rate(node_network_receive_errs_total[5m]) + rate(node_network_transmit_errs_total[5m])USE metrics are available from node-exporter. CPU utilization should stay below 80%. Memory utilization below 85%. Disk I/O queue depth near 0 is healthy. Network errors should be zero.
SLO Dashboards
SLO (Service Level Objective) dashboards show compliance with reliability targets. They display: current SLO achievement, error budget remaining, burn rate trends, and time remaining before budget exhaustion.
# SLO Achievement (% of successful requests)
expr: |
(
1 - (
sum(rate(http_requests_total{code=~"5.."}[30d])) by (service)
/
sum(rate(http_requests_total[30d])) by (service)
)
) * 100
# Error Budget Remaining (seconds)
expr: |
(
(1 - 0.999) * 30 * 24 * 3600
-
sum(increase(http_requests_total{code=~"5.."}[30d])) by (service)
/
sum(rate(http_requests_total[30d])) by (service) * 30 * 24 * 3600
)SLO achievement shows the percentage of successful requests over the SLO window (30d). Error budget remaining shows how many seconds of errors you have left. When budget reaches zero, you must halt feature releases and focus on reliability.
Burn Rate
Burn rate measures how fast you're consuming your error budget. A burn rate of 1 means you're consuming budget at the exact rate that would exhaust it by the end of the period. A burn rate of 2 means you'll exhaust in half the period.
# Burn rate over 1 hour (2x budget consumption)
# Alert: you'll exhaust budget in 15 days instead of 30
- alert: HighBurnRate1h
expr: |
(
1 - (
sum(rate(http_requests_total{code!~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
)
) / (1 - 0.999) > 2
for: 2m
labels:
severity: warning
# Burn rate over 5 minutes (10x budget consumption)
# Alert: you'll exhaust budget in 3 days
- alert: HighBurnRate5m
expr: |
(
1 - (
sum(rate(http_requests_total{code!~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
)
) / (1 - 0.999) > 10
for: 1m
labels:
severity: criticalBurn rate alerts use multiple time windows to catch both sustained degradation (1h window) and sudden spikes (5m window). The burn rate is calculated as: (error rate) / (1 - SLO target). A burn rate > 1 means you're consuming budget faster than allowed.
Error Budget
Error budget is the inverse of SLO. For a 99.9% SLO, you have 0.1% budget (43 minutes per month). When the budget is exhausted, stop feature releases and focus on reliability. This aligns engineering velocity with reliability goals.
SLO: 99.9% availability
Period: 30 days (2,592,000 seconds)
Budget: 0.1% = 2,592 seconds = 43.2 minutes
Current errors: 1,800 seconds
Remaining: 792 seconds = 13.2 minutes
Burn rate: 1.5x (exhaust in 20 days instead of 30)
Action: Halt feature releases, fix reliability issuesError budget is a concrete, measurable metric for reliability. Teams that stay within budget can ship features fast. Teams that exceed budget must invest in reliability. This prevents the reliability vs velocity conflict.
Install the Grafana SLO plugin for built-in SLO tracking. It automatically calculates burn rates, error budgets, and compliance. Create SLOs per service and view them in a unified dashboard.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.