Stage 6 · Operate
SLOs & Error Budgets
Multi-Window Burn Rate
Fast and slow burn detection for paging versus ticketing — catching incidents without false alarms.
Why Multi-Window?
A single time window for burn rate detection forces a tradeoff. Short windows respond quickly but trigger on transient spikes. Long windows are stable but miss acute incidents. The multi-window approach uses two windows simultaneously — a short window for sensitivity and a long window for confirmation.
This is the standard technique from the Google SRE Workbook. It has been proven effective at thousands of companies running Prometheus-based alerting.
The short window catches acute problems fast. The long window confirms the problem is sustained, not a blip. Both must fire for an alert to trigger. This is how you get fast response without false positives.
Fast Burn Detection
Fast burn alerts detect incidents that will exhaust your error budget within days. A burn rate of 14.4x over a 6-hour long window means your 30-day budget will be gone in 2 days. The short window (typically 1 hour) provides the sensitivity to catch the start of the incident quickly.
# Fast burn: burn rate 14.4x
# Short window: 1 hour (sensitivity)
# Long window: 6 hours (confirmation)
# Will exhaust 30-day budget in 2 days
groups:
- name: fast_burn
rules:
- record: fast_burn:short
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[1h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[1h])), 1)
) > (14.4 * 0.001)
labels:
burn_rate: "14.4"
window: "1h"
- record: fast_burn:long
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[6h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[6h])), 1)
) > (14.4 * 0.001)
labels:
burn_rate: "14.4"
window: "6h"Slow Burn Detection
Slow burn alerts detect gradual degradation that will exhaust your budget within weeks. A burn rate of 3x over a 3-day long window means your budget will be gone in 10 days. The short window (typically 6 hours) gives enough sensitivity to catch the trend.
# Slow burn: burn rate 3x
# Short window: 6 hours (sensitivity)
# Long window: 3 days (confirmation)
# Will exhaust 30-day budget in 10 days
groups:
- name: slow_burn
rules:
- record: slow_burn:short
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[6h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[6h])), 1)
) > (3 * 0.001)
labels:
burn_rate: "3"
window: "6h"
- record: slow_burn:long
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[3d]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[3d])), 1)
) > (3 * 0.001)
labels:
burn_rate: "3"
window: "3d"Window Pairing
The pairing of short and long windows follows a specific ratio. For fast burn, the long window is 6x the short window. For slow burn, the long window is 12x the short window. This ratio ensures the long window provides meaningful confirmation without being too slow to respond.
| Alert Type | Short Window | Long Window | Ratio | Burn Rate |
|---|---|---|---|---|
| Fast burn (page) | 1 hour | 6 hours | 6x | 14.4x |
| Fast burn (ticket) | 1 hour | 6 hours | 6x | 6x |
| Slow burn (page) | 6 hours | 3 days | 12x | 3x |
| Slow burn (ticket) | 12 hours | 6 days | 12x | 1x |
Prometheus Alert Rules
Here are the complete alert rules that combine both windows. The alert fires only when both the short and long window conditions are true simultaneously.
groups:
- name: multi_window_burn_rate
interval: 1m
rules:
# Fast burn - page (14.4x, exhausts budget in 2 days)
- alert: ErrorBudgetBurnFastPage
expr: |
(
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[1h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[1h])), 1)
) > (14.4 * 0.001)
)
and
(
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[6h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[6h])), 1)
) > (14.4 * 0.001)
)
for: 2m
labels:
severity: critical
action: page
annotations:
summary: "Error budget burning at 14.4x — will exhaust in 2 days"
# Slow burn - ticket (3x, exhausts budget in 10 days)
- alert: ErrorBudgetBurnSlowTicket
expr: |
(
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[6h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[6h])), 1)
) > (3 * 0.001)
)
and
(
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[3d]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[3d])), 1)
) > (3 * 0.001)
)
for: 5m
labels:
severity: warning
action: ticket
annotations:
summary: "Error budget burning at 3x — will exhaust in 10 days"Replace 0.001 with your actual SLO error rate. For a 99.9% SLO, use 0.001. For 99.99%, use 0.0001. For 99%, use 0.01. The burn rate formula divides the observed error rate by this budget.
Tuning Your Windows
Default windows work for most services, but you may need to adjust based on traffic patterns. Low-traffic services need longer windows to accumulate enough samples. High-traffic services can use shorter windows for faster detection.
- Low traffic (under 100 req/min): extend windows by 2x to avoid statistical noise.
- Normal traffic (100-10k req/min): use the default windows above.
- High traffic (over 10k req/min): can shorten windows by 2x for faster detection.
- Seasonal traffic: use the traffic level during peak hours for window sizing.
- Batch services: use longer windows since request patterns are bursty by nature.
Before deploying burn rate alerts, run the queries against historical incident data. Verify that the alerts would have fired during known incidents and would not have fired during normal operation. This validation prevents both missed incidents and alert fatigue.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.