Stage 6 · Operate
SLOs & Error Budgets
Setting SLO Targets
User expectations, business impact, and the cost of nines — defining reliability goals that matter.
What Are SLOs?
A Service Level Objective is a target value for an SLI over a specified time window. If your SLI measures latency, your SLO might state that 99% of requests complete within 200ms over a 30-day rolling window. SLOs turn vague reliability goals into measurable commitments.
SLOs sit between user expectations and engineering capacity. They represent the minimum reliability level you promise users. Anything above the SLO is acceptable. Anything below triggers an error budget breach.
An SLO is not an internal engineering target. It is a promise to users about how reliable your service will be. Treat it as seriously as a contract. Breaking an SLO means breaking a promise.
Choosing Targets
SLO targets are derived from user expectations, not from what is technically easy. A medical billing API serving hospitals needs a different SLO than an internal admin dashboard. Start by asking: what happens when users experience downtime?
- Survey user expectations — what do they consider acceptable downtime?
- Analyze business impact — what is the revenue or reputation cost of missing the target?
- Consider competitive parity — what do similar services promise?
- Factor in engineering cost — how much effort is required to meet each additional nine?
- Start conservative — it is easier to tighten an SLO than to relax one.
The Cost of Nines
Each additional nine of availability is 10x harder to achieve. The jump from 99% to 99.9% is often manageable. The jump from 99.99% to 99.999% may require massive engineering investment in redundancy, failover, and automated recovery.
| Availability | Downtime per Year | Downtime per Month | Typical Use Case |
|---|---|---|---|
| 99% | 3.65 days | 7.31 hours | Internal tools, dev environments |
| 99.9% | 8.76 hours | 43.8 minutes | Standard web applications |
| 99.99% | 52.6 minutes | 4.38 minutes | Critical APIs, payment systems |
| 99.999% | 5.26 minutes | 26.3 seconds | Medical, financial, emergency services |
A 99.99% SLO sounds impressive but may be unnecessary. If your users tolerate 4 minutes of monthly downtime, 99.9% is sufficient. Over-provisioning wastes engineering resources that could improve other aspects of the service.
SLO Types
SLOs can be defined as thresholds, windows, or combinations. The choice depends on your SLI type and the behavior you want to enforce.
# Threshold SLO - latency must stay below a value
slo:
name: api_latency_p99
target: 0.99 # 99% of requests under 200ms
threshold: 200ms
window: 30d
# Availability SLO - success ratio must stay above a value
slo:
name: api_availability
target: 0.999 # 99.9% success rate
window: 30d
# Composite SLO - multiple SLIs combined
slo:
name: api_overall
target: 0.995
components:
- weight: 50
slo: api_latency_p99
- weight: 50
slo: api_availabilityMultiple SLOs per Service
A single service often needs multiple SLOs. A search API might have a latency SLO for query performance, an availability SLO for uptime, and a freshness SLO for index updates. Each SLI gets its own target.
groups:
- name: search_slos
rules:
# Latency SLO: p99 under 150ms
- record: slo:search:latency_target
expr: 0.99
- record: slo:search:latency_good
expr: |
sum(rate(http_request_duration_seconds_bucket{
handler="/api/search",
code!~"5..",
le="0.15"
}[30d]))
- record: slo:search:latency_total
expr: |
sum(rate(http_request_duration_seconds_count{
handler="/api/search",
code!~"5.."
}[30d]))
# Availability SLO: 99.99% non-5xx
- record: slo:search:availability_target
expr: 0.9999
- record: slo:search:availability_good
expr: |
sum(rate(http_request_duration_seconds_count{
handler="/api/search",
code!~"5.."
}[30d]))
- record: slo:search:availability_total
expr: |
sum(rate(http_request_duration_seconds_count{
handler="/api/search"
}[30d]))Documenting SLOs
Every SLO should be documented with its rationale, owner, and error budget policy. Store SLO definitions in version control alongside your service code. Review SLOs quarterly with stakeholders.
## SLO: user-api-latency
**Service:** user-api
**SLI:** p99 latency for /api/users endpoints
**Target:** 99% of requests under 200ms
**Window:** 30-day rolling
**Owner:** platform-team
**Business Context:** User-facing API serving the mobile app.
Degrades mobile app experience when latency exceeds 500ms.
**Error Budget Policy:**
- Budget remaining > 50%: normal deployment velocity
- Budget remaining 25-50%: require additional approval for risky changes
- Budget remaining < 25%: deploy freeze, focus on reliability improvements
- Budget exhausted: mandatory rollback of recent changesSLOs are not static. Review them every quarter with product and engineering stakeholders. As user expectations change or the service evolves, your SLOs should evolve too.
Prometheus SLO Recording
The standard Prometheus pattern for SLOs records the good count, total count, and target separately. This allows Prometheus to calculate the error budget remaining and burn rate without complex queries.
groups:
- name: slo_recording
interval: 30s
rules:
# Compute SLO ratio (good / total)
- record: slo:http_availability:ratio_rate30d
expr: |
sum(rate(http_request_duration_seconds_count{
code!~"5.."
}[30d]))
/
clamp_min(
sum(rate(http_request_duration_seconds_count[30d])),
1
)
# Compute error budget remaining (percentage)
- record: slo:http_availability:budget_remaining
expr: |
(0.999 - slo:http_availability:ratio_rate30d)
/
(1 - 0.999) * -1Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.