Stage 7 · Master
Phase 20 — Production Readiness
SLI/SLO
Turn a subjective sense of 'payments feel slow lately' into a measured error budget that automatically freezes risky rollouts before it turns into an incident.
Every prior lesson in this module added a control — role isolation, off-cluster backups, a measured restore drill, versioned APIs, tamper-evident audit logs, staged flag rollouts. None of them says whether the platform is currently healthy enough to keep taking risk, or whether this week is the wrong week to advance charges_v2_response from 5% to 50%. A service-level objective, backed by a measured service-level indicator and an error budget, answers that question with a number instead of a feeling — and this lesson wires that number directly into the feature-flag rollout gate from the previous lesson.
Choose an SLI That Matches What a Tenant Actually Experiences
services/payment-service/internal/payment processes card charges through pkg/payments and a third-party processor; the metric that matters to a resident paying a maintenance fee is not 'is the payment service's process alive' but 'did my charge attempt complete successfully within a time I'd consider normal.' The SLI has to be built from exactly that request path, not from an infrastructure signal like CPU or pod restarts that can stay green while every payment attempt times out.
package telemetry
import "github.com/prometheus/client_golang/prometheus"
// ChargeAttemptDuration is the SLI's raw signal: every charge attempt,
// successful or not, recorded with its outcome as a label. A "good"
// event is success=true AND duration under the SLO's latency
// threshold — both conditions matter, because a slow success is not
// actually a good outcome from the resident's point of view.
var ChargeAttemptDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: "hoa",
Subsystem: "payment",
Name: "charge_attempt_duration_seconds",
Help: "Duration of a charge attempt from request receipt to processor response.",
Buckets: []float64{0.1, 0.25, 0.5, 1, 2, 5, 10},
},
[]string{"outcome"}, // "success", "declined", "processor_error", "timeout"
)
func RecordChargeAttempt(outcomeLabel string, durationSeconds float64) {
ChargeAttemptDuration.WithLabelValues(outcomeLabel).Observe(durationSeconds)
}outcome="declined" is deliberately not counted as bad in the SLO below: a card being declined is the processor working correctly and telling the truth about the card, not the payment service failing — collapsing that into the same bucket as processor_error or timeout would make the SLO punish the service for its users' banks.
Define the SLO as a Ratio, Then Compute the Error Budget It Buys
Deriving an alert threshold from an objective rather than guessing a round number is not new — Organization Service did exactly that when it turned its request metrics into its first two alert rules, and Observability applied the same method to asynchronous event delivery. What neither of them needed was a *budget*: a quantity that depletes, that several teams spend from at once, and that something other than a human can read. 99.5% of non-declined charge attempts completing in under 2 seconds over 30 days leaves roughly 3 hours and 36 minutes of bad events, and every incident, slow rollout, and infrastructure blip draws from that same balance.
groups:
- name: payment-charge-slo
rules:
- record: hoa:payment_charge_good_events:ratio_rate5m
expr: |
sum(rate(hoa_payment_charge_attempt_duration_seconds_bucket{le="2",outcome!="declined"}[5m]))
/
sum(rate(hoa_payment_charge_attempt_duration_seconds_count{outcome!="declined"}[5m]))
- record: hoa:payment_charge_good_events:ratio_rate1h
expr: |
sum(rate(hoa_payment_charge_attempt_duration_seconds_bucket{le="2",outcome!="declined"}[1h]))
/
sum(rate(hoa_payment_charge_attempt_duration_seconds_count{outcome!="declined"}[1h]))
- record: hoa:payment_charge_good_events:ratio_rate6h
expr: |
sum(rate(hoa_payment_charge_attempt_duration_seconds_bucket{le="2",outcome!="declined"}[6h]))
/
sum(rate(hoa_payment_charge_attempt_duration_seconds_count{outcome!="declined"}[6h]))
- record: hoa:payment_charge_good_events:ratio_rate3d
expr: |
sum(rate(hoa_payment_charge_attempt_duration_seconds_bucket{le="2",outcome!="declined"}[3d]))
/
sum(rate(hoa_payment_charge_attempt_duration_seconds_count{outcome!="declined"}[3d]))
- name: payment-charge-slo-alerts
rules:
# Fast burn: consuming the 30-day budget in ~1 day if sustained.
- alert: PaymentChargeSLOFastBurn
expr: |
(1 - hoa:payment_charge_good_events:ratio_rate5m) > (14.4 * 0.005)
and
(1 - hoa:payment_charge_good_events:ratio_rate1h) > (14.4 * 0.005)
for: 2m
labels:
severity: page
annotations:
summary: "payment charge SLO burning 14.4x too fast — pages on-call within minutes"
# Slow burn: consuming the budget in ~6 days if sustained — real,
# but not an emergency; it becomes tomorrow's rollout-freeze
# decision, not tonight's page.
- alert: PaymentChargeSLOSlowBurn
expr: |
(1 - hoa:payment_charge_good_events:ratio_rate6h) > (6 * 0.005)
and
(1 - hoa:payment_charge_good_events:ratio_rate3d) > (6 * 0.005)
for: 15m
labels:
severity: ticket
annotations:
summary: "payment charge SLO burning 6x too fast — investigate before next rollout stage"Requiring both the short window (5m/6h) and the matching long window (1h/3d) to breach together, before firing, is what keeps a five-minute traffic spike from paging on-call for a burn rate that self-corrects before the long window ever confirms it — the short window alone reacts fast but is noisy; the long window alone is accurate but reacts too slowly to a genuinely fast regression.
Freeze Risky Rollouts Automatically When the Budget Is Spent
The Feature Flags lesson staged charges_v2_response through four percentage steps, each held for at least one SLO observation window. The missing piece is enforcing that gate automatically: an operator advancing a rollout stage by hand, during a week the error budget is already exhausted by an unrelated incident, is exactly the mistake a burn-rate-aware gate exists to prevent.
package admin
import (
"context"
"fmt"
"github.com/prometheus/client_golang/api"
promv1 "github.com/prometheus/client_golang/api/prometheus/v1"
)
// ErrBudgetExhausted is returned when the platform-wide payment SLO's
// error budget is currently below the floor this gate enforces —
// distinct from any per-flag concern, because a rollout of an
// unrelated feature should not proceed while the platform is already
// unhealthy for a different reason.
var ErrBudgetExhausted = fmt.Errorf("error budget exhausted: rollout stage advance blocked")
type RolloutGate struct {
promAPI promv1.API
minRemainingRatio float64 // e.g. 0.10: refuse to spend the last 10% of budget on a rollout
}
func NewRolloutGate(client api.Client, minRemainingRatio float64) *RolloutGate {
return &RolloutGate{promAPI: promv1.NewAPI(client), minRemainingRatio: minRemainingRatio}
}
// AllowAdvance queries the current 30-day burn ratio and refuses to
// let a rollout stage advance if the remaining error budget has
// already fallen below the configured floor — independent of whether
// any alert has fired, because a slow, steady burn can exhaust a
// budget without ever crossing either burn-rate alert's threshold.
func (g *RolloutGate) AllowAdvance(ctx context.Context) error {
result, warnings, err := g.promAPI.Query(ctx, "hoa:payment_charge_good_events:ratio_rate3d", timeNow())
if err != nil {
return fmt.Errorf("query error budget: %w", err)
}
if len(warnings) > 0 {
return fmt.Errorf("prometheus query returned warnings: %v", warnings)
}
goodRatio, err := scalarFrom(result)
if err != nil {
return fmt.Errorf("parse error budget query result: %w", err)
}
const sloTarget = 0.995
budgetConsumed := (1 - goodRatio) / (1 - sloTarget)
remaining := 1 - budgetConsumed
if remaining < g.minRemainingRatio {
return ErrBudgetExhausted
}
return nil
}minRemainingRatio is a deliberate floor above zero, not a check for 'budget fully exhausted': refusing to spend the last slice of budget on a discretionary rollout keeps a reserve available for the incidents that will happen regardless of any flag decision, which is exactly the reserve a floor of exactly 0% would spend down to nothing.
AllowAdvance is called at the top of SetPercentage before the transaction that updates flag_state — if it returns ErrBudgetExhausted, the handler returns 409 Conflict with the current budget-remaining ratio in the response body, and the audit entry the Feature Flags lesson writes never happens, because the flip itself never happens. The gate does not need its own audit trail; it works by preventing the audited action, not by being a separate audited action itself.
Applied exercise
Decide whether to advance a rollout stage given real burn-rate data
charges_v2_response is at 5% for four days. hoa:payment_charge_good_events:ratio_rate3d currently reads 0.9931 (below the 0.995 target). PaymentChargeSLOSlowBurn has been firing intermittently for the last two days but PaymentChargeSLOFastBurn has never fired.
- Compute the fraction of the 30-day error budget already consumed, showing the formula from AllowAdvance applied to 0.9931.
- State whether AllowAdvance should permit an advance to 50% if minRemainingRatio is configured at 0.10, showing your arithmetic.
- Explain why the slow-burn alert firing intermittently for two days, with no fast-burn alert ever firing, is consistent with the budget math you computed rather than contradicting it.
- Recommend a concrete action (advance, hold, or roll back to 0%) and justify it using both the gate's math and the actual user-facing impact this SLI represents.
Deliverable
A worked budget-consumption calculation, an explicit allow/deny determination against the 0.10 floor, a short explanation reconciling the alert history with the math, and a justified recommendation.
Completion checks
- The budget-consumed calculation correctly applies (1 - goodRatio) / (1 - sloTarget) to 0.9931 and 0.995.
- The allow/deny determination is arithmetically consistent with the computed remaining budget and the 0.10 floor.
- The recommendation references the SLI's real meaning (charge attempts failing or running slow for residents) rather than treating the numbers as abstract.
Why does ChargeAttemptDuration exclude outcome="declined" from the SLO's good-event ratio instead of counting every non-successful attempt as bad?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.