Stage 6 · Operate
Alerting Hygiene & Paging Design
Actionable Alerts
Page on user-impacting symptoms using Prometheus alerts, runbook links, and ownership labels.
What Makes an Alert Actionable
An actionable alert is one where a human must do something. If the correct response is 'do nothing and wait,' it is not actionable. If the correct response is 'restart the service' or 'scale up the deployment,' it is actionable.
Ask: 'If this alert fires at 3 AM, must a human wake up and do something?' If yes, it is actionable. If no, it belongs on a dashboard or as a low-priority ticket.
Essential Alert Labels
Every alert must carry enough context for the on-call engineer to understand the problem immediately. Labels provide this context: severity, service, team, and runbook link.
essential_labels:
severity:
description: "Page severity (critical, warning, info)"
values: ["critical", "warning"]
required: true
service:
description: "Affected service name"
example: "user-api"
required: true
team:
description: "Owning team for routing"
example: "platform-sre"
required: true
runbook_url:
description: "Link to the runbook for this alert"
example: "https://runbooks.internal/high-error-rate"
required: true
dashboard_url:
description: "Link to relevant Grafana dashboard"
example: "https://grafana.internal/d/user-api"
required: true
slo_target:
description: "Related SLO target"
example: "99.9% availability"
required: false
impact:
description: "User impact description"
example: "All users seeing 503 errors"
required: trueRunbook Links
Every alert must link to a runbook. Without a runbook link, the on-call engineer wastes time figuring out what to do. The runbook link should be in the alert annotations, visible in PagerDuty and Slack.
groups:
- name: api_alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_request_duration_seconds_count{code=~"5.."}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
> 0.05
for: 5m
labels:
severity: critical
service: user-api
team: platform-sre
annotations:
summary: "Error rate above 5% for user-api"
description: |
Current error rate: {{ $value | humanizePercentage }}
This exceeds the 5% threshold.
runbook_url: "https://runbooks.internal/high-error-rate"
dashboard_url: "https://grafana.internal/d/user-api"Prometheus Alert Examples
groups:
- name: actionable_alerts
rules:
# High error rate - page
- alert: HighErrorRate
expr: |
sum(rate(http_request_duration_seconds_count{code=~"5.."}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
> 0.05
for: 5m
labels:
severity: critical
service: user-api
team: platform-sre
annotations:
runbook_url: "https://runbooks.internal/high-error-rate"
# High latency - page
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{code!~"5.."}[5m])) by (le)
) > 2
for: 5m
labels:
severity: critical
service: user-api
team: platform-sre
annotations:
runbook_url: "https://runbooks.internal/high-latency"
# Disk space low - ticket
- alert: DiskSpaceLow
expr: |
(node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.15
for: 30m
labels:
severity: warning
team: infrastructure
annotations:
runbook_url: "https://runbooks.internal/disk-space"
# Certificate expiring - ticket
- alert: CertificateExpiringSoon
expr: |
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
for: 1h
labels:
severity: warning
team: platform-sre
annotations:
runbook_url: "https://runbooks.internal/certificate-renewal"Alert Filtering
Filter alerts by severity, team, and service. This ensures the right people get the right alerts. Do not page the entire team for every alert — route alerts to the team that owns the affected service.
route:
receiver: default
routes:
# Critical alerts - page on-call
- match:
severity: critical
receiver: pagerduty-oncall
continue: true
# Team-specific routing
- match:
team: platform-sre
receiver: slack-platform-sre
continue: true
- match:
team: payments-sre
receiver: slack-payments-sre
continue: true
# Warning alerts - ticket only
- match:
severity: warning
receiver: slack-reliability
group_wait: 5m
group_interval: 30m
receivers:
- name: pagerduty-oncall
pagerduty_configs:
- service_key: <key>
severity: critical
- name: slack-platform-sre
slack_configs:
- channel: "#platform-sre-alerts"
- name: slack-payments-sre
slack_configs:
- channel: "#payments-sre-alerts"
- name: slack-reliability
slack_configs:
- channel: "#reliability"Alert Gardening
Alert gardening is the regular maintenance of your alerting rules. Review alerts weekly, delete unused alerts, tune noisy alerts, and ensure every alert has a runbook. This keeps your alerting healthy.
alert_gardening:
weekly_review:
- "Review all alerts that fired in the past week"
- "Delete alerts that were not actionable"
- "Tune alerts with high false positive rates"
- "Verify all alerts have runbook links"
- "Check alert routing is correct"
monthly_review:
- "Audit all alerting rules"
- "Delete alerts that have not fired in 90 days"
- "Review alert volume trends"
- "Update thresholds based on service changes"
- "Verify escalation policies are current"
quarterly_review:
- "Review entire alerting strategy"
- "Align alerts with current SLOs"
- "Remove deprecated service alerts"
- "Update team ownership labels"
- "Train team on alert quality best practices"Set a maximum number of alerts per team per week. If you exceed the budget, you must retire or tune existing alerts before adding new ones. This prevents alert proliferation.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.