Stage 6 · Operate
On-Call Culture & Sustainability
Alert Quality & Tuning
Signal-to-noise, actionable alerts, and eliminating paging on symptoms — alerts that matter.
The Alert Quality Problem
Most SRE teams have too many alerts. Alert fatigue causes engineers to ignore or mute alerts, which means real incidents get missed. The goal is fewer, better alerts — each one demanding action.
If an alert fires and no one needs to do anything, delete it. Alerts that require no action teach engineers to ignore alerts. This is how real incidents get missed.
Signal-to-Noise Ratio
Signal-to-noise ratio measures how many alerts are real incidents (signal) versus false positives (noise). A good ratio is above 80% — 8 out of 10 alerts should be real incidents. Below 50% means your alerting is causing more harm than good.
alert_quality_metrics:
- name: "Signal-to-Noise Ratio"
formula: "actionable_alerts / total_alerts * 100"
target: "> 80%"
frequency: "Weekly"
- name: "Mean Time to Acknowledge (MTTA)"
formula: "avg(alert_fired_at - alert_acknowledged_at)"
target: "< 5 minutes"
frequency: "Weekly"
- name: "Alert Response Rate"
formula: "alerts_with_action / total_alerts * 100"
target: "> 90%"
frequency: "Weekly"
- name: "Mute Rate"
formula: "muted_alerts / total_alerts * 100"
target: "< 10%"
frequency: "Weekly"
- name: "False Positive Rate"
formula: "alerts_resolved_without_action / total_alerts * 100"
target: "< 10%"
frequency: "Weekly"Actionable Alerts
An actionable alert is one where the on-call engineer must take a specific action. If the alert fires and the correct response is 'do nothing,' it is not actionable. If the correct response is 'restart the service,' it is actionable.
actionable_alerts:
- name: "HighErrorRate"
description: "Error rate above 5% for 5 minutes"
response: "Investigate and remediate"
action_required: true
- name: "DiskSpaceLow"
description: "Disk usage above 90%"
response: "Clean up or expand disk"
action_required: true
- name: "CertificateExpiringSoon"
description: "Certificate expires within 7 days"
response: "Renew certificate"
action_required: true
non_actionable_alerts:
- name: "HighCPU"
description: "CPU above 80% for 10 minutes"
response: "Wait and see if it resolves"
action_required: false
fix: "Remove or make threshold higher"
- name: "PodRestarted"
description: "Pod restarted once"
response: "Kubernetes auto-restarted it"
action_required: false
fix: "Only alert if restarts exceed threshold"Symptom vs Cause
Alert on symptoms, not causes. A symptom is what users experience — high latency, errors, unavailability. A cause is what triggered the symptom — disk full, CPU spike, memory leak. Causes can vary, but symptoms are consistent.
| Symptom Alert (Good) | Cause Alert (Avoid) |
|---|---|
| API latency > 500ms for 5 minutes | CPU > 90% for 5 minutes |
| Error rate > 5% for 2 minutes | Pod restarted 3 times |
| Availability < 99.9% for 15 minutes | Disk usage > 95% |
| P95 latency > 1s for 5 minutes | Memory usage > 80% |
When a root cause triggers multiple downstream symptoms, cause alerts create an alert storm. One disk-full event might trigger CPU, memory, latency, and error alerts simultaneously. Symptom alerts remain single alerts.
Alert Tuning Process
Tune alerts weekly. Review every alert that fired in the past week. For each alert, ask: was it actionable? Did it require human intervention? Could it have been automated? Adjust or delete based on the answers.
tuning_process:
frequency: "Weekly"
participants:
- "On-call engineer who handled the alerts"
- "SRE lead"
- "Rotating team member"
review_criteria:
- question: "Did this alert fire in the past week?"
action: "If no, consider removing it"
- question: "Did the on-call engineer take action?"
action: "If no, consider making it a ticket or dashboard"
- question: "Did the alert fire multiple times for the same issue?"
action: "If yes, add grouping or increase duration"
- question: "Was the alert actionable within 5 minutes?"
action: "If no, consider increasing the threshold"
- question: "Did the alert correctly identify the problem?"
action: "If no, refine the query or threshold"
tuning_actions:
- action: "Delete"
when: "Alert is not actionable"
- action: "Modify threshold"
when: "Alert fires too frequently"
- action: "Increase duration"
when: "Alert fires on transient spikes"
- action: "Add grouping"
when: "Alert fires multiple times per incident"
- action: "Convert to ticket"
when: "Alert is important but not urgent"
- action: "Automate response"
when: "Alert has a clear automated fix"Alert Lifecycle
Alerts have a lifecycle. They are created, tuned, and eventually retired. Track this lifecycle to prevent alert proliferation. An alert that has not fired in 90 days should be reviewed for retirement.
alert_lifecycle:
creation:
requirements:
- "Documented runbook linked"
- "Clear severity assigned"
- "Threshold justified with data"
- "Owner assigned"
approval: "SRE lead review"
review:
frequency: "Quarterly"
criteria:
- "Alert has not fired in 90 days"
- "Alert has fired more than 10 times in 30 days"
- "Alert has a false positive rate > 20%"
actions:
- "Retire if not firing"
- "Tune if too frequent"
- "Keep if appropriate"
retirement:
process:
- "Remove from Alertmanager"
- "Remove from Prometheus rules"
- "Remove from dashboards"
- "Document why it was retired"
- "Notify the team"Set an alert budget — a maximum number of alerts per team per week. If you exceed the budget, you must retire or tune existing alerts before adding new ones. This prevents alert proliferation.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.