Alert Correlation & Noise Reduction

Grouping a storm of alerts into one probable root cause.

8 min readAI for SRE WorkflowsMaster

The Alert Storm Problem

When a service degrades, dozens of alerts fire simultaneously. A single database slowdown produces alerts for CPU, memory, latency, error rate, and connection pool exhaustion across multiple services. An engineer seeing 50 alerts wastes time determining they are all the same problem.

Alert correlation groups related alerts into a single incident, reducing noise and pointing to the root cause.

Correlation Signals

Use multiple signals to determine which alerts are related.

Temporal proximity — alerts firing within a short time window are likely related.
Service dependency — alerts on services in the same dependency chain are connected.
Metric similarity — alerts with similar metric patterns share a cause.
Topology — alerts on the same host, cluster, or availability zone are correlated.
Alert name — multiple instances of the same alert type across services indicate a shared cause.

Clustering Alerts

Temporal clustering with DBSCAN


from sklearn.cluster import DBSCAN
import numpy as np
from datetime import datetime

def cluster_alerts(alerts, time_window_minutes=5):
    """Cluster alerts that fire within a time window of each other."""
    # Convert timestamps to minutes since first alert
    timestamps = [a["timestamp"].timestamp() / 60 for a in alerts]
    t0 = min(timestamps)
    X = np.array([[t - t0] for t in timestamps])

    # DBSCAN clusters points within time_window_minutes of each other
    clustering = DBSCAN(eps=time_window_minutes, min_samples=1).fit(X)

    clustered_alerts = {}
    for idx, label in enumerate(clustering.labels_):
        if label not in clustered_alerts:
            clustered_alerts[label] = []
        clustered_alerts[label].append(alerts[idx])

    return clustered_alerts

# Example: 10 alerts in 3 clusters
alerts = [
    {"name": "HighCPU", "service": "web", "timestamp": t1},
    {"name": "HighMemory", "service": "web", "timestamp": t2},
    {"name": "HighLatency", "service": "api", "timestamp": t3},
    # ...
]
clusters = cluster_alerts(alerts, time_window_minutes=5)

DBSCAN is ideal for temporal clustering because it does not require specifying the number of clusters and handles noise points.

Grouping into Incidents

Correlation by service dependency


# Service dependency graph
DEPENDENCIES = {
    "checkout": ["payment", "inventory", "cart"],
    "payment": ["billing-db", "stripe-api"],
    "inventory": ["inventory-db"],
}

def correlate_by_dependency(alerts):
    """Group alerts by dependency chain."""
    groups = {}
    for alert in alerts:
        service = alert["service"]
        # Find the top-level service
        root = find_root_service(service, DEPENDENCIES)
        if root not in groups:
            groups[root] = []
        groups[root].append(alert)
    return groups

def find_root_service(service, deps):
    """Walk up the dependency tree to find the root."""
    for parent, children in deps.items():
        if service in children:
            return find_root_service(parent, deps)
    return service

If checkout, payment, and billing-db all alert together, they are grouped under checkout as the root service.

Intelligent Suppression

Suppress derived alerts


class AlertSuppressor:
    def __init__(self):
        self.suppression_rules = {
            "HighCPU": {"derives": ["HighLatency", "HighErrorRate"], "window": 300},
            "DatabaseSlow": {"derives": ["HighLatency", "ConnectionPoolExhausted"], "window": 300},
        }

    def should_suppress(self, alert, recent_alerts):
        """Suppress alerts that are likely derived from a root cause."""
        for rule_name, rule in self.suppression_rules.items():
            # Check if root cause alert fired recently
            root_alerts = [a for a in recent_alerts
                          if a["name"] == rule_name
                          and alert["timestamp"] - a["timestamp"] < rule["window"]]

            if root_alerts and alert["name"] in rule["derives"]:
                return True, f"Suppressed: derived from {rule_name}"
        return False, None

# Usage
suppressor = AlertSuppressor()
for alert in new_alerts:
    suppressed, reason = suppressor.should_suppress(alert, recent_alerts)
    if not suppressed:
        notify_oncall(alert)

When a root cause alert fires, suppress the downstream alerts it causes. This reduces the alert count from 50 to 1.

Correlation is not root cause

Correlated alerts share a cause, but correlation does not identify the root cause. Use dependency analysis and metric patterns to determine which service is the actual root cause.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

AIOps: Detection & Triage