Stage 7 · Master
Incident Response With AI
Root-Cause Assistance
Correlating deploys, metrics, and changes to suggest likely causes.
The RCA Challenge
Root-cause analysis is the hardest part of incident response. It requires correlating multiple data sources — deploys, config changes, metric anomalies, and external events — to identify the single most likely cause. An AI assistant can process all these sources simultaneously.
Correlation Approach
def analyze_root_cause(incident):
"""Correlate multiple data sources to suggest root cause."""
# Gather context
deploys = get_deploys(
service=incident["service"],
hours_before=incident["started_at"],
hours=48
)
config_changes = get_config_changes(
service=incident["service"],
hours_before=incident["started_at"],
hours=48
)
metric_anomalies = get_metric_anomalies(
service=incident["service"],
time_range=incident["time_range"]
)
similar_incidents = search_similar_incidents(
description=incident["description"],
service=incident["service"],
top_k=3
)
# Build context for LLM
context = format_rca_context(
incident, deploys, config_changes, metric_anomalies, similar_incidents
)
# Ask LLM to analyze
response = openai.ChatCompletion.create(
model="gpt-4-turbo",
temperature=0.0,
messages=[
{"role": "system", "content": RCA_SYSTEM_PROMPT},
{"role": "user", "content": context}
]
)
return json.loads(response["choices"][0]["message"]["content"])The pipeline gathers deploys, config changes, metric anomalies, and similar incidents, then asks the LLM to correlate them.
Deploy Correlation
def check_deploy_correlation(incident, deploys):
"""Check if the incident started near a deploy."""
correlations = []
incident_time = incident["started_at"]
for deploy in deploys:
deploy_time = deploy["deployed_at"]
delta_minutes = (incident_time - deploy_time).total_seconds() / 60
if 0 < delta_minutes < 60: # Incident started within 60 minutes of deploy
correlations.append({
"deploy": deploy,
"delta_minutes": delta_minutes,
"confidence": max(0, 1 - delta_minutes / 60),
})
# Sort by time proximity
correlations.sort(key=lambda x: x["delta_minutes"])
return correlationsTemporal correlation between deploys and incident start is the strongest signal for deploy-caused incidents.
Metric Correlation
def find_metric_correlations(anomalies, incident_window):
"""Find metrics that changed around the same time."""
correlations = {}
for anomaly in anomalies:
metric = anomaly["metric"]
change_time = anomaly["detected_at"]
# Check if this metric changed before others
for other in anomalies:
if other["metric"] == metric:
continue
delta = abs((change_time - other["detected_at"]).total_seconds())
if delta < 300: # Within 5 minutes
key = tuple(sorted([metric, other["metric"]]))
if key not in correlations:
correlations[key] = []
correlations[key].append(delta)
# Rank by how often they co-occur
ranked = sorted(correlations.items(), key=lambda x: len(x[1]), reverse=True)
return [{"metrics": k, "co_occurrences": len(v)} for k, v in ranked]Metrics that change simultaneously share a cause. CPU spike + latency spike = likely the same root cause.
Suggesting Likely Causes
RCA_SYSTEM_PROMPT = """You are a root-cause analysis assistant.
Given incident context, deploys, config changes, metric anomalies, and similar past incidents, suggest the most likely root cause.
RULES:
- Rank hypotheses by likelihood
- For each hypothesis, cite the evidence that supports it
- Note what evidence would confirm or rule out each hypothesis
- Distinguish between the trigger and the underlying cause
- If you cannot determine root cause, say what additional data is needed
Output format: JSON with:
- hypotheses: array of {cause, confidence, evidence, confirming_tests}
- timeline: key events that led to the incident
- recommendation: what to investigate next"""The prompt explicitly asks for multiple hypotheses ranked by likelihood, with evidence for each. This prevents the model from committing to a single wrong answer.
The AI suggests what to investigate. The engineer investigates and confirms. Never skip the verification step based on AI confidence alone.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.