Incident Observability

Incident Response Flow

During an incident, time is critical. Your observability stack should provide a clear path from alert to root cause. The flow is: alert fires, dashboards confirm impact, traces locate the failure, logs reveal the error, runbooks guide resolution.

• Alert fires and pages the on-call engineer.
• Engineer opens the service dashboard to confirm impact.
• Golden signals show which dimension is affected.
• Exemplars link to specific traces.
• Trace reveals the failing downstream service.
• Logs from that service show the error message.
• Runbook provides step-by-step resolution.

Alert Triage

Triage determines the severity and scope of an incident. Start with the alerting alert, then check the service dashboard for the four golden signals. Determine if the issue is user-facing, which users are affected, and whether the issue is getting worse.

# Current error rate
sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m]))

# Current latency P99
histogram_quantile(0.99,
  sum by(le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
)

# Affected endpoints
topk(5, sum by(endpoint) (rate(http_requests_total{status=~"5.."}[5m])))

Dashboards for Incidents

Incident dashboards should answer three questions immediately: What is broken? How bad is it? Is it getting worse? The top of the dashboard shows current status. Below that, time series show trends. Side panels show related services.

• Stat panels for current error rate, latency P99, and request rate.
• Time series for the last 1 hour showing error rate trend.
• Downstream dependency health in a separate row.
• Infrastructure metrics (CPU, memory) in a separate row.
• Links to logs, traces, and runbooks in the panel description.

Runbooks

Runbooks are step-by-step guides for resolving known issues. Every alert should link to a runbook. Runbooks should include diagnostic steps, mitigation actions, escalation paths, and rollback procedures.

# Runbook: HighErrorRate

## Impact
- Users experiencing 500 errors on checkout flow
- Error rate above 5% for 5 minutes

## Diagnostic Steps
1. Check service dashboard for error rate trend
2. Identify failing endpoint from topk panel
3. Check trace exemplar for specific failure
4. Review logs for error message

## Mitigation
1. Check if recent deployment caused the issue
2. If yes, rollback to previous version
3. If no, check downstream service health
4. Scale up if saturation is the cause

## Escalation
- If unresolved in 15 minutes, escalate to team lead
- If data loss suspected, page engineering manager

Post-Incident Review

After every incident, conduct a blameless post-incident review. Analyze the timeline, identify what went well, what failed, and what needs improvement. Use the review to improve alerts, dashboards, and runbooks.

• Document the timeline from first signal to resolution.
• Identify which alerts fired and when.
• Note which dashboards were useful and which were missing.
• Record the root cause and contributing factors.
• Create action items for preventing recurrence.
• Update runbooks based on lessons learned.

Observability Checklist