Stage 6 · Operate
Incident Response & RCA
Incident Command System
Roles, communication channels, and escalation policies — structured response for chaotic situations.
Why Structure Matters
During an incident, chaos is the default. Without structure, multiple people try to fix the same thing, communication breaks down, and decisions happen in isolation. The Incident Command System provides a clear structure for roles, communication, and decision-making under pressure.
It may seem counterintuitive, but adding structure during an incident makes response faster. When everyone knows their role, there is no confusion about who is doing what. Structure eliminates wasted effort.
Incident Roles
Every incident needs four roles. One person can fill multiple roles in small incidents, but each role must be explicitly assigned. This prevents gaps in the response.
roles:
incident_commander:
responsibility: "Owns the incident, makes decisions, coordinates"
tasks:
- Declare the incident and assign severity
- Assign and coordinate roles
- Make escalation decisions
- Approve remediation actions
- Declare incident resolved
anti_patterns:
- "Debugging instead of commanding"
- "Making unilateral technical decisions"
- "Ignoring communication role"
operations:
responsibility: "Technical investigation and remediation"
tasks:
- Investigate root cause
- Execute remediation steps
- Verify fix effectiveness
- Document technical findings
anti_patterns:
- "Communicating directly to stakeholders"
- "Making severity decisions"
communications:
responsibility: "Internal and external communication"
tasks:
- Post status updates in Slack
- Update status page
- Notify stakeholders
- Maintain incident timeline
anti_patterns:
- "Posting unverified information"
- "Skipping regular updates"
scribe:
responsibility: "Document the incident in real-time"
tasks:
- Record timeline of events
- Log decisions and rationale
- Track action items
- Prepare postmortem draft
anti_patterns:
- "Getting pulled into debugging"
- "Missing key events"Communication Channels
Every incident needs dedicated communication channels. Use a dedicated Slack channel for the war room, a separate channel for updates, and a status page for external communication. Mixing channels causes confusion.
channels:
war_room:
name: "#inc-YYYY-MM-DD-brief-description"
purpose: "Real-time coordination between IC, ops, comms, scribe"
access: "Incident participants only"
rules:
- "Only incident-related messages"
- "Timestamp important decisions"
- "Use threads for side discussions"
updates:
name: "#inc-updates"
purpose: "Status updates for the broader team"
access: "All engineering"
frequency: "Every 15-30 minutes"
format: |
[HH:MM] Status Update
Impact: [description]
Status: [investigating/mitigating/resolved]
Next update: [time]
status_page:
provider: "Statuspage.io / Atlassian"
purpose: "External customer communication"
access: "Public"
frequency: "Every 30 minutes or on status change"Escalation Policies
Escalation policies define when and how to involve additional people. They should be based on severity, duration, and impact. Automate escalations where possible to avoid delays during high-stress situations.
escalation_policies:
- name: "SRE Escalation"
stages:
- targets:
- type: "user"
id: "on-call-sre"
wait: 0
- targets:
- type: "user"
id: "sre-lead"
wait: 15m # Escalate if no response in 15 minutes
- targets:
- type: "user"
id: "sre-manager"
wait: 15m
- targets:
- type: "schedule"
id: "sre-backup"
wait: 10m
- name: "SEV1 Escalation"
stages:
- targets:
- type: "user"
id: "on-call-sre"
- type: "user"
id: "sre-lead"
wait: 0 # Page both immediately
- targets:
- type: "user"
id: "sre-manager"
- type: "user"
id: "vp-engineering"
wait: 15mEscalating an incident is not admitting failure. It is following the process. A SEV1 that is escalated early gets resolved faster than a SEV1 that is escalated late. Err on the side of escalating too early.
Incident Lifecycle
Incidents follow a lifecycle: detection, triage, mitigation, resolution, and postmortem. Each phase has specific actions and responsibilities. Following the lifecycle ensures nothing is missed.
- Detection — Alert fires or user reports issue. Acknowledge within 5 minutes.
- Triage — Assess impact, assign severity, declare incident, assign roles.
- Mitigation — Stop the bleeding. Roll back, scale up, or disable the feature.
- Resolution — Fix the root cause. Deploy the permanent fix.
- Postmortem — Write a blameless postmortem. Identify action items. Schedule follow-up.
ICS Template
# Incident: [Brief Description]
## Metadata
- **Severity:** SEV[1-4]
- **Incident Commander:** [name]
- **Operations:** [name]
- **Communications:** [name]
- **Scribe:** [name]
- **Started:** [YYYY-MM-DD HH:MM UTC]
- **Duration:** [calculated]
## Impact
- **Users affected:** [number or percentage]
- **Services affected:** [list]
- **Revenue impact:** [estimated or "unknown"]
## Timeline
- [HH:MM] Alert fired: [alert name]
- [HH:MM] IC declared incident, assigned roles
- [HH:MM] Root cause identified: [description]
- [HH:MM] Mitigation applied: [action]
- [HH:MM] Impact resolved
- [HH:MM] Incident closed
## Current Status
[Current status and next steps]
## Communication Log
- [HH:MM] Status page updated: [link]
- [HH:MM] Slack update posted: [link]When in doubt, declare an incident. It is better to declare an incident and stand it down than to not declare one and scramble later. A declared incident activates the ICS and gets everyone aligned.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.