Incident Management on Azure

Incident Management Overview

Incident management is the process of detecting, triaging, resolving, and learning from unplanned events. On Azure, incidents can originate from your own resources, Azure platform issues, or third-party dependencies.

1. Detect — Automated monitoring detects the issue
2. Triage — Assess severity and assign ownership
3. Respond — Take immediate action to mitigate impact
4. Resolve — Implement a permanent fix
5. Review — Conduct a post-incident review (PIR)

Azure Service Health

Azure Service Health provides real-time information about Azure service incidents and planned maintenance. It includes Service Health Alerts that notify you when Azure services affecting your resources are impacted.

# Create an action group for Azure Service Health
az monitor action-group create \
  --resource-group rg-monitoring \
  --name ag-azure-health \
  --short-name health \
  --action email sre-team sre@contoso.com \
  --action sms sre-oncall +15551234567

# Create a Service Health alert
az monitor activity-log alert create \
  --resource-group rg-monitoring \
  --name "Azure Service Health Alert" \
  --condition category=ServiceHealth \
  --action-group "/subscriptions/{sub-id}/resourceGroups/rg-monitoring/providers/microsoft.insights/actionGroups/ag-azure-health" \
  --description "Alert on Azure platform incidents"

Maintenance Windows

Azure performs planned maintenance on VM hosts, networking equipment, and platform services. Maintenance windows define when these changes can affect your resources. Configure maintenance windows to control when updates are applied.

# Create a maintenance configuration
az maintenance configuration create \
  --resource-group rg-payments \
  --name maintenance-aks-weekly \
  --location eastus \
  --maintenance-region eastus \
  --scope All \
  --start-time "2025-01-20 02:00" \
  --duration 4 \
  --recur-recur-daily \
  --recur-days 7

# Assign maintenance configuration to a VM
az maintenance assignment create \
  --resource-group rg-payments \
  --resource-name vm-payments-01 \
  --resource-type Microsoft.Compute/virtualMachines \
  --maintenance-configuration-id maintenance-aks-weekly

Incident Response Process

Communication During Incidents

• Internal — Slack/Teams incident channel, status page, bridge call
• External — Status page updates, customer emails, social media
• Leadership — Regular updates to stakeholders at defined intervals
• Engineering — War room coordination, task tracking

# Post to Teams when an alert fires (via webhook)
curl -X POST "$TEAMS_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d '{
    "@type": "MessageCard",
    "summary": "SEV2 Incident: AKS payments API degraded",
    "sections": [{
      "activityTitle": "🔴 SEV2 Incident Opened",
      "facts": [
        {"name": "Service", "value": "Payments API"},
        {"name": "Impact", "value": "API latency >2s"},
        {"name": "Start Time", "value": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"},
        {"name": "Incident Channel", "value": "#inc-payments-api"}
      ]
    }]
  }'

Post-Incident Review

A Post-Incident Review (PIR) or blameless retrospective documents what happened, why, and what to change. It focuses on systemic improvements, not individual blame.

1. Timeline — What happened and when (UTC timestamps)
2. Impact — Who was affected and for how long
3. Root cause — The underlying systemic cause
4. Contributing factors — What made the incident worse
5. Resolution — What was done to fix it
6. Action items — Specific, assigned, time-bound improvements
7. Detection — How the issue was detected (or should have been)