Stage 5 · Platform
SRE Patterns on Azure
Disaster Recovery on Azure
Azure Site Recovery, geo-redundant storage, and cross-region failover runbooks.
Disaster Recovery Overview
Disaster recovery (DR) ensures your systems can recover from a catastrophic failure — region outage, data corruption, or natural disaster. A DR strategy defines RTO (how fast to recover) and RPO (how much data loss is acceptable).
| DR Strategy | RTO | RPO | Complexity | Cost |
|---|---|---|---|---|
| Backup & Restore | Hours | Hours | Low | Low |
| Pilot Light | 30-60 min | <15 min | Medium | Medium |
| Warm Standby | <30 min | <5 min | Medium-High | High |
| Active-Active | Seconds | Near zero | High | Very High |
Azure Site Recovery
Azure Site Recovery (ASR) replicates VM workloads from a primary region to a secondary region. It handles replication, failover, and failback automatically with recovery plans that orchestrate multi-VM failover.
# Create a Recovery Services vault in the secondary region
az backup vault create \
--resource-group rg-dr-secondary \
--name rv-dr-westus2 \
--location westus2
# Enable replication for a VM
az site-recovery protection-container mapping \
--resource-group rg-dr-primary \
--vault-name rv-dr-eastus \
--fabric-name Azure \
--protection-container container1 \
--mapping mapping1ASR replicates VM disks asynchronously. Recovery point objectives depend on the replication frequency — 30 seconds for Premium, 5 minutes for Standard.
# Create a recovery plan with multiple VMs
az site-recovery recovery-plan create \
--resource-group rg-dr-primary \
--vault-name rv-dr-eastus \
--name plan-payments-failover \
--groups '[
{
"groupType": "Shutdown",
"failedOverRecoveryVMs": [],
"startActionOrder": 1
},
{
"groupType": "Failover",
"failedOverRecoveryVMs": [
{"vmId": "/subscriptions/{sub-id}/resourceGroups/rg-payments/providers/Microsoft.Compute/virtualMachines/vm-payments-db"},
{"vmId": "/subscriptions/{sub-id}/resourceGroups/rg-payments/providers/Microsoft.Compute/virtualMachines/vm-payments-api"}
],
"startActionOrder": 2
}
]'Recovery plans group VMs into failover stages. The database must start before the API servers. Add scripts (pre/post) for manual steps like DNS updates.
Azure Site Recovery replicates VMs, not AKS clusters. For AKS DR, use Velero for backup/restore or deploy clusters in both regions with AKS Fleet Manager.
Geo-Redundant Storage
Geo-Redundant Storage (GRS) replicates data to a paired region automatically. If the primary region becomes unavailable, the data is accessible from the secondary. GRS provides 99.99999999999999% (16 nines) durability.
# Create a geo-redundant storage account
az storage account create \
--name stpaymentsgrs \
--resource-group rg-payments \
--sku Standard_GRS \
--kind StorageV2
# Check replication status
az storage account show \
--name stpaymentsgrs \
--query "{sku:sku.name, primaryLocation:primaryLocation, secondaryLocation:secondaryLocation}" \
--output json
# Initiate a failover to the secondary region
az storage account failover \
--name stpaymentsgrs \
--resource-group rg-paymentsGRS failover is manual — you initiate it when you determine the primary region is unrecoverable. After failover, the account becomes LRS in the secondary region.
Failover Runbooks
#!/usr/bin/env bash
set -euo pipefail
# Failover runbook: AKS payments service
PRIMARY_CLUSTER="aks-payments-eastus"
SECONDARY_CLUSTER="aks-payments-westus2"
PRIMARY_RG="rg-aks-primary"
SECONDARY_RG="rg-aks-secondary"
echo "Step 1: Verify secondary cluster health"
az aks show --resource-group "$SECONDARY_RG" --name "$SECONDARY_CLUSTER" \
--query "{status:provisioningState, nodes:agentPoolProfiles[0].count}" --output json
echo "Step 2: Scale up secondary cluster"
az aks nodepool scale \
--resource-group "$SECONDARY_RG" \
--cluster-name "$SECONDARY_CLUSTER" \
--name pool-app \
--node-count 5
echo "Step 3: Update DNS to point to secondary"
az network dns record-set a update \
--resource-group rg-dns \
--zone-name contoso.com \
--name payments \
--set aRecords[0].ipv4Address="payments-westus2.westus2.cloudapp.azure.com"
echo "Step 4: Verify DNS propagation"
dig payments.contoso.com
echo "Failover complete. Verify application health."Runbooks should be tested quarterly. Document manual steps and assign owners. Use Azure Automation for unattended execution.
RTO/RPO Planning
# Example: RTO/RPO matrix
# Service | RTO | RPO | Strategy
# -----------------|----------|----------|------------------
# Payment API | 15 min | 0 | Active-active + Cosmos DB
# User database | 30 min | 5 min | Geo-redundant SQL
# File storage | 1 hour | 1 hour | GRS blob storage
# Dev environment | 4 hours | 24 hours | Backup & restore
# Static website | 4 hours | 24 hours | CDN + backupRTO/RPO targets drive your DR strategy. Active-active achieves the lowest RTO/RPO but costs the most. Match the strategy to business requirements.
DR Testing
DR plans must be tested regularly. An untested DR plan is not a plan — it is a hope. Schedule quarterly drills and document the results.
- Tabletop exercise — Walk through the runbook with the team, no actual failover
- Component test — Failover a single non-critical service to validate the process
- Full DR drill — Execute the complete failover runbook in a test environment
- Chaos engineering — Inject failures (node deletion, network partition) randomly
- Post-incident review — Document lessons learned and update the runbook
Azure Chaos Studio lets you inject controlled failures (VM deletion, network latency, DNS failure) into your environment. Use it to test DR resilience without manual intervention.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.