Stage 6 · Operate
Disaster Recovery & Continuity
Failover Architecture
Active-active, active-passive, DNS failover, traffic managers, and regional dependencies.
Failover Patterns
Failover architecture determines how your system recovers from regional or component failures. The right pattern depends on your RTO, RPO, budget, and complexity tolerance.
| Pattern | RTO | RPO | Cost | Complexity |
|---|---|---|---|---|
| Active-Active | < 1 minute | 0 | High | High |
| Active-Passive (hot) | < 5 minutes | < 1 minute | Medium-High | Medium |
| Active-Passive (warm) | < 30 minutes | < 1 hour | Medium | Medium |
| Active-Passive (cold) | hours | hours | Low | Low |
| Pilot Light | < 15 minutes | < 5 minutes | Medium | Medium |
Active-Active
active_active:
description: "Traffic served from multiple regions simultaneously"
benefits:
- "Zero downtime during regional failure"
- "Better performance (users served from nearest region)"
- "Higher capacity (all regions handle traffic)"
tradeoffs:
- "Data consistency challenges"
- "Complex replication"
- "Higher cost (full infrastructure in each region)"
configuration:
regions:
- name: "us-east-1"
role: "active"
weight: 50
traffic_share: "50%"
- name: "eu-west-1"
role: "active"
weight: 50
traffic_share: "50%"
load_balancing:
method: "latency-based routing"
health_check:
interval: "10s"
timeout: "5s"
healthy_threshold: 2
unhealthy_threshold: 3
data_sync:
method: "multi-primary replication"
conflict_resolution: "last-write-wins"
consistency: "eventual"Active-Passive
active_passive:
description: "One region serves traffic, another is standby"
benefits:
- "Simpler data consistency"
- "Lower cost than active-active"
- "Easier to reason about"
tradeoffs:
- "Failover takes time"
- "Standby infrastructure is idle"
- "Capacity reduced during failover"
configuration:
regions:
- name: "us-east-1"
role: "active"
traffic_share: "100%"
- name: "eu-west-1"
role: "passive"
traffic_share: "0%"
standby_type: "hot" # or "warm" or "cold"
failover_trigger:
method: "health check failure"
threshold: "3 consecutive failures"
action: "DNS update to point to passive"
failback:
method: "manual"
process: |
1. Verify primary region is healthy
2. Sync data from secondary to primary
3. Verify data consistency
4. Update DNS to point to primary
5. Verify traffic flowing to primary
6. Monitor for issuesDNS Failover
dns_failover:
provider: "Route53 / Cloudflare / Google Cloud DNS"
configuration:
primary_record:
name: "api.example.com"
type: "A"
value: "primary-lb.example.com"
health_check: "enabled"
failover: "PRIMARY"
secondary_record:
name: "api.example.com"
type: "A"
value: "secondary-lb.example.com"
health_check: "enabled"
failover: "SECONDARY"
health_check:
protocol: "HTTPS"
port: 443
path: "/healthz"
interval: 30
failure_threshold: 3
regions:
- "us-east-1"
- "eu-west-1"
- "ap-southeast-1"
ttl: 60 # Low TTL for fast failover
considerations:
- "DNS propagation takes time (TTL-dependent)"
- "Some clients cache DNS longer than TTL"
- "Health checks have their own latency"
- "Consider using anycast for faster failover"Traffic Managers
traffic_managers:
global_load_balancer:
provider: "Cloudflare Load Balancer / AWS Global Accelerator"
benefits:
- "Faster failover than DNS alone"
- "Health checks at the load balancer level"
- "Traffic steering based on latency"
- "Automatic failover"
configuration:
pools:
- name: "us-pool"
origins:
- address: "us-east-1.example.com"
weight: 1
health_checks:
interval: 30
timeout: 5
- name: "eu-pool"
origins:
- address: "eu-west-1.example.com"
weight: 1
health_checks:
interval: 30
timeout: 5
steering_policy:
type: "latency"
fallback: "random"
kubernetes_ingress:
provider: "NGINX Ingress / Istio / Linkerd"
benefits:
- "Service-level traffic management"
- "Canary deployments"
- "Circuit breaking"
- "Rate limiting"
configuration: |
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: user-api
spec:
hosts:
- user-api
http:
- route:
- destination:
host: user-api
subset: primary
weight: 90
- destination:
host: user-api
subset: canary
weight: 10Regional Dependencies
regional_dependencies:
services:
user-api:
primary_region: "us-east-1"
dependencies:
- service: "database"
type: "regional"
replication: "synchronous"
failover: "automatic"
- service: "cache"
type: "regional"
replication: "none"
failover: "rebuild from database"
- service: "payment-gateway"
type: "global"
replication: "n/a"
failover: "automatic (provider managed)"
risk_assessment:
single_region_dependencies:
- "If database fails, user-api fails"
- "Mitigation: database replication to second region"
cross_region_dependencies:
- "If payment gateway fails, payments fail globally"
- "Mitigation: multiple payment providers"
global_dependencies:
- "If DNS fails, all services unreachable"
- "Mitigation: anycast DNS, CDN fallback"Failover architecture that has never been tested is a theoretical architecture. Test failover regularly in staging. Verify that health checks work, DNS updates propagate, and traffic shifts correctly.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.