Stage 4 · Provision
Designing for High Availability
Failover Strategies
Automatic vs manual, DNS vs load balancer, and stateful failover for production systems.
What Is Failover?
Failover is the process of switching traffic from a failed component to a healthy one. It is the mechanism that makes redundancy actually work. Without failover, redundancy is just idle capacity sitting in a data center.
Automatic vs Manual Failover
Automatic failover detects failure and switches traffic without human intervention. Manual failover requires an operator to trigger the switch. The choice depends on the criticality of the service and the risk of false positives.
| Approach | Speed | Risk | Best For |
|---|---|---|---|
| Automatic | Sub-second to seconds | False positives, split-brain | Stateless services, databases |
| Manual | Minutes to hours | Slow during off-hours | Critical data migrations, stateful systems |
| Semi-automatic | Seconds to minutes | Balanced | Payment systems, compliance-critical |
haproxy:
backend:
name: api_servers
balance: roundrobin
health_check:
interval: 2s
timeout: 1s
fall: 3 # 3 failures = mark down
rise: 2 # 2 successes = mark up
servers:
- name: api-1
address: 10.0.1.10:8080
check: true
- name: api-2
address: 10.0.1.11:8080
check: trueHAProxy checks each server every 2 seconds. After 3 consecutive failures, it marks the server as down and stops sending traffic. After 2 successes, it marks it back up.
Automatic failover without proper fencing can cause split-brain, where two nodes both think they are the primary. This corrupts data. Always implement fencing mechanisms like STONITH or quorum-based decisions.
DNS-Based Failover
DNS failover uses health-check-enabled DNS providers to remove unhealthy endpoints from DNS records. This is slow (TTL-based) but simple and works across regions. It is best used as a last resort for regional failover.
- TTL determines failover speed — set low (60s) during incidents.
- DNS caching means not all clients will see the change immediately.
- Use health checks that validate actual service health, not just ICMP.
- CloudFlare, Route 53, and NS1 support health-checked DNS failover.
Load Balancer Failover
Load balancers detect unhealthy backends faster than DNS and redirect traffic within the same data center or region. This is the most common failover mechanism for intra-region failover with sub-second detection.
Stateful Failover
Stateful services like databases, caches, and message brokers require special failover handling. You cannot just redirect traffic — you must also transfer state. The standby must be synchronized with the primary before it can take over.
1. Primary fails
2. Failover controller detects failure (2-10s)
3. Standby promoted to primary
4. Standby applies any remaining replication log
5. Clients reconnect to new primary
6. Old primary rejoins as standby
7. Full resynchronization beginsStateful failover is slower because it must ensure data consistency before accepting writes. The recovery time depends on replication lag at the time of failure.
Testing Failover
Failover that has never been tested will fail when you need it most. Regular failover testing is a core SRE practice. Use chaos engineering, game days, and regular DR drills to validate your failover mechanisms.
Build failover tests into your CI/CD pipeline. Netflix's Chaos Monkey, Gremlin, and Litmus run automated failover tests in production. If you cannot automate it, schedule it as a recurring game day.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.