Stage 5 · Platform
Deployment Strategies
Automated Rollbacks
Triggering rollback from failed health checks, Prometheus alerts, synthetic tests, and error-budget burn.
Why Automate Rollbacks?
Manual rollbacks are slow. An engineer must notice the issue, diagnose it, decide to rollback, and execute the rollback. This can take 15-30 minutes. Automated rollbacks detect issues and revert in seconds, minimizing the impact on users.
Automated rollbacks require confidence in your health checks, metrics, and monitoring. If your health checks are unreliable, automated rollbacks will cause false-positive rollbacks. Invest in accurate monitoring before automating rollbacks.
Health Check Rollbacks
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
progressDeadlineSeconds: 300
template:
spec:
containers:
- name: myapp
image: myapp:v2.0.0
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3Kubernetes automatically rolls back if the deployment does not progress within progressDeadlineSeconds. Readiness probes prevent traffic from reaching unhealthy pods. Liveness probes restart pods that become unresponsive.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: myapp
spec:
replicas: 3
strategy:
canary:
steps:
- setWeight: 20
- pause: { duration: 5m }
- setWeight: 40
- pause: { duration: 5m }
- setWeight: 60
- pause: { duration: 5m }
- setWeight: 80
- pause: { duration: 5m }
analysis:
templates:
- templateName: success-rate
args:
- name: service-name
value: myapp
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:v2.0.0Argo Rollouts automatically rolls back if the AnalysisTemplate fails. The canary progresses through traffic weight steps with pauses. If any step fails analysis, the rollout is aborted and traffic shifts back to the stable version.
Prometheus Alert Rollbacks
groups:
- name: deployment-rollbacks
rules:
- alert: HighErrorRateAfterDeploy
expr: |
(
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m])
) > 0.05
and
changes(kube_deployment_status_replicas_available[5m]) > 0
for: 2m
labels:
severity: critical
annotations:
summary: "High error rate after deployment"
description: "Error rate is {{ $value | humanizePercentage }} after recent deployment"
- alert: HighLatencyAfterDeploy
expr: |
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
and
changes(kube_deployment_status_replicas_available[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "High latency after deployment"These alerts fire when error rate exceeds 5% or p95 latency exceeds 1 second after a deployment. The changes() function detects recent deployments. Connect these alerts to a webhook that triggers rollback.
Synthetic Monitoring
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy
run: kubectl set image deployment/myapp myapp=ghcr.io/myorg/myapp:${{ github.sha }}
- name: Wait for rollout
run: kubectl rollout status deployment/myapp --timeout=300s
- name: Run synthetic tests
run: |
for i in {1..10}; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://app.example.com/health)
if [ "$STATUS" != "200" ]; then
echo "Health check failed with status $STATUS"
kubectl rollout undo deployment/myapp
exit 1
fi
sleep 5
done
- name: Run end-to-end tests
run: npx playwright test --project=smoke
- name: Rollback on failure
if: failure()
run: kubectl rollout undo deployment/myappSynthetic tests verify the deployed version works correctly from the user's perspective. The health check loop verifies basic availability. Playwright tests verify critical user flows. If any test fails, the deployment is rolled back.
Error Budget Burn
Error budget burn rate measures how fast you are consuming your error budget. If a deployment accelerates error budget consumption, it should be rolled back before the budget is exhausted.
groups:
- name: error-budget
rules:
- alert: ErrorBudgetBurnRateHigh
expr: |
(
rate(http_requests_total{status=~"5.."}[1h])
/ rate(http_requests_total[1h])
) > (14.4 * 0.001)
and
(
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m])
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Error budget burning too fast"
description: "At current rate, error budget will be exhausted in less than 2 hours"The burn rate calculation uses two time windows (1 hour and 5 minutes) to reduce false positives. A burn rate of 14.4x means the 30-day error budget will be exhausted in 2 days. This triggers an immediate rollback.
Rollback Patterns
- Immutable rollback — revert to the previous container image tag. Fast and reliable.
- Feature flag rollback — disable the feature flag without redeploying. Instant and targeted.
- Database rollback — revert schema changes using backward-compatible migrations. Complex but sometimes necessary.
- Traffic rollback — shift traffic back to the previous version (blue-green, canary). Fast but requires previous version running.
An untested rollback is not a rollback — it is a hope. Regularly test your rollback process in staging. Verify that rolling back the application, database, and configuration all work together. Practice makes perfect.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.