Rollback Drills

Canary aborts, feature flag disables, database rollback plans, and deployment verification checks.

7 min readSRE PlaybookOperate

Why Rollback Drills?

Rollback is your most common incident response action. If your rollback procedure does not work during an incident, you are in trouble. Rollback drills validate that your rollback procedures actually work under realistic conditions.

Rollback should be your fastest remediation

A rollback should take minutes, not hours. If your rollback procedure requires manual steps, documentation lookup, or multiple approvals, it is too slow for an incident. Practice until rollback is automatic.

Canary Abort Drills

Canary abort drill


canary_abort_drill:
  scenario: "Rollback canary deployment when error rate exceeds threshold"

  setup:
    - "Deploy canary version to 10% of traffic"
    - "Configure canary analysis with error rate threshold"
    - "Inject errors to canary pods"

  steps:
    - "Inject errors to canary (simulating bad deploy)"
    - "Wait for canary analysis to detect errors"
    - "Verify automatic rollback triggers"
    - "Measure time from error injection to rollback completion"

  success_criteria:
    - "Canary analysis detects errors within 2 minutes"
    - "Automatic rollback triggers within 1 minute"
    - "Rollback completes within 5 minutes"
    - "No user impact after rollback"

  abort_criteria:
    - "Rollback fails"
    - "Production traffic affected"
    - "Duration exceeds 30 minutes"

Feature Flag Disable

Feature flag disable drill


feature_flag_drill:
  scenario: "Disable feature flag when feature causes errors"

  setup:
    - "Enable feature flag for new checkout flow"
    - "Inject errors related to new feature"
    - "Configure alert for feature-specific errors"

  steps:
    - "Inject errors to new feature"
    - "Verify alert fires for feature errors"
    - "Disable feature flag via ConfigMap update"
    - "Verify traffic routes to old code path"
    - "Measure time from detection to feature disable"

  success_criteria:
    - "Alert fires within 1 minute"
    - "Feature flag disable takes effect within 2 minutes"
    - "Error rate drops to baseline after disable"
    - "No user impact after disable"

  rollback_method:
    configmap_update: |
      kubectl patch configmap feature-flags -n production \
        --type merge -p '{"data":{"new-checkout-flow":"false"}}'
    verification: |
      kubectl get configmap feature-flags -n production -o yaml

Database Rollback

Database rollback drill


database_rollback_drill:
  scenario: "Rollback database migration when it causes errors"

  setup:
    - "Create test database with known state"
    - "Apply migration that introduces breaking change"
    - "Deploy application that expects new schema"

  steps:
    - "Verify application fails with new migration"
    - "Execute rollback migration"
    - "Verify database state is restored"
    - "Verify application works with rolled-back schema"
    - "Measure total rollback time"

  success_criteria:
    - "Rollback migration executes without errors"
    - "Database state is fully restored"
    - "Application works after rollback"
    - "No data loss"

  rollback_procedure:
    step_1: "Stop application traffic (if needed)"
    step_2: "Execute rollback migration"
    step_3: "Verify schema matches expected state"
    step_4: "Restart application"
    step_5: "Verify application health"
    step_6: "Resume traffic

  safety:
    - "Always backup before rollback"
    - "Test rollback in staging first"
    - "Have DBA on standby for complex rollbacks"

Deployment Verification

Deployment verification checklist


deployment_verification:
  pre_deploy:
    - "All tests passing in CI"
    - "Canary analysis configured"
    - "Rollback command ready"
    - "Monitoring dashboards open"
    - "On-call engineer notified"

  during_deploy:
    - "Canary deployed to 10% traffic"
    - "Canary analysis running"
    - "Error rate monitored"
    - "Latency monitored"

  post_deploy:
    - "Canary analysis passed"
    - "Full rollout complete"
    - "Error rate stable"
    - "Latency stable"
    - "No customer reports"

  verification_queries:
    error_rate: |
      sum(rate(http_request_duration_seconds_count{code=~"5.."}[5m]))
      /
      sum(rate(http_request_duration_seconds_count[5m]))
    latency_p99: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket{code!~"5.."}[5m])) by (le)
      )
    pod_health: |
      sum(kube_pod_status_ready{namespace="production", condition="true"})
      /
      sum(kube_pod_status_status{namespace="production"})

Drill Schedule

Rollback drill schedule


drill_schedule:
  frequency: "Monthly"

  drill_rotation:
    month_1: "Canary abort drill"
    month_2: "Feature flag disable drill"
    month_3: "Database rollback drill"
    month_4: "Full deployment rollback drill"

  participants:
    - "On-call engineer (responder)"
    - "SRE lead (observer)"
    - "DevOps engineer (attacker)"

  documentation:
    - "Record drill results"
    - "Track rollback times"
    - "Identify improvement areas"
    - "Update runbooks if needed"

  metrics:
    - name: "Rollback time"
      target: "< 5 minutes"
      tracking: "Per drill"

    - name: "Rollback success rate"
      target: "100%"
      tracking: "Per drill"

    - name: "Drill completion rate"
      target: "100% (monthly)"
      tracking: "Monthly"

Rollback drills build confidence

When you have practiced rollback many times, you have confidence it will work during a real incident. This confidence reduces stress and improves decision-making during actual emergencies.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

Reliability Testing & Chaos Engineering