Stage 6 · Operate

Toil Reduction Strategies

Deployment Automation

Progressive delivery, rollback automation, and deployment safety — shipping code without the toil.

7 min readSRE PlaybookOperate

Deployment as Toil

Manual deployments are toil. They are repetitive, error-prone, and scale linearly with the number of services. Every time a human manually applies Kubernetes manifests or SSH into servers to deploy code, that is time that could be spent on engineering.

Deployments should be events, not processes

A deployment should be triggered by a merge to main, validated by automated tests, and rolled out by automation. The human role is to approve the release, not to execute it.

Progressive Delivery

Progressive delivery extends continuous delivery by adding speed controls. Instead of deploying to all users at once, you deploy to a small percentage and gradually increase. This reduces the blast radius of bad deployments.

Argo Rollouts progressive delivery


apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: user-api
  namespace: production
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: user-api-canary
      stableService: user-api-stable
      trafficRouting:
        istio:
          virtualService:
            name: user-api-vsvc
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 2
        args:
          - name: service-name
            value: user-api-canary
      steps:
        # Step 1: Deploy canary to 10% of traffic
        - setWeight: 10

        # Step 2: Run analysis for 5 minutes
        - pause:
            duration: 5m

        # Step 3: If analysis passes, increase to 30%
        - setWeight: 30

        # Step 4: Run analysis for 5 minutes
        - pause:
            duration: 5m

        # Step 5: If analysis passes, increase to 60%
        - setWeight: 60

        # Step 6: Run analysis for 5 minutes
        - pause:
            duration: 5m

        # Step 7: Full rollout
        - setWeight: 100

Rollback Automation

Automated rollback is the safety net of progressive delivery. If the canary analysis fails, the rollout automatically rolls back. No human intervention required. This is faster and more reliable than manual rollback.

Automatic rollback analysis template


apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      successCondition: result[0] >= 0.995
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_request_duration_seconds_count{
              code!~"5..",
              app="{{args.service-name}}"
            }[5m]))
            /
            clamp_min(sum(rate(http_request_duration_seconds_count{
              app="{{args.service-name}}"
            }[5m])), 1)

Canary Deployments

Canary deployments route a small percentage of traffic to the new version. This lets you test the new version with real traffic before committing to a full rollout. The canary should be identical to production except for the version.

Canary deployment configuration


# Canary deployment for user-api
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-api-canary
  namespace: production
  labels:
    app: user-api
    track: canary
spec:
  replicas: 1  # 10% of traffic (1 of 10 total pods)
  selector:
    matchLabels:
      app: user-api
      track: canary
  template:
    metadata:
      labels:
        app: user-api
        track: canary
    spec:
      containers:
        - name: user-api
          image: registry.internal/user-api:v1.3.0-canary
          ports:
            - containerPort: 8080
          env:
            - name: DEPLOYMENT_TRACK
              value: "canary"
            - name: LOG_LEVEL
              value: "debug"  # More verbose for canary analysis

Canary analysis must be fast

If canary analysis takes 30 minutes, your deployment velocity suffers. Aim for 5-10 minute analysis windows. Use high-frequency metrics (30s intervals) and meaningful success criteria (error rate, latency, saturation).

Feature Flags

Feature flags separate deployment from release. You can deploy code to production without enabling it. This lets you test features in production with real traffic and roll back instantly by toggling the flag.

Feature flag configuration


# Feature flag in a ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-flags
  namespace: production
data:
  flags.yaml: |
    flags:
      new-checkout-flow:
        enabled: false
        description: "Redesigned checkout experience"
        owner: "payments-team"
        rollout:
          strategy: "percentage"
          percentage: 0
          schedule:
            - date: "2024-02-01"
              percentage: 10
            - date: "2024-02-08"
              percentage: 50
            - date: "2024-02-15"
              percentage: 100

      new-search-algorithm:
        enabled: true
        description: "ML-based search ranking"
        owner: "search-team"
        kill_switch: true  # Can be instantly disabled

Deployment Safety

Deployment automation needs safety controls. Implement deployment windows, approval gates, blast radius limits, and automatic rollback. These controls prevent automation from causing outages.

Deployment safety configuration


deployment_safety:
  windows:
    allowed:
      - "Mon-Fri 09:00-17:00 UTC"
    blocked:
      - "Fri 15:00-Mon 09:00 UTC"  # No weekend deployments
      - "2024-12-20 to 2025-01-05"  # Holiday freeze

  approval_gates:
    required_for:
      - "database migrations"
      - "infrastructure changes"
      - "security patches"
    approvers:
      - "sre-lead"
      - "on-call-engineer"

  blast_radius:
    max_percentage: 50%
    max_pods: 20
    cooldown_between_deployments: 10m

  rollback:
    auto_rollback: true
    trigger: "error_rate > 5% for 2m"
    max_rollback_time: 5m

Deploy during business hours

Schedule deployments during business hours when the team is available to respond. Avoid Friday afternoon deployments and holiday weekends. If you must deploy outside business hours, ensure on-call is staffed.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.