The Day a `kubectl drain` Took Down Payments (and the PDB We Didn't Have)

"3 replicas" felt safe. That was the entire justification for why nobody worried about the payments-api deployment during routine node maintenance. Three replicas, spread across three nodes, should mean losing one node never takes the whole service down. It should. What actually happened during a routine kubectl drain as part of a cluster node upgrade was that all three replicas happened to land on the two nodes being drained back to back, and Kubernetes evicted all three within about eight seconds of each other, because nothing was in place to tell it not to.

What `kubectl drain` actually does (and doesn't check by default)

kubectl drain cordons a node (stops new pods from scheduling there) and then evicts every pod currently running on it, so the node can be safely taken down for maintenance. By default, Kubernetes has no built-in knowledge of "this deployment needs at least N healthy replicas at all times" — it will happily evict every replica of a deployment if they all happen to be on the node(s) being drained, one drain operation at a time, with no coordination between separate drain operations unless something explicitly enforces it.

The incident: two drains, three minutes apart, zero PDB

1. Cluster upgrade automation begins draining nodes one at a time to replace them with a newer node image, a standard rolling node-replacement procedure.
2. Node A is drained. It happens to be running 2 of the 3 payments-api replicas. Both are evicted and rescheduled — but only 1 successfully reschedules immediately; the other is briefly pending due to a resource quota edge case. Payments-api is now down to 2 healthy replicas, briefly 1.
3. Three minutes later, before the automation's own healthiness check catches up, Node B begins draining — it's running the 3rd replica. It gets evicted too.
4. For roughly 45 seconds, payments-api has 0 healthy replicas. Every request to it fails.

flowchart TB
    subgraph nodeA["Node A (draining first)"]
        P1[payments-api replica 1]
        P2[payments-api replica 2]
    end
    subgraph nodeB["Node B (draining 3 min later)"]
        P3[payments-api replica 3]
    end
    subgraph nodeC["Node C (untouched)"]
        Other[other workloads]
    end
    SVC[payments-api Service] --> P1
    SVC --> P2
    SVC --> P3
    Drain[cluster upgrade automation] -->|1: drains, no PDB check| nodeA
    Drain -->|2: drains 3 min later, no PDB check| nodeB
    SVC -.->|"0 healthy endpoints for ~45s"| Down[all requests fail]

What a PodDisruptionBudget actually enforces

A PDB doesn't prevent evictions in general — it prevents *voluntary* evictions (drains, descheduling, cluster-autoscaler scale-downs) from violating a minimum-availability constraint you define. With a PDB in place, kubectl drain on Node B would have been *blocked* from evicting the last remaining payments-api replica — the drain command would have failed with a clear error instead of silently succeeding and taking the service down.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api-pdb
  namespace: prod
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api

Why a missing PDB is invisible until exactly this happens

A missing PDB produces zero symptoms during normal operation. There's no warning, no degraded state, nothing in a dashboard that says "this deployment has no disruption protection." It only manifests the one time multiple replicas happen to be co-located on nodes that get drained close together — which might be rare, but cluster upgrades happen regularly, and "rare" events across enough maintenance windows eventually happen.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-pdb-for-multi-replica
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-pdb-exists
      match:
        resources:
          kinds: ["Deployment"]
      preconditions:
        all:
          - key: "{{ request.object.spec.replicas }}"
            operator: GreaterThan
            value: 1
      validate:
        message: "Deployments with >1 replica must have a matching PodDisruptionBudget."
        deny:
          conditions:
            all:
              - key: "{{ length(pdbs) }}"
                operator: Equals
                value: 0

A 10-minute audit worth running on your own cluster right now

kubectl get deployments -A -o json | jq -r '
  .items[] | select(.spec.replicas > 1) | "\(.metadata.namespace)/\(.metadata.name)"
' > all-multi-replica.txt

kubectl get pdb -A -o json | jq -r '
  .items[] | .spec.selector.matchLabels | to_entries[] | "\(.key)=\(.value)"
' > all-pdb-selectors.txt

# Manually cross-reference: any deployment in all-multi-replica.txt whose
# labels don't match anything in all-pdb-selectors.txt has no protection.