Drift Detection

Detecting and correcting manual kubectl changes with sync status, diff views, and prune policies.

7 min readCI/CD & GitOpsPlatform

What Is Drift?

Drift occurs when the actual cluster state diverges from the desired state in Git. This happens when someone runs kubectl edit, kubectl scale, or applies manifests directly. Drift makes the cluster state unpredictable and unreproducible.

Drift detection is the process of identifying these divergences. Drift correction is the process of reconciling the cluster back to the desired state. GitOps controllers handle both automatically.

ArgoCD Drift Detection

ArgoCD self-heal configuration


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-repo.git
    targetRevision: main
    path: applications/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: autoscaling
      kind: HorizontalPodAutoscaler
      jsonPointers:
        - /spec/minReplicas
        - /spec/maxReplicas

selfHeal: true tells ArgoCD to automatically correct drift. When someone scales a deployment manually, ArgoCD reverts it to the desired state. ignoreDifferences allows specific fields to vary (like HPA-managed replicas).

Flux Drift Detection

Flux drift detection and correction


apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  path: ./overlays/production
  sourceRef:
    kind: GitRepository
    name: myapp
  prune: true
  force: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: production
  postBuild:
    substitute:
      CLUSTER_NAME: production
    substituteFrom:
      - kind: ConfigMap
        name: cluster-config

Flux reconciles every interval (5m). force: true forces apply even if there are conflicts. The health check ensures the deployment is healthy after reconciliation. postBuild substitutes variables in the manifests.

Diff Tools

ArgoCD diff configuration


# argocd-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # Enable server-side diff
  server.diff.mode: "server-side"

  # Custom diff rules
  resource.customizations.diff.apps_Deployment: |
    hs = {}
    hs.enabled = true
    hs.normalisedYaml = ""
    return hs

  # Exclude fields from diff
  resource.exclusions: |
    - apiGroups:
        - ""
      kinds:
        - ConfigMap
      selectors:
        - matchExpressions:
            - key: app.kubernetes.io/managed-by
              operator: In
              values:
                - Helm

ArgoCD diff shows what will change when syncing. Server-side diff handles fields managed by controllers. Resource exclusions prevent diffs on resources managed by Helm or other tools.

Prune Policies

Resource pruning configuration


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      # Prune resources removed from Git
      - PrunePropagationPolicy=foreground
      # Prune last, after applying new resources
      - PruneLast=true
      # Only prune resources with this annotation
      - Prune=true

---
# Annotate resources for pruning
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
  annotations:
    argocd.argoproj.io/prune: "true"

PrunePropagationPolicy=foreground deletes dependents before owners. PruneLast ensures new resources are applied before old ones are removed. Only resources annotated with argocd.argoproj.io/prune=true are pruned.

Drift Prevention

RBAC — restrict who can run kubectl commands in production. Use namespaces and RoleBindings to limit access.
Admission controllers — use OPA Gatekeeper or Kyverno to block manual changes. Only allow changes through Git.
Self-heal — enable auto-correction in GitOps controllers. Manual changes are reverted within minutes.
Audit logging — track all kubectl commands in the cluster. Alert on production namespace changes.
Developer education — explain why drift is dangerous and how the GitOps workflow works.

Emergency changes still need Git

Even during incidents, changes should go through Git. Create a hotfix branch, apply the change, and merge. The extra 5 minutes of process prevents configuration drift and maintains auditability. Use short-lived branches for emergency changes.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 5 · Platform

GitOps & Progressive Delivery