Stage 6 · Operate
Toil Reduction Strategies
Self-Healing Systems
Auto-remediation, controllers, and the control loop pattern — systems that fix themselves.
The Control Loop Pattern
Self-healing systems follow the control loop pattern: observe the current state, compare it to the desired state, and take action to close the gap. This is the same pattern used in thermostats, cruise control, and Kubernetes itself. The loop runs continuously, detecting and correcting drift.
# The fundamental control loop
loop:
1. OBSERVE: Read current state from metrics/APIs/logs
2. COMPARE: Calculate gap between current and desired state
3. ACT: Execute remediation to close the gap
4. VERIFY: Confirm remediation had the desired effect
5. WAIT: Sleep for observation interval
6. REPEAT
# Example: Auto-restart unhealthy pods
desired_state:
minimum_healthy_pods: 3
health_check: http://localhost:8080/healthz
current_state:
healthy_pods: 2 # One pod is failing health checks
action:
restart_failing_pods: true
scale_up_if_needed: trueIn a self-healing system, the desired state is declared once and the controller continuously works to achieve it. You do not tell the system how to heal — you tell it what healthy looks like. The controller figures out the how.
Kubernetes Controllers
Kubernetes is the most common self-healing platform. Its controllers implement control loops for pods, deployments, nodes, and more. Understanding Kubernetes controllers is understanding self-healing at scale.
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-api
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: user-api
template:
metadata:
labels:
app: user-api
spec:
containers:
- name: user-api
image: registry.internal/user-api:v1.2.3
ports:
- containerPort: 8080
# Liveness: restart if unhealthy
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 15
failureThreshold: 3
# Readiness: remove from load balancer if not ready
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 2
# Startup: give time for initialization
startupProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
failureThreshold: 30
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512MiAuto-Remediation Patterns
Beyond Kubernetes, you can build custom self-healing controllers for application-specific problems. Common patterns include auto-restart, auto-scaling, auto-rotation, and auto-rollback.
apiVersion: apps/v1
kind: Deployment
metadata:
name: cert-healer
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: cert-healer
template:
spec:
serviceAccountName: cert-healer
containers:
- name: cert-healer
image: internal/cert-healer:v1.0.0
env:
- name: CHECK_INTERVAL
value: "3600" # Check every hour
- name: RENEW_DAYS_BEFORE
value: "30" # Renew 30 days before expiry
- name: DOMAINS
value: "api.example.com,web.example.com"
volumeMounts:
- name: kubeconfig
mountPath: /etc/kubernetes
readOnly: true
volumes:
- name: kubeconfig
secret:
secretName: cert-healer-kubeconfigChaos-Driven Healing
Chaos engineering validates that self-healing works. Inject failures and verify that your controllers detect and correct them. This builds confidence that your system will heal during real incidents.
# LitmusChaos experiment to validate auto-restart
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: pod-delete-healing-test
namespace: production
spec:
appinfo:
appns: production
applabel: app=user-api
appkind: deployment
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: "30"
- name: CHAOS_INTERVAL
value: "10"
- name: FORCE
value: "false"
probe:
- name: pod-recovery-check
type: httpProbe
httpProbe/inputs:
url: http://user-api.production.svc:8080/healthz
method:
get:
criteria: ==
responseCode: "200"
mode: Edge
retryProperties:
timeout: 60
retry: 10
delay: 5Self-healing works for known failure modes: crashed pods, disk full, certificate expiry. It cannot heal unknown bugs, data corruption, or cascading failures. Always have a human escalation path for failures the system cannot handle.
Safety Boundaries
Self-healing controllers need safety boundaries to prevent runaway remediation. Limit how many actions they can take, what resources they can modify, and how quickly they can act.
safety_boundaries:
rate_limits:
max_restarts_per_hour: 3
max_scale_events_per_hour: 5
max_rollbacks_per_day: 2
blast_radius:
max_pods_affected: 50%
max_services_affected: 1
requires_approval_above: 10_pods
circuit_breaker:
trigger: "healing_action_failed_3_times"
action: "stop_healing_and_alert"
escalation:
trigger: "healing_failed_after_5_attempts"
action: "page_on_call_human"
channel: "pagerduty-sre"Monitoring Self-Healing
Monitor your self-healing controllers themselves. Track how often they take action, whether those actions succeed, and whether the system actually heals. A controller that heals 99% of the time but fails 1% of the time needs improvement.
groups:
- name: self_healing_metrics
rules:
# Track healing actions taken
- record: healing:actions:total
expr: |
sum(increase(healing_controller_actions_total[1h])) by (action, result)
# Track healing success rate
- record: healing:success:ratio
expr: |
sum(increase(healing_controller_actions_total{result="success"}[1h]))
/
clamp_min(sum(increase(healing_controller_actions_total[1h])), 1)
# Alert if healing is failing
- alert: SelfHealingFailing
expr: healing:success:ratio < 0.8
for: 15m
labels:
severity: warning
annotations:
summary: "Self-healing success rate below 80%"
description: "Current success rate: {{ $value | humanizePercentage }}"Track healing actions as key performance indicators. A decreasing need for healing means your system is improving. An increasing failure rate means your self-healing needs work. These metrics tell you whether your reliability investments are paying off.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.