Stage 5 · Platform
Observability & Autoscaling
Cluster Autoscaler
Node group discovery, scale-down delays, bin packing, and overprovisioning pods.
Cluster Autoscaler Overview
Cluster Autoscaler adjusts the number of nodes based on pending pods. When pods cannot be scheduled due to insufficient resources, CA adds nodes. When nodes are underutilized for 10+ minutes, CA drains and removes them. It works with cloud provider node groups (ASG, MIG, VMSS).
Pending pod detected
-> Check if pod can schedule on existing nodes
-> If yes: schedule normally
-> If no: check if adding a node from any node group would help
-> If yes: scale up the most suitable node group
-> If no: pod remains pending
Node underutilized for 10+ minutes
-> Check if pods can be moved to other nodes
-> If yes: drain node, remove from node group
-> If no: node remains (bin packing prevents removal)CA runs a simulation to determine which node group to scale. It considers resource requests, node selectors, taints, and affinity. The scale-up decision is immediate; scale-down has a 10-minute delay.
Node Group Discovery
CA discovers node groups through cloud provider APIs or auto-discovery tags. Auto-discovery uses tags to identify node groups and their constraints (min/max size, labels, taints). This avoids hardcoding node group names.
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
spec:
template:
spec:
containers:
- name: cluster-autoscaler
image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.0
command:
- ./cluster-autoscaler
- --v=4
- --cloud-provider=azure
- --node-group-auto-discovery=acs:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
- --expander=least-waste
- --balance-similar-node-groups
- --skip-nodes-with-local-storage=false
- --scale-down-delay-after-add=10m
- --scale-down-delay-after-delete=0s
- --scale-down-unneeded-time=10m
- --max-graceful-termination-sec=600
- --max-node-provision-time=15mAuto-discovery tags identify which node groups CA should manage. expander=least-waste picks the node group that results in the least unused resources. balance-similar-node-groups distributes nodes across similar groups.
Scale Down Behavior
Scale-down removes underutilized nodes after a delay. CA checks every 10 seconds (scan-interval) and marks candidates for removal. After the scale-down-delay, it drains pods and removes the node. Pods with PDBs or local storage are not evicted.
# Key scale-down flags
- --scale-down-delay-after-add=10m # Wait after adding a node
- --scale-down-delay-after-delete=0s # Wait after removing a node
- --scale-down-unneeded-time=10m # How long node must be unneeded
- --scale-down-utilization-threshold=0.5 # Node utilization threshold
- --max-graceful-termination-sec=600 # Max time to drain a node
- --skip-nodes-with-local-storage=false # Don't skip nodes with emptyDir
- --skip-nodes-with-system-pods=false # Don't skip system pod nodesscale-down-utilization-threshold: 0.5 means nodes using less than 50% of resources are candidates for removal. max-graceful-termination-sec gives pods time to shut down gracefully. Pods with PDBs prevent node removal.
If a node has pods with PDBs that prevent eviction, CA cannot remove the node. Ensure PDBs allow voluntary disruptions (minAvailable or maxUnavailable). Without this, underutilized nodes accumulate and costs increase.
Bin Packing
Bin packing packs pods onto as few nodes as possible, reducing the number of nodes needed. CA supports this through the most-pods expander, which picks the node group that can schedule the most pending pods. This reduces costs but may increase contention.
apiVersion: apps/v1
kind: Deployment
metadata:
name: bin-packed-app
spec:
replicas: 20
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: bin-packed-app
containers:
- name: app
resources:
requests:
cpu: "200m"
memory: "256Mi"ScheduleAnyway spreads pods across nodes when possible but packs them when necessary. This balances availability (spread) with efficiency (pack). CA adds nodes when packing cannot satisfy all pods.
Overprovisioning Pods
Overprovisioning pods are low-priority pods that reserve capacity. When a high-priority pod is scheduled, it preempts the overprovisioning pod. This ensures capacity is always available for critical workloads without waiting for node provisioning.
apiVersion: apps/v1
kind: Deployment
metadata:
name: overprovisioning
namespace: kube-system
spec:
replicas: 3
template:
metadata:
labels:
app: overprovisioning
spec:
priorityClassName: system-cluster-critical
containers:
- name: pause
image: registry.k8s.io/pause:3.9
resources:
requests:
cpu: "1"
memory: "2Gi"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: overprovisioning
value: -1
globalDefault: false
description: "Low priority class for overprovisioning"The pause image consumes no resources but reserves CPU and memory. When a real pod needs scheduling, it preempts the pause pod. CA then adds a node to replace the preempted capacity. This reduces scheduling latency from minutes to seconds.
Provisioning Request
Provisioning Request (KEP-3570) is a new CRD that tells CA to pre-provision nodes before pods are scheduled. This eliminates the delay between pod creation and node availability. CA immediately provisions nodes matching the ProvisioningRequest.
apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequest
metadata:
name: batch-job-request
spec:
requestType: BestEffortAtomicScaleUp
provisionedClassName: default
podSets:
- name: worker
count: 10
template:
spec:
containers:
- name: worker
resources:
requests:
cpu: "1"
memory: "2Gi"BestEffortAtomicScaleUp provisions all requested nodes atomically. If any node group cannot scale up, the entire request fails. This is useful for batch jobs that need all nodes to start together.
Check CA logs with kubectl logs -n kube-system -l app=cluster-autoscaler. Key metrics: cluster_autoscaler_unschedulable_pods_count (pending pods), cluster_autoscaler_nodes_count (total nodes), and cluster_autoscaler_scaled_up_seconds_total (scale-up events).
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.