Stage 5 · Platform
Workloads & Scheduling
Scheduling Deep Dive
Predicates, priorities, node affinity, taints/tolerations, and topology spread.
Scheduling Pipeline
The kube-scheduler places pods on nodes. It runs a multi-step pipeline: filtering eliminates nodes that cannot run the pod, scoring ranks remaining nodes, and binding assigns the pod to the highest-scoring node. Understanding each step helps you control where workloads run.
- Filtering — Removes nodes with insufficient resources, unmet node selectors, taints without tolerations, port conflicts, and volume constraints
- Scoring — Ranks remaining nodes by resource balance, affinity preferences, topology spread, and inter-pod affinity
- Binding — Assigns the pod to the selected node by writing the nodeName field
Node Affinity
Node affinity tells the scheduler which nodes are preferred or required for a pod. RequiredDuringSchedulingIgnoredDuringExecution is a hard constraint — the pod won't schedule if no node matches. PreferredDuringSchedulingIgnoredDuringExecution is a soft preference — the scheduler tries but doesn't guarantee it.
apiVersion: v1
kind: Pod
metadata:
name: gpu-workload
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: accelerator
operator: In
values: ["nvidia-gpu", "amd-gpu"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a"]
- weight: 20
preference:
matchExpressions:
- key: node.kubernetes.io/instance-type
operator: In
values: ["g5.xlarge"]The pod requires a node with an accelerator label. It prefers us-east-1a zone (weight 80) and g5.xlarge instances (weight 20). The IgnoredDuringExecution suffix means affinity is checked only during scheduling, not during runtime.
Pod Affinity and Anti-Affinity
Pod affinity places pods near other pods that match a label selector. Pod anti-affinity keeps pods away from other pods. These are used for co-location (frontend next to cache) or isolation (replicas across failure domains).
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: ["web"]
topologyKey: topology.kubernetes.io/zone
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values: ["web"]
topologyKey: kubernetes.io/hostnamerequiredDuringScheduling ensures no two web pods are in the same zone. preferredDuringScheduling tries to place them on different nodes within a zone. topologyKey defines the scope — zone, region, or hostname.
If you have 3 zones but request 5 replicas with required zone anti-affinity, 2 pods will remain Pending. Use preferredDuringScheduling for soft constraints, or ensure you have enough topology domains.
Taints and Tolerations
Taints repel pods from nodes. Only pods with matching tolerations can schedule on tainted nodes. This is used for dedicated nodes (e.g., GPU nodes), control plane isolation, and node maintenance.
# Add a taint to a node
kubectl taint nodes node-1 dedicated=gpu:NoSchedule
# Pod toleration
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
containers:
- name: gpu-app
image: my-gpu-app:1.0
resources:
limits:
nvidia.com/gpu: "1"NoSchedule means new pods won't be scheduled but existing pods are not evicted. NoExecute also evicts existing pods. PreferNoSchedule is a soft version — the scheduler avoids but doesn't guarantee it.
Topology Spread Constraints
Topology spread constraints give you fine-grained control over pod distribution. They replace pod anti-affinity for most use cases. You specify a maxSkew to control how uneven the distribution can be across topology domains.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 6
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: apimaxSkew:1 means at most 1 difference between the most and least populated domains. DoNotSchedule blocks scheduling if the constraint cannot be met. ScheduleAnyway tries to spread but doesn't block.
Use ScheduleAnyway when you want best-effort spreading without blocking scheduling. This is useful for cluster autoscaler scenarios where adding a node would satisfy the constraint — don't block waiting for a node that doesn't exist yet.
Scheduler Extenders
Scheduler extenders let you add custom logic to the scheduling pipeline. You can add filtering, scoring, and binding phases via HTTP webhooks. Modern clusters use the Scheduler Framework (in-process plugins) instead of extenders for better performance.
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
plugins:
score:
disabled:
- name: NodeResourcesFit
enabled:
- name: CustomScorePlugin
weight: 10
pluginConfig:
- name: CustomScorePlugin
args:
scoringParam: "balanced"
extenders:
- urlPrefix: "https://extender.example.com"
filterVerb: "filter"
prioritizeVerb: "prioritize"
bindVerb: "bind"
nodeCacheCapable: trueThe Scheduler Framework lets you enable/disable built-in plugins and add custom ones. Extenders are HTTP webhooks for cases where you need to call external services during scheduling.
Use kubectl describe pod <name> and look at Events for scheduling decisions. For more detail, run kubectl get events --field-selector reason=Scheduled -w. The scheduler logs show filter/score details at --v=4 verbosity.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.