Stage 5 · Platform
Observability & Autoscaling
HPA, VPA & KEDA
Resource metrics, custom metrics, event-driven scaling, and conflicting scaler behavior.
HPA Deep Dive
HPA computes desired replicas using: desiredReplicas = ceil[currentReplicas * (currentMetric / targetMetric)]. It runs every 15s (configurable). HPA supports resource metrics, custom metrics, and external metrics. Multiple metrics produce separate desired counts — HPA takes the maximum.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 3
maxReplicas: 50
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "1000"
- type: Object
object:
metric:
name: requests-per-pod
selector:
matchLabels:
app: api
describedObject:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: api-ingress
target:
type: Value
value: "100"
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60HPA uses three metric sources: CPU utilization (70%), per-pod request rate (1000 rps), and per-ingress request rate (100 rps). It takes the maximum desired count. scaleDown stabilization prevents thrashing.
VPA Operation
VPA adjusts pod resource requests based on actual usage. It has three components: Recommender (analyzes usage), Updater (evicts pods that need resizing), and Admission Controller (sets requests on new pods). VPA cannot resize running pods — it must evict and recreate.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: api-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: api
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: "app"
mode: "Auto"
minAllowed:
cpu: "100m"
memory: "128Mi"
maxAllowed:
cpu: "8"
memory: "16Gi"
- containerName: "sidecar"
mode: "Off" # Don't adjust sidecar resources
controlledValues: RequestsAndLimitsVPA in Auto mode evicts pods when their actual usage exceeds 80% of requests. It adjusts requests to match actual usage with a safety margin. controlledValues: RequestsAndLimits adjusts both requests and limits.
VPA adjusts CPU requests; HPA scales on CPU utilization. If both target CPU, they fight: VPA increases requests (lowering utilization), HPA adds pods (lowering utilization), VPA increases requests again. Use VPA for memory and HPA for CPU to avoid conflict.
KEDA Deep Dive
KEDA (Kubernetes Event-Driven Autoscaler) extends HPA with 60+ scalers. Each scaler watches a specific external system (Kafka, RabbitMQ, Prometheus, CloudWatch) and provides metrics for scaling. KEDA can scale to zero and back.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-processor
spec:
scaleTargetRef:
name: order-processor
pollingInterval: 15
cooldownPeriod: 300
minReplicaCount: 0
maxReplicaCount: 100
fallback:
failureThreshold: 3
replicas: 6
triggers:
- type: kafka
metadata:
bootstrapServers: kafka:9092
consumerGroup: order-group
topic: orders
lagThreshold: "50"
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: order_queue_depth
threshold: "100"
query: |
sum(order_queue_depth{queue="orders"})
- type: cron
metadata:
timezone: America/New_York
start: 0 8 * * 1-5
end: 0 20 * * 1-5
desiredReplicas: "10"KEDA uses three triggers: Kafka lag, Prometheus queue depth, and cron schedule. fallback.replicas: 6 ensures minimum capacity if triggers fail. cron provides baseline capacity during business hours.
Conflicting Scaler Behavior
Multiple scalers can conflict: HPA scales on CPU while KEDA scales on queue depth. If queue depth drops but CPU is high, KEDA tries to scale down while HPA tries to scale up. Resolve conflicts by separating concerns: KEDA for event-driven workloads, HPA for request-driven.
# Request-driven service: use HPA
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
---
# Event-driven worker: use KEDA
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: worker-keda
spec:
scaleTargetRef:
name: worker
minReplicaCount: 0
maxReplicaCount: 50
triggers:
- type: kafka
metadata:
bootstrapServers: kafka:9092
consumerGroup: worker-group
topic: events
lagThreshold: "100"The API is request-driven — use HPA with CPU. The worker is event-driven — use KEDA with Kafka lag. Don't mix HPA and KEDA on the same workload unless you carefully design non-overlapping metrics.
Scale to Zero
Scale to zero saves costs by running no pods when there's no work. HPA supports minReplicas: 0 since v1.23. KEDA supports it natively. The tradeoff is cold-start latency — the first request waits for the pod to start.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: batch-processor
spec:
scaleTargetRef:
name: batch-processor
minReplicaCount: 0
maxReplicaCount: 20
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: batch_queue_size
threshold: "1"
query: 'sum(batch_queue_size{queue="main"})'
- type: cron
metadata:
timezone: UTC
start: "0 6 * * 1-5"
end: "0 18 * * 1-5"
desiredReplicas: "2"KEDA scales to zero when batch_queue_size drops to 0. During business hours (cron), it maintains 2 replicas for quick response. This combines cost savings with availability during peak hours.
Scaling Best Practices
- Always set resource requests — HPA calculates utilization from requests
- Use multiple metrics for HPA — CPU for cost, custom metrics for business logic
- Configure scaleDown stabilization to prevent thrashing
- Use VPA in Off mode first to get recommendations before enabling Auto
- Separate HPA (request-driven) and KEDA (event-driven) on different workloads
- Set PDBs to prevent scaling from causing downtime
- Test scaling by generating load and observing replica count changes
Use VPA for memory scaling and HPA for CPU scaling. VPA adjusts memory requests to prevent OOMKilled. HPA adjusts replica count based on CPU utilization. This avoids conflicts while optimizing both dimensions.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.