Stage 5 · Platform
Observability & Autoscaling
Autoscaling SLO Feedback
Connecting latency SLOs, queue depth, saturation, and cost signals to scaling policy.
SLO-Driven Scaling
SLO-driven scaling ties autoscaling decisions to Service Level Objectives. Instead of scaling on raw metrics (CPU, memory), scale on SLO compliance. If error budget is being consumed, scale up. If budget is healthy, scale down. This aligns scaling with business goals.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-slo-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 3
maxReplicas: 30
metrics:
- type: External
external:
metric:
name: slo:burn_rate:1h
selector:
matchLabels:
service: api
target:
type: Value
value: "1"
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 50
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60The HPA scales based on burn rate. When burn rate > 1 (consuming budget faster than allowed), it scales up. When burn rate < 1, it gradually scales down. This ensures SLO compliance while optimizing costs.
Latency Signal Scaling
Scale based on latency percentiles instead of throughput. If p99 latency exceeds the SLO threshold, add capacity. This directly addresses user experience — slow responses mean poor experience, even if throughput is normal.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: api-latency-scaler
spec:
scaleTargetRef:
name: api
minReplicas: 3
maxReplicas: 50
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
metricName: api_p99_latency
threshold: "500"
query: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le)
) * 1000This KEDA scaler monitors p99 latency in milliseconds. When p99 exceeds 500ms, it scales up. The threshold is the SLO target. Scale down happens when latency drops below the threshold for the cooldown period.
Queue Depth Scaling
Queue depth scaling adds capacity based on pending work. More items in the queue means more capacity needed. This is the most direct signal for event-driven workloads — you scale exactly as fast as work arrives.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: queue-scaler
spec:
scaleTargetRef:
name: worker
minReplicaCount: 1
maxReplicaCount: 100
cooldownPeriod: 60
triggers:
- type: kafka
metadata:
bootstrapServers: kafka:9092
consumerGroup: my-group
topic: events
lagThreshold: "50"
offsetResetPolicy: latest
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: queue_processing_rate
threshold: "100"
query: |
sum(rate(queue_messages_processed_total{queue="events"}[5m]))Kafka lag threshold: when lag > 50 messages, scale up. The Prometheus trigger adds scaling based on processing rate. Together, they ensure the worker keeps up with incoming events.
Saturation Signals
Saturation indicates how close a resource is to capacity. CPU throttling, memory pressure, and connection pool exhaustion are saturation signals. Scale before saturation to prevent performance degradation.
# Prometheus alert rule that exposes throttling metric
- record: container_cpu_throttling_ratio
expr: |
rate(container_cpu_cfs_throttled_periods_total[5m])
/
rate(container_cpu_cfs_periods_total[5m])
# KEDA scaler based on throttling
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: throttle-scaler
spec:
scaleTargetRef:
name: api
minReplicas: 3
maxReplicas: 30
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: cpu_throttling_ratio
threshold: "0.25"
query: |
sum(container_cpu_throttling_ratio{namespace="production", pod=~"api-.*"})
/ count(container_cpu_throttling_ratio{namespace="production", pod=~"api-.*"})When average CPU throttling exceeds 25%, scale up. Throttling means pods are waiting for CPU — adding capacity eliminates the wait. This is more responsive than CPU utilization because it detects contention before utilization spikes.
Cost Signal Scaling
Cost signals factor in the financial impact of scaling decisions. Scale down more aggressively during off-peak hours. Use spot instances for non-critical workloads. Scale to zero for development environments outside business hours.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: cost-aware-scaler
spec:
scaleTargetRef:
name: api
minReplicaCount: 0
maxReplicaCount: 20
triggers:
# Business hours: maintain capacity
- type: cron
metadata:
timezone: America/New_York
start: "0 8 * * 1-5"
end: "0 20 * * 1-5"
desiredReplicas: "5"
# Off-hours: scale to zero
- type: cron
metadata:
timezone: America/New_York
start: "0 20 * * *"
end: "0 8 * * *"
desiredReplicas: "0"
# Weekend: scale to zero
- type: cron
metadata:
timezone: America/New_York
start: "0 18 * * 5"
end: "0 8 * * 1"
desiredReplicas: "0"
# Demand-based scaling during business hours
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: http_requests_per_second
threshold: "500"
query: 'sum(rate(http_requests_total{service="api"}[5m]))'Cron triggers set baseline capacity for business hours. The Prometheus trigger scales beyond baseline based on demand. Off-hours and weekends scale to zero for cost savings. This balances cost and availability.
Closed-Loop Feedback
Closed-loop feedback connects observability to autoscaling. Metrics flow from Prometheus to KEDA/HPA, which adjusts replicas. The new metrics flow back to Prometheus. This creates a continuous feedback loop that maintains SLO compliance.
Application -> Metrics (Prometheus) -> Scaling Decision (HPA/KEDA)
^ |
| v
+---- New Metrics <--- Pod Scale Change <---+The loop: Application emits metrics, Prometheus collects them, HPA/KEDA evaluates thresholds, scaling adjusts replica count, application behavior changes, new metrics are collected. This continuous loop maintains equilibrium.
Use multiple signals: latency for user experience, queue depth for workload, CPU for resource efficiency, and cost for budget. KEDA supports multiple triggers — combine them for comprehensive scaling policy.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.