Stage 6 · Operate
Alerting with Alertmanager
Alerting Rules
Writing Prometheus alert rules with for durations, labels, annotations, severity, and runbook URLs.
Alert Rule Structure
Alerting rules define conditions that trigger alerts. When the PromQL expression evaluates to true for longer than the for duration, Prometheus sends the alert to Alertmanager. Each rule specifies a name, expression, duration, and labels.
groups:
- name: application_alerts
rules:
- alert: HighErrorRate
expr: |
sum by(job) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by(job) (rate(http_requests_total[5m]))
> 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate on {{ $labels.job }}"
description: "Error rate is {{ $value | humanizePercentage }} on {{ $labels.job }}"
runbook_url: "https://runbooks.example.com/high-error-rate"The for Duration
The for field specifies how long the condition must be continuously true before firing. This prevents alerts from triggering on brief transients. A 5-minute for duration means the condition must hold for 300 consecutive seconds.
| for Duration | Use Case | Example |
|---|---|---|
| 0s | Immediate alert | Service is down |
| 2m - 5m | Short transient filter | Error rate spike |
| 10m - 15m | Sustained problem | High latency |
| 30m+ | Long-term degradation | Slow memory leak |
Setting for: 0s causes alerts to fire on every evaluation. This generates excessive pages for brief spikes. Use at least 2m for user-facing alerts to filter transient noise.
Labels and Annotations
Labels identify the alert in Alertmanager and determine routing. Annotations provide descriptive text for humans. Use labels for routing decisions and annotations for context in notifications.
labels:
severity: critical # Used for routing
team: platform # Used for routing
service: api # Used for grouping
annotations:
summary: "High error rate on {{ $labels.service }}"
description: "The error rate has exceeded 5% for 5 minutes."
runbook_url: "https://runbooks.example.com/high-error-rate"
dashboard_url: "https://grafana.example.com/d/api-overview"Severity Levels
Consistent severity labels enable structured paging. Define severity levels that map to response expectations. Every alert must have a severity label that Alertmanager can use for routing.
| Severity | Response Time | Example |
|---|---|---|
| critical | Immediate page | Service down, data loss risk |
| warning | Within 1 hour | Degraded performance, high error rate |
| info | Next business day | Non-urgent anomalies, capacity planning |
Common Alert Patterns
groups:
- name: service_alerts
rules:
- alert: ServiceDown
expr: up{job="my-service"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.job }} is down"
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum by(job, le) (rate(http_request_duration_seconds_bucket{job="my-service"}[5m]))
) > 1.0
for: 5m
labels:
severity: warning
- alert: HighMemoryUsage
expr: |
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.85
for: 10m
labels:
severity: warningEvery alert should link to a runbook. When the pager fires at 3am, the on-call engineer needs immediate context. A runbook_url annotation links to step-by-step instructions for resolving the alert.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.