Stage 6 · Operate
Alerting with Alertmanager
Paging Integrations
Mapping severities to PagerDuty services, escalation policies, schedules, and incident priorities.
PagerDuty Integration
PagerDuty is the most common paging tool paired with Alertmanager. The integration uses PagerDuty's Events API to create incidents from Prometheus alerts. Severity maps to incident priority and determines which escalation policy handles the alert.
PagerDuty Configuration
receivers:
- name: "pagerduty-critical"
pagerduty_configs:
- service_key: "your-integration-key"
severity: critical
description: '{{ .CommonAnnotations.summary }}'
details:
firing: '{{ .Alerts.Firing | len }}'
resolved: '{{ .Alerts.Resolved | len }}'
alertname: '{{ .CommonLabels.alertname }}'
job: '{{ .CommonLabels.job }}'
- name: "pagerduty-warning"
pagerduty_configs:
- service_key: "your-warning-service-key"
severity: warningOpsgenie Integration
Opsgenie is an alternative paging platform. The integration uses the Opsgenie API to create alerts from Prometheus. Configure team routing, priority mapping, and tags to direct alerts to the right responders.
receivers:
- name: "opsgenie-critical"
opsgenie_configs:
- api_url: "https://api.opsgenie.com/v2/alerts"
api_key: "your-api-key"
message: '{{ .CommonAnnotations.summary }}'
priority: P1
tags: '{{ .CommonLabels.alertname }},{{ .CommonLabels.job }}'
details:
severity: '{{ .CommonLabels.severity }}'
runbook: '{{ .CommonAnnotations.runbook_url }}'Escalation Policies
Escalation policies define who gets paged and when. A typical policy starts with the primary on-call engineer, then escalates to the secondary, then to the team lead. Map alert severity to different escalation policies in Alertmanager.
route:
receiver: "default"
routes:
- match:
severity: critical
receiver: "pagerduty-critical"
# Escalation: Primary -> Secondary -> Team Lead -> Manager
- match:
severity: warning
receiver: "pagerduty-warning"
# Escalation: Primary -> Secondary
- match:
severity: info
receiver: "slack-info"
# No paging, just notificationOn-Call Schedules
On-call schedules rotate responsibility among team members. PagerDuty and Opsgenie manage schedules internally. Alertmanager does not know about schedules, it only sends to integration endpoints. The paging platform handles rotation and escalation.
If you use PagerDuty for paging but Slack for notifications, keep the on-call schedule in sync. Use PagerDuty's Slack integration to automatically update the on-call channel topic with the current responder.
Incident Priority Mapping
| Alert Severity | PagerDuty Priority | Opsgenie Priority | Response Expectation |
|---|---|---|---|
| critical | P1 | P1 | Immediate response |
| warning | P2 | P2 | Response within 1 hour |
| info | P3 | P3 | Next business day |
Ensure every alert has a severity label and that severity maps to the correct paging policy. Inconsistent severity labels cause alerts to go to the wrong escalation policy, delaying incident response.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.