Stage 6 · Operate

Alerting Hygiene & Paging Design

Alert Routing

Alertmanager routes, PagerDuty services, team ownership maps, and maintenance silences.

7 min readSRE PlaybookOperate

Routing Architecture

Alert routing ensures the right alert reaches the right person at the right time. A well-designed routing architecture routes by severity, team, and service. It prevents alert storms, reduces noise, and ensures fast response.

Route by ownership, not by alert name

Do not route alerts by matching specific alert names. Route by team ownership labels. When a new alert is added, it automatically routes correctly because it carries the team label.

Alertmanager Routes

Complete Alertmanager routing configuration


global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/xxx"

route:
  receiver: default-receiver
  group_by: ["alertname", "service", "team"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical alerts - page immediately
    - match:
        severity: critical
      receiver: pagerduty-critical
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 5m

    # Warning alerts - Slack notification
    - match:
        severity: warning
      receiver: slack-warnings
      group_wait: 5m
      group_interval: 30m
      repeat_interval: 24h

    # Team-specific routing
    - match:
        team: platform-sre
      receiver: slack-platform-sre

    - match:
        team: payments-sre
      receiver: slack-payments-sre

    # Info alerts - dashboard only
    - match:
        severity: info
      receiver: slack-info
      group_wait: 10m
      group_interval: 1h
      repeat_interval: 24h

receivers:
  - name: default-receiver
    slack_configs:
      - channel: "#alerts-default"

  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: "<key>"
        severity: critical
        description: "{{ .GroupLabels.alertname }}"

  - name: slack-warnings
    slack_configs:
      - channel: "#alerts-warnings"
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ .CommonAnnotations.summary }}"

  - name: slack-platform-sre
    slack_configs:
      - channel: "#platform-sre-alerts"

  - name: slack-payments-sre
    slack_configs:
      - channel: "#payments-sre-alerts"

  - name: slack-info
    slack_configs:
      - channel: "#alerts-info"

inhibit_rules:
  # Inhibit warning if critical is firing for same service
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["service"]

  # Inhibit info if warning is firing for same service
  - source_match:
      severity: warning
    target_match:
      severity: info
    equal: ["service"]

PagerDuty Services

Create separate PagerDuty services for each team or service. This ensures alerts route to the correct on-call rotation. Use service metadata to link back to Grafana dashboards and runbooks.

PagerDuty service configuration


pagerduty_services:
  - name: "Platform SRE"
    description: "Platform infrastructure alerts"
    escalation_policy: "SRE Standard"
    auto_resolve_timeout: 14400  # Auto-resolve after 4 hours
    acknowledge_timeout: 600     # Auto-escalate after 10 minutes
    integrations:
      - type: "generic_events_api"
        name: "Alertmanager"
        configuration:
          service_key: "<key>"

  - name: "Payments SRE"
    description: "Payment system alerts"
    escalation_policy: "SRE Critical"
    auto_resolve_timeout: 7200
    acknowledge_timeout: 300
    integrations:
      - type: "generic_events_api"
        name: "Alertmanager"
        configuration:
          service_key: "<key>"

Team Ownership Maps

Every alert must carry a team ownership label. This label determines routing. Keep the ownership map updated as teams change. Unowned alerts are unhandled alerts.

Service ownership map


service_ownership:
  user-api:
    team: "platform-sre"
    pagerduty_service: "Platform SRE"
    slack_channel: "#platform-sre-alerts"

  payment-service:
    team: "payments-sre"
    pagerduty_service: "Payments SRE"
    slack_channel: "#payments-sre-alerts"

  data-pipeline:
    team: "data-sre"
    pagerduty_service: "Data SRE"
    slack_channel: "#data-sre-alerts"

  web-frontend:
    team: "frontend-sre"
    pagerduty_service: "Frontend SRE"
    slack_channel: "#frontend-sre-alerts"

  shared-infrastructure:
    team: "infrastructure-sre"
    pagerduty_service: "Infrastructure SRE"
    slack_channel: "#infra-sre-alerts"

rules:
  - "Every service must have exactly one owning team"
  - "Every alert must carry the team label"
  - "Ownership changes must update both Prometheus rules and PagerDuty"
  - "Unowned services must be assigned within 1 week"

Maintenance Silences

During planned maintenance, silence alerts to prevent false pages. Use Alertmanager silences with specific matchers, time limits, and descriptions. Never silence broadly or permanently.

Alertmanager silence configuration


# Create silence via amtool
# amtool silence add \
#   alertname=DiskSpaceLow \
#   instance=node-1 \
#   --duration=4h \
#   --comment="Disk expansion in progress"

silence_configuration:
  rules:
    - match:
        alertname: "DiskSpaceLow"
        instance: "node-1"
      duration: "4h"
      comment: "Disk expansion in progress"
      creator: "alice@example.com"

    - match:
        service: "user-api"
      duration: "2h"
      comment: "Scheduled deployment"
      creator: "bob@example.com"

  best_practices:
    - "Always set a duration — never create permanent silences"
    - "Include a comment explaining why the silence exists"
    - "Use specific matchers — do not silence broad categories"
    - "Review active silences daily"
    - "Remove silences when maintenance completes"

  monitoring:
    - "Track active silence count"
    - "Alert when silences exceed 24 hours"
    - "Report on silence usage monthly"

Testing Routes

Test your alert routing before you need it. Send test alerts through Alertmanager and verify they reach the correct channels. Test routing changes in staging before applying to production.

Test alert routing with amtool


# Send a test alert
amtool alert add \
  alertname=TestAlert \
  severity=critical \
  service=test-service \
  team=platform-sre \
  runbook_url="https://example.com/runbook" \
  summary="Test alert for routing verification" \
  --alertmanager.url=http://alertmanager:9093

# Verify the alert was routed correctly
amtool alert query alertname=TestAlert

# Check silence creation
amtool silence add \
  alertname=TestAlert \
  --duration=5m \
  --comment="Routing test"

# Verify silence is active
amtool silence query

# Clean up test alert
amtool alert expire alertname=TestAlert

Test routing quarterly

Test your alert routing quarterly. Send test alerts for each team and verify they reach the correct PagerDuty service and Slack channel. This catches routing problems before they cause missed pages.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.