Stage 6 · Operate
Incident Response & On-Call
Paging Policy
Page only actionable symptoms using PagerDuty schedules, escalation rules, and ownership maps.
Paging Principles
Pages interrupt lives. Every page should require human action. If a page fires and no one needs to do anything, it should not be a page. Design your paging policy around this principle: page only on actionable symptoms.
A page is an interruption to someone's life. It may wake them at 3 AM. It may interrupt dinner with family. Treat every page as sacred — only page when a human must act. Everything else is a ticket, a dashboard, or noise.
What to Page On
Page on symptoms that affect users. Do not page on causes, infrastructure metrics, or warnings. A user-facing error rate spike is a page. A CPU spike is not — it may resolve on its own or be handled by autoscaling.
paging_criteria:
page:
- "Error rate > 5% for 5 minutes (user-facing)"
- "Latency p99 > 2s for 5 minutes (user-facing)"
- "Availability < 99.9% for 15 minutes"
- "Data loss detected"
- "Security breach confirmed"
- "Complete service unavailability"
ticket:
- "Error rate > 1% for 15 minutes"
- "Disk usage > 85%"
- "Certificate expires within 14 days"
- "Deployment rollback needed"
- "Performance degradation (not user-impacting)"
dashboard:
- "CPU > 80% for 10 minutes"
- "Memory > 75%"
- "Pod restarts < 5 per hour"
- "SSL certificate expires within 30 days"
ignore:
- "Single pod restart"
- "Transient network blip"
- "Scheduled maintenance window"
- "Known flaky test"PagerDuty Schedules
PagerDuty schedules define who is on-call and when. Design schedules that are fair, predictable, and aligned with your team structure. Use layers for primary/secondary coverage.
pagerduty_schedules:
- name: "SRE Primary"
type: "weekly_rotation"
time_zone: "America/New_York"
rotation:
type: "weekly"
start_time: "2024-01-08T09:00:00-05:00"
users:
- "alice@example.com"
- "bob@example.com"
- "charlie@example.com"
- "diana@example.com"
- name: "SRE Secondary"
type: "weekly_rotation"
time_zone: "America/New_York"
rotation:
type: "weekly"
start_time: "2024-01-15T09:00:00-05:00"
users:
- "bob@example.com"
- "charlie@example.com"
- "diana@example.com"
- "alice@example.com"
- name: "SRE Follow-the-Sun"
type: "layered"
layers:
- name: "Americas"
start: "14:00 UTC"
end: "22:00 UTC"
users: ["alice", "bob"]
- name: "EMEA"
start: "07:00 UTC"
end: "15:00 UTC"
users: ["charlie", "diana"]Escalation Rules
Escalation rules ensure pages are answered. If the primary does not acknowledge within a set time, escalate to the secondary. If the secondary does not respond, escalate to management.
escalation_policies:
- name: "SRE Standard"
rules:
- level: 1
targets:
- type: "schedule"
id: "sre-primary"
wait: 0
notification:
- "push"
- "sms"
- "phone"
- level: 2
targets:
- type: "schedule"
id: "sre-secondary"
wait: 5m
notification:
- "push"
- "sms"
- "phone"
- level: 3
targets:
- type: "user"
id: "sre-manager"
wait: 10m
notification:
- "push"
- "phone"
- name: "SEV1 Escalation"
rules:
- level: 1
targets:
- type: "schedule"
id: "sre-primary"
- type: "schedule"
id: "sre-secondary"
wait: 0
notification:
- "push"
- "sms"
- "phone"Ownership Maps
Ownership maps define which team is responsible for which service. When an alert fires, the ownership map determines who gets paged. Keep this map updated as teams and services change.
service_ownership:
user-api:
primary_team: "platform-sre"
secondary_team: "backend-eng"
pagerduty_service: "PD_SERVICE_USER_API"
escalation_policy: "sre-standard"
payment-service:
primary_team: "payments-sre"
secondary_team: "payments-eng"
pagerduty_service: "PD_SERVICE_PAYMENTS"
escalation_policy: "sre-critical"
data-pipeline:
primary_team: "data-sre"
secondary_team: "data-eng"
pagerduty_service: "PD_SERVICE_PIPELINE"
escalation_policy: "sre-standard"
web-frontend:
primary_team: "frontend-sre"
secondary_team: "frontend-eng"
pagerduty_service: "PD_SERVICE_WEB"
escalation_policy: "sre-standard"Every service must have exactly one primary team. If two teams think they own a service, neither takes responsibility. If no team owns it, incidents go unresolved. Assign clear, unambiguous ownership.
Paging Metrics
Track paging metrics to measure and improve your paging policy. High page volume, low acknowledgment rates, and frequent false positives indicate policy problems.
paging_metrics:
- name: "Pages per week"
target: "< 10 per engineer"
alert: "If > 15, investigate alert quality"
- name: "Mean time to acknowledge (MTTA)"
target: "< 5 minutes"
alert: "If > 5 minutes, check schedule accuracy"
- name: "Mean time to resolve (MTTR)"
target: "< 30 minutes"
alert: "If > 30 minutes, improve runbooks"
- name: "False positive rate"
target: "< 10%"
alert: "If > 10%, tune or delete alerts"
- name: "Pages outside business hours"
target: "< 20% of total pages"
alert: "If > 20%, investigate root causes"Monthly page volume reviews catch alert quality problems before they cause burnout. If page volume is increasing, investigate why. More pages do not mean more reliability — they often mean more noise.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.