Stage 7 · Master
Phase 17 — Observability
Alerting
Derive three new alert rules from an event-processing SLO this course has never stated before, add a meta-alert for the exporter outage the Dashboards lesson just warned about, and explain why not one of them is ever allowed to carry a per-tenant label.
99% of Events Reach notification_jobs Within 60 Seconds — a New SLO for a New Kind of Work
Phase 2's OrganizationErrorBudgetFastBurn and OrganizationReadLatencyHigh alert on synchronous HTTP requests, where success and latency are observed the instant a response is written. Nothing in this platform has ever stated an SLO for asynchronous work — a payment.completed event's journey from outbox row to notification job has no single request whose response time to measure. This lesson states one explicitly: 99% of events must reach notification_jobs within 60 seconds of being written to payment_outbox, and derives every alert below from that one number, exactly the way Phase 2 derived its own thresholds from a stated SLO rather than a guessed round number.
Backlog Growth, Consumer Lag, and Dead-Letter Rate — None of Which Phase 2 Could Have Written
groups:
- name: event-pipeline
rules:
- alert: PaymentOutboxBacklogGrowing
expr: |
deriv(hoa_payment_outbox_backlog[10m]) > 0
and
hoa_payment_outbox_backlog > 50
for: 5m
labels:
severity: page
annotations:
summary: "payment_outbox backlog is growing, not just temporarily elevated"
description: "The relay is either down or falling behind faster than it can catch up — deriv() confirms the trend is upward, not a brief spike already draining. At the 60-second delivery SLO, a sustained backlog above 50 rows is already at risk of breach."
- alert: NotificationConsumerLagHigh
expr: |
sum(kafka_consumergroup_lag{consumergroup="notification-service", topic="payment.completed"}) > 100
for: 5m
labels:
severity: page
annotations:
summary: "notification-service's consumer group is falling behind payment.completed"
description: "At this platform's typical throughput, a lag above 100 messages implies the 60-second delivery SLO is already at risk — check whether the consumer process has crashed (Part 16, Lesson 4) before assuming it is merely slow."
- alert: DeadLetterRateElevated
expr: |
sum(rate(notification_dlq_messages_total[15m])) > 0.05
for: 10m
labels:
severity: ticket
annotations:
summary: "Dead-letter rate for payment.completed is elevated"
description: "Unlike the two rules above, this one alone does not necessarily mean the pipeline is behind — a poison-event spike can coexist with healthy consumer lag. severity: ticket, not page: this needs investigation, not an immediate wake-up, since no event is being silently lost (Part 16, Lesson 6's DLQ guarantees that)."
- alert: KafkaExporterMissing
expr: |
absent(up{job="kafka-lag"} == 1)
for: 2m
labels:
severity: ticket
annotations:
summary: "kafka-exporter is not being scraped — consumer lag is currently unmeasured, not necessarily zero"
description: "This is a meta-alert about missing data, not about a bad value — exactly the gap-versus-zero distinction the Dashboards lesson's Consumer group lag panel warned a human reader to notice on their own. This rule catches the same failure mode without depending on someone happening to look at the dashboard."
A one-time burst of 60 payments in a single minute would briefly push the backlog above 50 even with a perfectly healthy relay running normally — that spike drains on its own within seconds and deserves no page. deriv(...) > 0 requires the backlog to actually be trending upward over the last 10 minutes, not merely elevated at one instant, which is what distinguishes 'the relay is falling behind' from 'traffic briefly spiked and is already recovering.'
Two Pages, Two Tickets — Derived From Consequence, Not From Habit
| Alert | Severity | Why |
|---|---|---|
| PaymentOutboxBacklogGrowing | page | A stuck or crashed relay silently delays every payment confirmation currently in flight — the SLO is actively at risk right now |
| NotificationConsumerLagHigh | page | A stuck or crashed consumer has the same user-facing consequence as a stuck relay, just on the other side of the topic |
| DeadLetterRateElevated | ticket | No event is lost — Part 16's DLQ already guarantees that — this is a data-quality or upstream-bug signal worth investigating on the next business day, not an emergency |
| KafkaExporterMissing | ticket | A missing exporter is an observability gap, not a pipeline failure — the pipeline itself may be perfectly healthy and simply unmeasured |
No Alert Here Is Ever Allowed to Carry an organization_id Label — For a Different Reason Than the Metrics Themselves
The metrics feeding these rules are already unlabeled by org_id, for the cardinality reasons this module established from the Prometheus lesson onward. Even setting that aside, an alert rule keyed per tenant would introduce a second, independent hazard: a single unusually noisy organization generating its own backlog or lag spike would page on-call once per tenant experiencing it, rather than once for the platform-wide condition — turning one root cause into a multiplying number of pages exactly proportional to how many tenants happen to be affected. Aggregate-only alerting, with tenant-level investigation happening afterward via Lesson 5's traces, is the only way both hazards are avoided at once.
Validate the New Rule File the Same Way Phase 2 Validated Its Own
Checking deploy/observability/event-pipeline-alerts.yml
SUCCESS: 4 rules found
rule_files:
- /etc/prometheus/organization-alerts.yml
- /etc/prometheus/event-pipeline-alerts.yml
prometheus:
volumes:
- ./deploy/observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./deploy/observability/organization-alerts.yml:/etc/prometheus/organization-alerts.yml:ro
- ./deploy/observability/event-pipeline-alerts.yml:/etc/prometheus/event-pipeline-alerts.yml:ro
Applied exercise
Trigger PaymentOutboxBacklogGrowing for real and confirm it does not fire on a harmless burst
The alert's whole design point is distinguishing a real problem from a harmless traffic spike. Prove it does that, in both directions.
- With the relay running normally, send 60 payment requests in quick succession and confirm PaymentOutboxBacklogGrowing does not fire, since the backlog should drain within the for: 5m window.
- Stop the relay process entirely and send another 60 requests over several minutes, confirming the backlog is now trending upward with the relay not running.
- Confirm PaymentOutboxBacklogGrowing fires within roughly 5-10 minutes of the relay being stopped, and check its rendered annotations describe the situation correctly.
- Restart the relay, confirm the backlog drains, and confirm the alert resolves on its own.
Deliverable
Prometheus's own alert state (pending/firing/resolved) captured at each stage of the exercise.
Completion checks
- The harmless burst with the relay running does not trigger a firing alert.
- Stopping the relay does trigger the alert within a reasonable window, and it resolves automatically once the relay resumes and the backlog drains.
Why does PaymentOutboxBacklogGrowing require both deriv(...) > 0 and an absolute threshold, rather than alerting on the absolute threshold alone?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.