Stage 7 · Master
Phase 14 — Notification Service
Monitoring
Track delivery outcomes per channel, alert on a growing job backlog before residents notice delayed messages, and correlate a request to its eventual delivery through one shared identifier.
One Counter, Three Channels, Two Outcomes
Email, SMS, and push fail for different reasons at different rates — bundling them into one undifferentiated 'notifications sent' counter would hide a channel-specific outage behind an average. Every metric here is labeled by channel and, where relevant, by outcome.
package observability
import "github.com/prometheus/client_golang/prometheus"
var (
NotificationsSent = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "notifications_sent_total",
Help: "Delivery attempts by channel and outcome.",
},
[]string{"channel", "status"}, // status: sent | failed | dead
)
DeliveryLatency = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "notification_delivery_latency_seconds",
Help: "Time from job creation to a terminal sent or dead status.",
Buckets: []float64{1, 5, 15, 60, 300, 900},
},
[]string{"channel"},
)
JobsPending = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "notification_jobs_pending",
Help: "Current count of jobs waiting to be claimed, by channel.",
},
[]string{"channel"},
)
)
func Register(registry *prometheus.Registry) {
registry.MustRegister(NotificationsSent, DeliveryLatency, JobsPending)
}JobsPending is a gauge refreshed on a short interval by a small background loop querying SELECT channel, count(*) FROM notification_jobs WHERE status='pending' GROUP BY channel — it is this exact metric the worker's autoscaler reads in the previous lesson.
A Growing Queue Is Worth Paging On, Even Without Errors
groups:
- name: notification-service
rules:
- alert: NotificationBacklogGrowing
expr: delta(notification_jobs_pending[15m]) > 500
for: 10m
labels:
severity: warning
annotations:
summary: "notification_jobs_pending has grown by more than 500 in 15 minutes"
- alert: EmailDeliveryFailureRateHigh
expr: |
sum(rate(notifications_sent_total{channel="email", status="dead"}[10m]))
/
sum(rate(notifications_sent_total{channel="email"}[10m])) > 0.1
for: 10m
labels:
severity: critical
annotations:
summary: "More than 10% of email sends are ending in dead status"EmailDeliveryFailureRateHigh uses a ratio, not a raw count, so it stays meaningful whether the organization is sending ten emails an hour or ten thousand.
One Notification ID Threads Through Every Log Line
A support agent investigating a missing SMS needs to go from 'resident says they never got it' to the exact job, its attempts, and the provider's own response — without grepping three services' logs by hand. Every log line from request acceptance through final delivery status carries the same notification_id.
package application
import "github.com/rs/zerolog"
func WithNotificationContext(logger zerolog.Logger, notificationID, organizationID, channel string) zerolog.Logger {
return logger.With().
Str("notification_id", notificationID).
Str("organization_id", organizationID).
Str("channel", channel).
Logger()
}Every log call inside the worker, the adapter, and the API handler for a given job uses a logger built from this helper, so "grep for one ID" reconstructs the full lifecycle across process boundaries.
Confirm the Alert Fires and the ID Actually Threads Through
Applied exercise
Design a per-organization delivery SLO dashboard
A large organization is asking Meridian for a delivery-reliability report they can show their own residents' association board.
- Choose the specific metrics and label combinations that answer 'what percentage of this organization's notifications were delivered within 5 minutes, last 30 days'.
- Decide whether this requires a new metric label, a new query over existing metrics, or both.
- State one thing the current metrics cannot answer that this report would need, and how you would close that gap.
- Sketch the panel layout in words: what is the headline number, and what supports it.
Deliverable
A short written design naming exact metrics/queries and the one identified gap.
Completion checks
- The design references metrics that actually exist in this lesson's metrics.go, or explicitly proposes new ones.
- The identified gap is a real limitation, not a restatement of an existing metric.
Why does EmailDeliveryFailureRateHigh compute a ratio of dead-to-total sends rather than alerting on a raw count of dead emails?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.