Stage 7 · Master
Phase 11 — Notice Service
Monitoring
Expose the signals that tell you when scheduled notices lag, searches slow down, or attachment verification starts failing.
Which metrics show the publisher is falling behind?
The best Notice-specific signal is publish lag: actual published_at minus scheduled_at. A worker that is alive but consistently late is still a resident-facing failure, especially for time-sensitive water, lift, or security notices. That is why notice_publish_lag_seconds deserves its own metric instead of being inferred indirectly from generic job duration metrics.
Lag also helps diagnose failure modes. A sudden p95 spike can mean the CronJob is not firing, the advisory lock is stuck behind another unhealthy process, or the database is too slow to scan due rows on time. You learn more from business lag than from container CPU alone.
How do search and attachment failures become time-series?
Search latency should be a histogram because you care about distribution, not just totals. Attachment verification failures should be a counter labeled by reason so you can distinguish 'object missing' from 'content_type_mismatch' and spot whether the issue is user behavior, client bugs, or storage-side trouble.
package notice
import (
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
type Metrics struct {
publishLag prometheus.Histogram
searchLatency *prometheus.HistogramVec
attachmentUploadFailures *prometheus.CounterVec
}
func NewMetrics(reg prometheus.Registerer) *Metrics {
factory := promauto.With(reg)
return &Metrics{
publishLag: factory.NewHistogram(prometheus.HistogramOpts{
Name: "notice_publish_lag_seconds",
Help: "Observed lag between scheduled_at and actual published_at for notices.",
Buckets: []float64{1, 5, 10, 30, 60, 120, 300, 600},
}),
searchLatency: factory.NewHistogramVec(prometheus.HistogramOpts{
Name: "notice_search_latency_seconds",
Help: "Latency of full-text notice searches.",
Buckets: prometheus.DefBuckets,
}, []string{"query_kind"}),
attachmentUploadFailures: factory.NewCounterVec(prometheus.CounterOpts{
Name: "notice_attachment_upload_failures_total",
Help: "Count of notice attachment verification failures grouped by failure reason.",
}, []string{"reason"}),
}
}
func (m *Metrics) ObservePublished(scheduledAt, publishedAt time.Time) {
if scheduledAt.IsZero() || publishedAt.IsZero() {
return
}
lag := publishedAt.Sub(scheduledAt).Seconds()
if lag < 0 {
lag = 0
}
m.publishLag.Observe(lag)
}
func (m *Metrics) ObserveSearch(query string, duration time.Duration) {
queryKind := "keyword"
if len(query) > 24 || containsWhitespace(query) {
queryKind = "phrase"
}
m.searchLatency.WithLabelValues(queryKind).Observe(duration.Seconds())
}
func (m *Metrics) IncAttachmentUploadFailure(reason string) {
if reason == "" {
reason = "unknown"
}
m.attachmentUploadFailures.WithLabelValues(reason).Inc()
}
func containsWhitespace(value string) bool {
for _, r := range value {
if r == ' ' || r == ' ' || r == '
' {
return true
}
}
return false
}These helpers keep metric naming consistent and give the rest of the notice package one place to decide how failures are bucketed.
What should Prometheus alert on for Notice?
Alert on symptoms residents feel. Publish lag is the primary paging signal because it tells you scheduled notices are not becoming visible on time. Search latency can page at a higher threshold or create a warning-level alert, and attachment failures are often better for dashboards unless they spike sharply after a new client release.
groups:
- name: notice.rules
rules:
- alert: NoticePublisherLagHigh
expr: histogram_quantile(0.95, sum(rate(notice_publish_lag_seconds_bucket[15m])) by (le)) > 120
for: 10m
labels:
severity: warning
annotations:
summary: Notice publisher is falling behind schedule
description: 95th percentile notice publish lag has been above 120 seconds for 10 minutes.This alert fires on the distribution residents experience, not on whether one individual job happened to run slowly once.
| Grafana panel | What it shows | Why it matters |
|---|---|---|
| Notice publish lag p50/p95 | Histogram quantiles over notice_publish_lag_seconds | Makes it obvious when scheduled posts are routinely late even though the worker is technically alive |
| Notice search latency heatmap | Distribution of notice_search_latency_seconds by query_kind | Shows whether phrase searches are disproportionately slower than short keyword searches |
| Attachment verification failure breakdown | Rate of notice_attachment_upload_failures_total by reason | Separates client misuse from storage or metadata regressions |
High-cardinality labels turn observability into a performance problem and can leak tenant-specific data. Keep labels coarse, such as query_kind or failure reason, and use logs or traces for per-request detail.
How do you verify dashboards and scrapes locally?
Before shipping any alert, confirm the metric exists on the /metrics endpoint and changes when you exercise the path. Trigger one scheduled publish, run a couple of searches, and intentionally fail one attachment confirmation so the metrics move away from zero. This local feedback loop catches naming mistakes and forgotten registration calls long before Prometheus in-cluster silently misses them.
Applied exercise
Design the first Notice dashboard
Your team is about to hand Notice support to on-call engineers who did not build the feature. They need one dashboard and one alert that quickly explain whether scheduled publishing, search, and attachment verification are healthy.
- Choose the three top-level panels you would place on the dashboard and justify each one.
- Define the threshold and duration for the first publish-lag alert and explain why it is warning-level or page-level.
- Name one label you intentionally avoid on search metrics to keep cardinality under control.
- Describe a local action that should cause notice_attachment_upload_failures_total to increase so you can verify the panel wiring.
Deliverable
A one-page dashboard plan listing the panels, the first alert rule, one cardinality guardrail, and one local verification action per signal.
Completion checks
- The proposed alert is tied to resident-visible lag, not just generic pod health.
- The answer explicitly avoids high-cardinality labels such as raw query text or org IDs.
- The verification step changes a real notice metric rather than only checking that /metrics returns 200.
Why is notice_publish_lag_seconds more useful than only tracking CronJob runtime?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.