Stage 7 · Master
Phase 3 — User Service
Monitoring
Instrument user-service with Prometheus counters and histograms shaped around this module's actual failure modes: membership churn, search latency, and cursor tampering — not just generic HTTP counters.
Generic HTTP Metrics Don't Answer Domain Questions
http_requests_total{status="500"} tells you something broke. It does not tell you whether Maple Ridge's board is suddenly unable to search their own resident directory, or whether cursor tampering attempts are climbing after a client library regression. This lesson adds metrics shaped around the specific behaviors this module cares about, on top of — not instead of — the generic request metrics Phase 1's platform middleware already exports.
package metrics
import "github.com/prometheus/client_golang/prometheus"
var (
MembershipsCreated = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "membership_created_total",
Help: "Memberships created, labeled by initial role.",
},
[]string{"role"},
)
SearchQueryDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "search_query_duration_seconds",
Help: "Latency of tenant-scoped directory search queries.",
Buckets: prometheus.DefBuckets,
},
[]string{"result"},
)
CursorRejections = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "cursor_rejections_total",
Help: "Pagination requests rejected due to an invalid or tampered cursor.",
},
)
)
func MustRegister(registry *prometheus.Registry) {
registry.MustRegister(MembershipsCreated, SearchQueryDuration, CursorRejections)
}CursorRejections has no labels and is a plain Counter, not a Vec — a sustained rise in this single number, with no dimension needed, is itself the alert-worthy signal: someone is probing DecodeCursor.
Recording Metrics Where the Behavior Actually Happens
Metrics are recorded at the point where the domain event is known, not retrofitted from an HTTP middleware that only sees status codes. The membership invite handler increments MembershipsCreated with the role label right where the row is created; the search handler times the query and labels the result as hit or miss.
h.metrics.MembershipsCreated.WithLabelValues(string(m.Role)).Inc()One line, added exactly where the invite succeeds — no separate reconciliation job needed to keep this counter accurate, because it increments in the same code path that commits the row.
start := time.Now()
results, err := repo.Search(ctx, orgID, query, limit)
result := "hit"
if err != nil {
result = "error"
} else if len(results) == 0 {
result = "miss"
}
metrics.SearchQueryDuration.WithLabelValues(result).Observe(time.Since(start).Seconds())Labeling by hit/miss/error — not by the search query text itself — is deliberate: query text is high-cardinality and would explode the metric's label set, in addition to leaking what residents' names people are searching for into a metrics backend.
| Metric | User-domain failure it detects | Actionable first response |
|---|---|---|
| membership_created_total | Invite flow changed role mix or stopped creating memberships | Compare by role after a release and inspect invite handler errors. |
| search_query_duration_seconds | Tenant directory search became slow or error-heavy | Check pg_trgm indexes and query plans before blaming HTTP latency. |
| cursor_rejections_total | Signed cursors are being forged, replayed incorrectly, or encoded by a regressed client | Inspect recent client releases and API Gateway logs for rejected pagination requests. |
Expose Metrics Using the Established Registry Pattern
The Prometheus endpoint below is intentionally just the Organization Service metrics pattern with user-service's domain collectors registered into the private registry.
registry := prometheus.NewRegistry()
metrics.MustRegister(registry)
r.Handle("/metrics", promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))# Fires if cursor tampering attempts exceed 10/minute sustained for 5 minutes.
rate(cursor_rejections_total[5m]) * 60 > 10This single PromQL line is exactly the kind of alert that turns the Pagination lesson's HMAC signature from a theoretical protection into an operationally monitored one.
Applied exercise
Design the membership churn dashboard
The board of a large HOA community wants a monthly view of how many memberships were created vs. suspended.
- Name the metric(s) this lesson already provides that a churn dashboard could use, and the metric(s) still missing.
- Write the PromQL for 'memberships created per role, per day, over the last 30 days' using the counter defined above.
- Decide how you would add a 'suspended' counter without duplicating MembershipsCreated's label design.
Deliverable
The PromQL query plus a short design note for the missing suspended-membership counter.
Completion checks
- The PromQL uses rate() or increase() appropriately for a counter, not a raw counter value.
- The suspended-membership design reuses the role label rather than inventing an unrelated shape.
Why is CursorRejections a plain Counter with no labels, rather than a CounterVec?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.