Stage 7 · Master
Phase 12 — Visitor Service
Monitoring
Instrument visitor flows with low-cardinality metrics so replay spikes and approval timeouts become visible before guards start calling support.
Which metrics explain gate behavior without leaking secrets
Visitor monitoring should answer four operational questions quickly: how many check-ins are succeeding, how many are failing as replay, how many die because they are expired or unknown, and how long successful check-ins take. None of those questions requires org_id, flat_id, token hash, or visitor name as a Prometheus label. In fact, those would be harmful labels: they leak business context, explode cardinality, and make the monitoring system itself harder to operate.
The right metric shape is deliberately boring: a counter vector for visitor_checkin_attempts_total partitioned by a tiny outcome label set, a counter for approval timeouts, and a latency histogram for the check-in path. This keeps the dashboard useful without turning Prometheus into a shadow database of sensitive visitor data.
package visitor
import (
"errors"
"time"
"github.com/prometheus/client_golang/prometheus"
)
type Metrics struct {
checkInAttempts *prometheus.CounterVec
approvalTimeouts prometheus.Counter
checkInLatency prometheus.Histogram
}
func NewMetrics(reg prometheus.Registerer) *Metrics {
attempts := prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: "visitor",
Name: "checkin_attempts_total",
Help: "Total visitor check-in attempts partitioned by outcome.",
},
[]string{"outcome"},
)
approvalTimeouts := prometheus.NewCounter(
prometheus.CounterOpts{
Namespace: "visitor",
Name: "approval_timeout_total",
Help: "Total walk-in approval requests that timed out without a host response.",
},
)
latency := prometheus.NewHistogram(
prometheus.HistogramOpts{
Namespace: "visitor",
Name: "checkin_latency_seconds",
Help: "End-to-end latency for visitor check-in requests.",
Buckets: []float64{0.05, 0.1, 0.25, 0.5, 1, 2, 5},
},
)
reg.MustRegister(attempts, approvalTimeouts, latency)
return &Metrics{
checkInAttempts: attempts,
approvalTimeouts: approvalTimeouts,
checkInLatency: latency,
}
}
func (m *Metrics) ObserveCheckIn(started time.Time, err error) {
outcome := "success"
switch {
case errors.Is(err, ErrInvitationAlreadyUsed):
outcome = "replay"
case errors.Is(err, ErrInvitationExpired), errors.Is(err, ErrApprovalTimedOut):
outcome = "expired"
case errors.Is(err, ErrInvitationTokenNotFound):
outcome = "not_found"
case err != nil:
outcome = "error"
}
m.checkInAttempts.WithLabelValues(outcome).Inc()
m.checkInLatency.Observe(time.Since(started).Seconds())
}
func (m *Metrics) IncApprovalTimeout() {
m.approvalTimeouts.Inc()
}The label set is intentionally tiny. Adding org_id or token_hash here would both leak sensitive context and create unbounded metric cardinality.
| Metric design choice | Why it is safe | What to avoid |
|---|---|---|
| outcome labels: success, replay, expired, not_found, error | Low cardinality and directly useful for alerting | Per-token or per-flat labels that create secret leakage and cardinality blow-ups |
| Single latency histogram | Shows the guard-facing experience without exposing resident or visitor identity | Separate histograms per org or device that turn monitoring into a side-channel |
What alert catches a replay spike before it becomes a security incident
A replay here and there is normal: someone rescans, a response gets lost, or a visitor reopens an old QR screenshot. A replay spike is different. If the ratio of replay attempts to all check-in attempts jumps sharply across several minutes, that can mean leaked invitation tokens, a confused guard workflow, or active probing at the gate. The alert should therefore be ratio-based, not a raw count, so a busy evening does not page you just because traffic is high.
groups:
- name: visitor-service-visitor
rules:
- alert: VisitorReplaySpike
expr: |
sum(rate(visitor_checkin_attempts_total{outcome="replay"}[10m]))
/
clamp_min(sum(rate(visitor_checkin_attempts_total[10m])), 1)
> 0.15
for: 15m
labels:
severity: warning
service: visitor-service
domain: visitor
annotations:
summary: "Visitor replay attempts are unusually high"
description: "More than 15% of recent visitor check-in attempts are replay outcomes. Investigate token leakage, gate retry storms, or guard workflow issues."A ratio alert avoids false pages on high-volume evenings and instead focuses on unusual replay behavior as a fraction of all check-ins.
If you add org_id or flat_id as Prometheus labels, anyone with dashboard access can infer which societies are seeing unusual traffic. Monitoring data needs the same least-information discipline as the API itself.
How Grafana should present the guard-desk story
A useful visitor dashboard has three panels on one row. First, a stacked rate panel for visitor_checkin_attempts_total by outcome so operators can see success versus replay versus expired at a glance. Second, a stat panel for visitor_approval_timeout_total over the last hour so offline-host problems stand out immediately. Third, a percentile latency panel for visitor_checkin_latency_seconds showing p50, p95, and p99 so you can tell whether the gate queue problem is security-related or simply slow I/O.
A concrete failure mode here is over-instrumentation: if you emit one series per guard device or per token, the metrics backend becomes slower right when the gate is busiest. Good monitoring for visitor systems is less about maximum detail and more about fast, stable signals the on-call engineer can trust during a rush.
Which local commands prove the instrumentation is live
Applied exercise
Investigate a replay-rate page without leaking tenant data
Prometheus fires VisitorReplaySpike at 7:40 PM while the gate staff say the app is 'acting weird' during peak arrivals.
- Describe which dashboard panel you would inspect first and what healthy versus suspicious shapes would look like.
- Explain why the alert should key off replay ratio rather than total replay count.
- List the labels you would allow on visitor metrics and the sensitive labels you would forbid, with a tenant-isolation reason for each forbidden label.
- Write the first two hypotheses you would investigate: one security-oriented and one reliability-oriented.
Deliverable
An incident triage note that uses only approved metrics, interprets the replay alert correctly, and avoids exposing tenant-specific traffic in the monitoring design.
Completion checks
- Your permitted label list stays low-cardinality and excludes org_id, flat_id, token_hash, and visitor_name.
- One hypothesis addresses leaked or copied tokens, and another addresses retry storms or dropped responses.
- Your explanation for the ratio alert mentions why raw counts page noisily during busy but healthy periods.
Why is `visitor_checkin_attempts_total{outcome}` a better metric shape than labeling by org_id or token_hash?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.