Stage 7 · Master
Phase 13 — Staff Service
Monitoring
Instrument staff-service so a silent authorization failure or a stalled attendance sync shows up as an alert before a resident or auditor notices it first.
Generic HTTP Metrics Miss Domain-Specific Failure
Every service in this course exports request rate and latency, and the label-cardinality rules have not changed since Organization Service set them. The staff-specific problem is that this service's worst failures produce *successful* HTTP responses. A denied status transition returns a clean 422; a guard who never checks in produces no request at all. Both are invisible on a latency dashboard, and the second one — silence — is the harder of the two to alert on.
+var TransitionDenied = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "staff_transition_denied_total",
Help: "Count of rejected staff status transitions, labeled by attempted transition.",
},
[]string{"from_status", "to_status"},
)
+var CheckInDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "staff_attendance_checkin_duration_seconds",
Help: "Latency of the check-in request path, including the idempotency upsert.",
Buckets: prometheus.DefBuckets,
},
[]string{"result"},
)
+var VendorAssignmentConflicts = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "staff_vendor_assignment_conflicts_total",
Help: "Count of vendor assignment attempts rejected as not eligible.",
},
)
func Register(registry *prometheus.Registry) {
+ registry.MustRegister(TransitionDenied, CheckInDuration, VendorAssignmentConflicts)
}Labeling TransitionDenied by from_status and to_status turns one counter into a small table: a spike specifically in terminated→active attempts is a very different signal from scattered on_leave→suspended attempts.
A Log Line Without the Actor Is Not Evidence
Every denied or conflicting write logs the organization, the resource, the actor who attempted it, and the outcome — the same fields an operator would need to answer 'who tried to do what, and why was it blocked' without re-deriving it from request traces.
package httpadapter
import (
"github.com/hoa-platform/backend/pkg/authctx"
"github.com/rs/zerolog"
)
func logDenied(logger zerolog.Logger, claims authctx.Claims, resource, action, reason string) {
logger.Warn().
Str("organization_id", claims.TenantID.String()).
Str("actor_subject", claims.Subject).
Str("resource", resource).
Str("action", action).
Str("reason", reason).
Msg("staff_service_denied")
}This helper is called from every 403 and 409 branch shown in the earlier lessons; the field names stay identical across all of them so a log query can filter on "reason" once, across the whole service.
Alert on Absence, Not Only on Errors
A batch or sync job that silently stops running produces no error at all — only a gap. The alert below fires when no attendance check-in has been recorded anywhere in the last hour during business hours, which would otherwise look like a quiet, error-free system that has actually stopped working.
groups:
- name: staff-service
rules:
- alert: StaffTransitionDeniedSpike
expr: increase(staff_transition_denied_total[15m]) > 20
for: 5m
labels:
severity: warning
annotations:
summary: "Unusual volume of denied staff status transitions"
- alert: AttendanceSyncStalled
expr: increase(staff_attendance_checkin_duration_seconds_count[1h]) == 0
for: 10m
labels:
severity: critical
annotations:
summary: "No attendance check-ins recorded in the last hour"AttendanceSyncStalled is deliberately a critical alert: an absence of check-ins during business hours means either the scanners, the network, or the service itself has failed quietly — none of which will show up as a 5xx spike.
Confirm Metrics and Rules Are Actually Live
Applied exercise
Design an alert for vendor assignment conflicts
Operations wants to know if complaint-service is repeatedly attempting to assign vendors whose contracts have already lapsed, which could indicate a stale cache on their end rather than a one-off mistake.
- Write a PromQL expression using staff_vendor_assignment_conflicts_total that distinguishes a burst from occasional noise.
- Choose a
for:duration and justify it against how often this metric normally moves. - Decide the severity level and who should be paged versus notified asynchronously.
- State one non-alerting action (a dashboard panel or log query) that would help diagnose the root cause once paged.
Deliverable
A new alert rule appended to alerts.yaml plus a short rationale for the threshold and severity chosen.
Completion checks
- The alert would not fire on a single isolated conflict.
- The chosen severity matches how urgently a human needs to act.
Why is AttendanceSyncStalled based on an absence of activity rather than an explicit error metric?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.