Stage 7 · Master
Phase 4 — Authentication Service
Monitoring
Instrument auth-service with the handful of metrics that distinguish normal login friction from an active brute-force or token-theft campaign — especially refresh_replay_detected_total, the single most important signal this module produces.
A Health Dashboard Can Look Green During an Active Attack
auth-service's request latency and 5xx rate — the metrics the User module's monitoring lesson established as a baseline for every service — can look completely normal while a credential-stuffing bot slams the login endpoint with valid-looking requests that simply fail authentication, or while a stolen refresh token is actively being replayed by an attacker. Those are 401s and 200s respectively, not errors a generic dashboard would ever flag.
package observability
import "github.com/prometheus/client_golang/prometheus"
var (
LoginAttempts = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "login_attempts_total", Help: "Login attempts by outcome."},
[]string{"result"}, // success | bad_credentials | locked | breached_password_rejected
)
// The single highest-priority security metric in this module: a nonzero
// rate here means a refresh token that had already been rotated was
// presented again, the direct signature of token theft (refresh-tokens lesson).
RefreshReplayDetected = prometheus.NewCounter(
prometheus.CounterOpts{Name: "refresh_replay_detected_total", Help: "Refresh token families revoked due to detected reuse of an already-rotated token."},
)
JWTVerificationFailures = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "jwt_verification_failures_total", Help: "Access token verification failures by reason."},
[]string{"reason"}, // expired | bad_signature | unknown_kid | disallowed_alg | denylisted
)
AccountLockouts = prometheus.NewCounter(
prometheus.CounterOpts{Name: "account_lockouts_total", Help: "Accounts newly locked due to exceeding the failed-attempt threshold."},
)
)
func init() {
prometheus.MustRegister(LoginAttempts, RefreshReplayDetected, JWTVerificationFailures, AccountLockouts)
}jwt_verification_failures_total is labeled by reason specifically so unknown_kid (a genuine misconfiguration, worth paging on) is never confused with expired (routine, expected constantly) or disallowed_alg (a potential attack attempt, worth investigating) in the same aggregate number.
One Alert Rule That Justifies This Entire Lesson
groups:
- name: auth-service-security
rules:
- alert: RefreshTokenReplayDetected
expr: increase(refresh_replay_detected_total[5m]) > 0
for: 0m
labels: { severity: critical }
annotations:
summary: "Refresh token replay detected — at least one token family was just revoked due to reuse of an already-rotated token, indicating likely token theft."
- alert: LoginFailureSpike
expr: |
sum(rate(login_attempts_total{result="bad_credentials"}[5m]))
/ sum(rate(login_attempts_total[5m])) > 0.5
for: 10m
labels: { severity: warning }
annotations:
summary: "Over half of login attempts are failing over a sustained 10-minute window — consistent with an active credential-stuffing campaign."
- alert: UnknownSigningKeyID
expr: increase(jwt_verification_failures_total{reason="unknown_kid"}[15m]) > 0
for: 0m
labels: { severity: critical }
annotations:
summary: "A verifier rejected a token due to an unrecognized kid — check for a key-rotation runbook step executed out of order (deployment lesson)."RefreshTokenReplayDetected fires on any nonzero count with for: 0m — unlike most alerts, which tolerate some baseline noise, a single genuine replay detection is itself already the security incident, not an early warning of one.
None of these metrics are labeled by user_id or org_id — a per-organization breakdown of login failures, if ever needed for a specific investigation, belongs in a queryable audit log (a table, not a Prometheus label), for exactly the unbounded-cardinality reason the User module's monitoring lesson established for organization_id.
A Security-Focused Panel, Not Just a Latency Panel
- Login outcome breakdown over time (login_attempts_total by result) — a sudden shift in the bad_credentials proportion is the earliest signal of a credential-stuffing attempt, well before AccountLockouts starts climbing.
- refresh_replay_detected_total as a single always-visible counter, not buried in a drill-down — this number should be zero essentially always, and any nonzero value deserves to be impossible to miss on the primary dashboard.
- jwt_verification_failures_total broken down by reason, stacked — distinguishing routine expired noise from the much rarer, much more concerning disallowed_alg or unknown_kid categories at a glance.
Confirming the Replay Alert Actually Fires
Applied exercise
Design the dashboard panel for a suspected token-theft incident
RefreshTokenReplayDetected has just fired in production for a specific user account, and on-call staff need to investigate within minutes.
- List the specific questions an on-call engineer needs answered in the first five minutes (e.g., how many families were revoked, over what time window, from what apparent source).
- State which of those questions Prometheus metrics can answer versus which require the audit log or database directly, given the cardinality constraint on org_id and user_id labels.
- Propose one piece of information that should be included directly in the alert annotation itself (beyond the generic summary shown above) to save the first investigative step.
Deliverable
A short written incident-response checklist distinguishing metric-answerable questions from log/database-answerable ones.
Completion checks
- The checklist correctly routes user-specific or org-specific questions to logs/database rather than proposing new high-cardinality Prometheus labels.
- The proposed alert annotation addition is genuinely actionable within the first minute, not generic.
Why does the RefreshTokenReplayDetected alert use for: 0m while LoginFailureSpike uses for: 10m?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.