Stage 7 · Master
Phase 9 — Payment Service
Monitoring
Instrument payment flows so retries, stale transitions, and settlement drift become visible before residents open support tickets.
Payment Metrics Must Separate Initiation, Delivery, Rejection, and Drift
A single "payments_total" counter tells you almost nothing useful in production. Payment operations fail in distinct places: initiation can be rejected locally, webhooks can arrive with bad signatures, transitions can be dropped as stale, and reconciliation can discover drift long after the original request. Metrics should mirror those boundaries so an alert points you at the right subsystem immediately instead of sending you on a logging treasure hunt.
package payment
import (
"time"
"github.com/prometheus/client_golang/prometheus"
)
type Metrics struct {
initiations *prometheus.CounterVec
webhookEvents *prometheus.CounterVec
webhookLatency *prometheus.HistogramVec
transitionRejected prometheus.Counter
reconciliationMismatch *prometheus.CounterVec
nonTerminalOldestAgeSec *prometheus.GaugeVec
}
func NewMetrics(reg prometheus.Registerer) *Metrics {
m := &Metrics{
initiations: prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "payment_initiations_total",
Help: "Count of payment initiation attempts partitioned by result.",
},
[]string{"result"},
),
webhookEvents: prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "payment_webhook_events_total",
Help: "Count of webhook events partitioned by provider and outcome.",
},
[]string{"provider", "outcome"},
),
webhookLatency: prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "payment_webhook_processing_seconds",
Help: "End-to-end webhook processing latency in seconds.",
Buckets: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5},
},
[]string{"provider", "outcome"},
),
transitionRejected: prometheus.NewCounter(
prometheus.CounterOpts{
Name: "payment_state_transition_rejected_total",
Help: "Count of stale or duplicate payment state transitions rejected by the monotonic guard.",
},
),
reconciliationMismatch: prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "payment_reconciliation_mismatch_total",
Help: "Count of reconciliation drift records partitioned by class.",
},
[]string{"class"},
),
nonTerminalOldestAgeSec: prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "payment_non_terminal_oldest_age_seconds",
Help: "Age in seconds of the oldest payment still stuck in a non-terminal state by status.",
},
[]string{"status"},
),
}
reg.MustRegister(
m.initiations,
m.webhookEvents,
m.webhookLatency,
m.transitionRejected,
m.reconciliationMismatch,
m.nonTerminalOldestAgeSec,
)
return m
}
func (m *Metrics) ObserveInitiation(result string) {
m.initiations.WithLabelValues(result).Inc()
}
func (m *Metrics) ObserveWebhook(provider, outcome string, startedAt time.Time) {
m.webhookEvents.WithLabelValues(provider, outcome).Inc()
m.webhookLatency.WithLabelValues(provider, outcome).Observe(time.Since(startedAt).Seconds())
}
func (m *Metrics) ObserveRejectedTransition() {
m.transitionRejected.Inc()
}
func (m *Metrics) ObserveReconciliationMismatch(class string) {
m.reconciliationMismatch.WithLabelValues(class).Inc()
}
func (m *Metrics) SetOldestNonTerminalAge(status string, age time.Duration) {
m.nonTerminalOldestAgeSec.WithLabelValues(status).Set(age.Seconds())
}Do not label metrics by org_id or flat_id. That would explode cardinality and leak tenant-specific operational patterns into your monitoring backend.
Rejected Transition Spikes Are a Canary for Upstream Disorder
payment_state_transition_rejected_total is unusually valuable because it measures something your application explicitly decided not to do. A small steady rate is normal under retries and duplicate events. A sudden spike means the provider is replaying heavily, delivering events out of order more often than usual, or your own processing has drifted from the provider's event semantics. In other words, this counter is a canary for upstream disorder, not just local load.
| Metric | Healthy interpretation | Spike usually means |
|---|---|---|
| payment_initiations_total{result="conflict"} | Clients are safely retrying with stable keys | The frontend or mobile app is reusing keys incorrectly across different bodies |
| payment_webhook_events_total{outcome="bad_signature"} | Close to zero outside deliberate rotation tests | Forged traffic, wrong secret wiring, or broken rotation rollout |
| payment_state_transition_rejected_total | Low background noise from duplicates or equal-rank events | Upstream reordering storm or a bug in your event-to-status mapping |
| payment_reconciliation_mismatch_total{class="amount_mismatch"} | Rare and operator-reviewed | Partial capture bug, currency rounding defect, or CSV parser issue |
Payment monitoring is less about CPU and memory and more about proving that external truth, internal state, and eventual settlement still line up. Metrics should make those seams obvious.
Alert on Stuck Money Movement, Not Just Process Crashes
A payment service can be perfectly healthy at the process level while residents are stuck in initiated or authorized for too long. That is why a stuck-payment alert should watch the age of the oldest non-terminal payment instead of only checking pod restarts or HTTP 500 rates. This metric is also safer than a per-payment label because it compresses business risk into a bounded, low-cardinality surface.
groups:
- name: payment.rules
rules:
- alert: PaymentStuckTooLong
expr: max(payment_non_terminal_oldest_age_seconds{status=~"initiated|authorized"}) > 900
for: 10m
labels:
severity: page
service: payment-service
annotations:
summary: "Payment stuck in a non-terminal state for more than 15 minutes"
description: "At least one payment has remained initiated or authorized longer than the accepted checkout-to-capture window."Use a hold period (for: 10m) so a short provider hiccup does not page you during every sandbox wobble.
Dashboard Panels Should Correlate Webhook Latency With Reconciliation Drift
{
"title": "Payment webhook latency and reconciliation drift",
"panels": [
{
"type": "timeseries",
"title": "Webhook p95 latency",
"targets": [
{
"expr": "histogram_quantile(0.95, sum by (le, provider) (rate(payment_webhook_processing_seconds_bucket[5m])))"
}
]
},
{
"type": "timeseries",
"title": "Reconciliation mismatches by class",
"targets": [
{
"expr": "sum by (class) (increase(payment_reconciliation_mismatch_total[24h]))"
}
]
}
]
}Seeing latency and drift side by side helps you answer a common finance question quickly: did last night's mismatches follow a webhook delivery slowdown, or are they unrelated?
- Failure mode: without a rejected-transition metric, an upstream reordering storm can silently distort operations before anyone notices in logs.
- Security concern: bad-signature outcomes should be measured, but org-specific labels should never be exported because they leak tenant activity and create unbounded cardinality.
- Verification target: curl /metrics and run the exact PromQL for the stuck-payment alert before you trust the page to wake someone at night.
Applied exercise
Build a payment operations dashboard that tells one story
Support reports residents are saying "my money left, but the app still says pending" and finance reports a small rise in reconciliation mismatches on the same morning.
- Choose the minimum three panels you would place on a payment on-call dashboard to explain whether the problem is initiation, webhook processing, or settlement drift.
- Write one threshold-based alert and one trend-based alert that complement each other instead of duplicating signal.
- Explain why you would not add org_id as a metric label even though finance wants per-building visibility.
- Describe one runbook step that uses the rejected-transition counter before escalating to the gateway provider.
Deliverable
A dashboard sketch and alert pair that help an on-call engineer decide whether a pending-payment complaint is caused by client retries, webhook disorder, or reconciliation lag.
Completion checks
- The chosen panels distinguish initiation health, webhook behavior, and reconciliation outcomes.
- At least one alert explicitly watches non-terminal age instead of generic pod health.
- The explanation rejects high-cardinality tenant labels on sound operational grounds.
Why is `payment_state_transition_rejected_total` a strong canary metric?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.