Stage 7 · Master
Phase 5 — API Gateway
Monitoring
Instrument gateway-service with the request and upstream-latency metrics that let an on-call engineer tell, in seconds, whether a slowdown originates at the gateway or inside a specific backend — without ever attaching a raw org_id as a Prometheus label.
The Gateway Is the Only Place That Sees Every Backend at Once
user-service's monitoring lesson established request-count and latency histograms scoped to that one service. gateway-service needs the same baseline, plus one thing no backend service alone can measure: upstream_duration, the time spent specifically waiting on the backend gateway-service just proxied to, separate from the gateway's own processing overhead. That distinction is what tells an on-call engineer whether an SLO breach lives in the gateway or in a specific backend before they've even opened a second dashboard.
package observability
import "github.com/prometheus/client_golang/prometheus"
var (
RequestsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "gateway_requests_total", Help: "Total requests handled by the gateway, by upstream service and status class."},
[]string{"upstream", "status_class"}, // upstream: user-service | auth-service ; status_class: 2xx | 4xx | 5xx
)
UpstreamDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "gateway_upstream_duration_seconds",
Help: "Time spent waiting on the upstream backend, separate from gateway-side overhead.",
Buckets: prometheus.DefBuckets,
},
[]string{"upstream"},
)
RateLimitRejections = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "gateway_rate_limit_rejections_total", Help: "Requests rejected by either rate-limit tier."},
[]string{"tier"}, // ip | org
)
)
func init() {
prometheus.MustRegister(RequestsTotal, UpstreamDuration, RateLimitRejections)
}upstream and status_class are the only labels here, both drawn from a small fixed set — never org_id, and never the raw request path, for exactly the unbounded-cardinality reason the User module's monitoring lesson established and the Authentication module's monitoring lesson reinforced for its own metrics.
Bucketing into 2xx/4xx/5xx rather than a label per exact status code (200, 201, 401, 403, 404...) keeps the label's cardinality small and fixed, while still answering the one question that actually matters for alerting: is the error rate elevated, and in which direction (client errors vs. server errors) — a per-code breakdown, if ever needed for a specific investigation, belongs in the structured access log from the logging lesson, which already carries the exact code.
Measuring Only the Proxied Portion, Not the Whole Request
func newReverseProxy(target *url.URL, logger zerolog.Logger, upstreamName string) *httputil.ReverseProxy {
proxy := httputil.NewSingleHostReverseProxy(target)
// ... Director and ErrorHandler from the reverse-proxy lesson unchanged ...
originalTransport := http.DefaultTransport
proxy.Transport = roundTripperFunc(func(req *http.Request) (*http.Response, error) {
start := time.Now()
resp, err := originalTransport.RoundTrip(req)
observability.UpstreamDuration.WithLabelValues(upstreamName).Observe(time.Since(start).Seconds())
return resp, err
})
return proxy
}
type roundTripperFunc func(*http.Request) (*http.Response, error)
func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }Wrapping proxy.Transport rather than timing the whole handler is what isolates upstream_duration to exactly the network call to the backend — gateway-side work like JWT verification and rate limiting (both of which run in earlier middleware, before the proxy transport is ever invoked) are correctly excluded from this specific measurement.
One Alert That Distinguishes 'Gateway Is Slow' From 'A Backend Is Slow'
groups:
- name: gateway-service
rules:
- alert: UpstreamErrorRateHigh
expr: |
sum(rate(gateway_requests_total{status_class="5xx"}[5m])) by (upstream)
/ sum(rate(gateway_requests_total[5m])) by (upstream) > 0.05
for: 5m
labels: { severity: critical }
annotations:
summary: "Upstream {{ $labels.upstream }} 5xx rate exceeds 5% over 5 minutes — investigate that specific backend, not the gateway itself."
- alert: UpstreamLatencyP99High
expr: histogram_quantile(0.99, sum(rate(gateway_upstream_duration_seconds_bucket[5m])) by (le, upstream)) > 2
for: 10m
labels: { severity: warning }
annotations:
summary: "p99 upstream latency for {{ $labels.upstream }} exceeds 2s — check that specific backend's own dashboards next."Both alerts fire per upstream label specifically so the alert text itself points an on-call engineer directly at the struggling backend, rather than a generic 'the gateway has errors' page that would still require a manual investigation step to identify which of two services is actually at fault.
Confirming Upstream Duration Reflects Only Proxy Time
Applied exercise
Decide whether rate-limit rejections should count toward gateway_requests_total
A request rejected by PerIPLimiter or PerOrgFairLimiter never reaches the proxy layer at all, so it currently increments neither RequestsTotal (which is only wired into proxy.go) nor UpstreamDuration — only RateLimitRejections.
- State whether this is the correct behavior, or whether rate-limited requests should also appear in gateway_requests_total under some status_class.
- If they should appear, propose what upstream label value a rejected request should carry, given it was never actually routed to a backend.
- Explain how your answer affects the UpstreamErrorRateHigh alert's accuracy — could counting rate-limited requests there create a false impression of backend errors?
Deliverable
A short written decision on whether and how rate-limit rejections should be reflected in gateway_requests_total.
Completion checks
- The answer recognizes that mixing rate-limit rejections into the upstream-labeled request total would conflate gateway-side rejections with actual backend errors, potentially triggering UpstreamErrorRateHigh for a condition that has nothing to do with the backend's health.
Why does gateway_upstream_duration_seconds measure time only within proxy.Transport's RoundTrip, rather than the full request handler duration including middleware?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.