Stage 7 · Master
Phase 17 — Observability
Prometheus
Give the connection pool a metric Phase 2 never measured, move every operational endpoint off the listener real users hit, and stand up an actual Prometheus container to scrape both for the first time in this course.
organization_http_requests_total Cannot See a Starved Connection Pool
Phase 2's Metrics lesson instruments every HTTP request, which is exactly the right signal for "are users seeing errors or slow responses." It says nothing about why a request might be slow. A pool with MaxConns exhausted makes every request wait on pgxpool.Pool.Acquire before it ever reaches a repository query — from organization_http_requests_total's point of view, that just looks like ordinary latency, indistinguishable from a slow query, a slow disk, or a slow network hop. This lesson adds the one metric that actually answers the pool-specific question: how many of the pool's connections are in use, against how many it's allowed to hand out.
pgxmetrics Implements prometheus.Collector Directly
pgxpool.Pool.Stat() is already synchronous and cheap — it reads counters the pool maintains internally, with no I/O. A common but unnecessary pattern spins up a goroutine that calls Stat() on a ticker and sets Gauges from it, introducing a staleness window and a goroutine to manage for no benefit. Implementing prometheus.Collector directly means Collect() calls Stat() exactly once, synchronously, only when Prometheus actually scrapes /metrics — the value is never more stale than the scrape interval itself.
// Package pgxmetrics exposes a pgxpool.Pool's connection-pool statistics
// as Prometheus metrics, using the hoa_ project-wide prefix so this family
// stays unambiguous alongside every other service's own metric family —
// unlike organization_http_requests_total, which is this one service's
// own hand-written HTTP metric and only ever needs to be unambiguous
// within organization-service's own dashboard.
package pgxmetrics
import (
"github.com/jackc/pgx/v5/pgxpool"
"github.com/prometheus/client_golang/prometheus"
)
type collector struct {
pool *pgxpool.Pool
service string
acquired *prometheus.Desc
maxConns *prometheus.Desc
acquireDur *prometheus.Desc
}
// Register wires pool's statistics into reg under the hoa_<service>_postgres_pool_*
// family. service is a fixed, code-defined constant ("organization") — never a
// value derived from a request, a tenant, or anything else that could vary
// at runtime, since a Prometheus label's entire safety depends on its
// value set being small and known in advance.
func Register(reg prometheus.Registerer, service string, pool *pgxpool.Pool) {
reg.MustRegister(&collector{
pool: pool,
service: service,
acquired: prometheus.NewDesc(
"hoa_"+service+"_postgres_pool_acquired_connections",
"Number of connections currently acquired from the pool.",
nil, nil,
),
maxConns: prometheus.NewDesc(
"hoa_"+service+"_postgres_pool_max_connections",
"Maximum number of connections this pool is configured to hand out.",
nil, nil,
),
acquireDur: prometheus.NewDesc(
"hoa_"+service+"_postgres_pool_acquire_duration_seconds_total",
"Cumulative time every Acquire call has spent waiting, in seconds.",
nil, nil,
),
})
}
func (c *collector) Describe(ch chan<- *prometheus.Desc) {
ch <- c.acquired
ch <- c.maxConns
ch <- c.acquireDur
}
// Collect calls Stat() exactly once per scrape — synchronous, no
// goroutine, no cached/stale value. If pool has already been closed
// (process shutting down mid-scrape), Stat() still returns a valid,
// zeroed struct rather than panicking, so Collect never needs a nil check.
func (c *collector) Collect(ch chan<- prometheus.Metric) {
stat := c.pool.Stat()
ch <- prometheus.MustNewConstMetric(c.acquired, prometheus.GaugeValue, float64(stat.AcquiredConns()))
ch <- prometheus.MustNewConstMetric(c.maxConns, prometheus.GaugeValue, float64(stat.MaxConns()))
ch <- prometheus.MustNewConstMetric(c.acquireDur, prometheus.CounterValue, stat.AcquireDuration().Seconds())
}
These three metrics carry no labels at all beyond the metric name itself (nil, nil in every NewDesc call) — Collect emits exactly one hoa_organization_postgres_pool_acquired_connections series, ever, for this service, regardless of how many organizations use it or how many individual connections rotate through the pool over the process's lifetime. A pool is a process-level resource, not a per-tenant or per-connection one; labeling it as though it were would misrepresent what the number actually measures.
Moving /healthz, /readyz, and /metrics Off Port 8080
Since Phase 2, /healthz, /readyz, and /metrics have lived on the same gin.Engine and the same :8080 port as every real API request. That was a reasonable simplification while this service had no orchestrator to speak of. It stops being reasonable the moment a public load balancer or an unauthenticated caller could reach an endpoint that reveals internal connection-pool pressure or process liveness — operational surface that belongs only to Kubernetes' own probes and Prometheus's own scrape config, never to the public internet.
engine.Use(middleware.ErrorHandler())
engine.Use(m.Middleware())
- engine.GET("/healthz", healthHandler.Live)
- engine.GET("/readyz", healthHandler.Ready)
- engine.GET("/metrics", gin.WrapH(promhttp.HandlerFor(m.Registry(), promhttp.HandlerOpts{})))
v1 := engine.Group("/api/v1")
Request ID, recovery, error mapping, metrics middleware, and business routes remain unchanged. Only operational endpoints leave the public router; their new private listener is the lesson's actual implementation.
package main
import (
"net/http"
"github.com/gin-gonic/gin"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/hoa-platform/backend/services/organization/internal/handler"
"github.com/hoa-platform/backend/services/organization/internal/server"
)
// newInternalServer serves healthz, readyz, and metrics on a listener no
// Kubernetes Service or Ingress ever targets — only the kubelet's own
// liveness/readiness probes and Prometheus's own scrape config reach
// port 9090. It reuses server.New from Phase 1 exactly as the public
// listener does; a *gin.Engine satisfies http.Handler either way.
func newInternalServer(healthHandler *handler.HealthHandler, reg *prometheus.Registry) *http.Server {
engine := gin.New()
engine.GET("/healthz", healthHandler.Live)
engine.GET("/readyz", healthHandler.Ready)
engine.GET("/metrics", gin.WrapH(promhttp.HandlerFor(reg, promhttp.HandlerOpts{})))
return server.New(":9090", engine)
}
repo := repository.NewOrganizationPostgres(pool)
svc := service.NewOrganizationService(repo, logger)
orgHandler := handler.NewOrganizationHandler(svc, validation.New())
healthHandler := handler.NewHealthHandler(pool)
m := metrics.New()
pgxmetrics.Register(m.Registry(), "organization", pool)
engine := router.New(orgHandler, m, logger)
internalSrv := newInternalServer(healthHandler, m.Registry())
go func() {
if err := internalSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
logger.Error().Err(err).Msg("internal server stopped unexpectedly")
}
}()
srv := server.New(fmt.Sprintf(":%d", cfg.Port), engine)
logger.Info().Str("addr", srv.Addr).Str("internal_addr", internalSrv.Addr).Msg("listening")
if err := server.Run(srv, 10*time.Second); err != nil {
logger.Error().Err(err).Msg("server stopped")
os.Exit(1)
}
logger.Info().Msg("shut down cleanly")
internalSrv is deliberately not passed through server.Run's graceful-shutdown path — it is fire-and-forget, stopped only when the process itself exits. Losing a few milliseconds of in-flight scrapes during shutdown is an acceptable cost for keeping this file's control flow centered on the public listener, the one that actually serves users.
If port 9090 is already in use, internalSrv.ListenAndServe returns immediately and the goroutine above logs it — the public API on :8080 keeps serving real requests regardless. This is a deliberate trade-off: an operational listener failing to bind should never take down user-facing traffic. The one real consequence is that Kubernetes' readiness probe (also on :9090, moved here from :8080) would then fail too, correctly pulling this instance out of rotation — which is the right outcome if this instance's internal port genuinely cannot bind.
Phase 2 Only Ever Authored Files for a Future Prometheus — This One Actually Scrapes
promtool check rules validated deploy/observability/organization-alerts.yml's syntax back in Phase 2, but no Prometheus process has ever actually loaded it or scraped this service. This lesson adds a real one to the local stack.
global:
scrape_interval: 15s
rule_files:
- /etc/prometheus/organization-alerts.yml
scrape_configs:
- job_name: organization
static_configs:
- targets: ["organization:9090"]
prometheus:
image: prom/prometheus:v2.54.1
restart: unless-stopped
volumes:
- ./deploy/observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./deploy/observability/organization-alerts.yml:/etc/prometheus/organization-alerts.yml:ro
ports:
- "9091:9090"
networks:
- hoa-net
9091 on the host maps to Prometheus's own default port 9090 inside the container — deliberately not 9090 on the host, which this lesson just reserved for organization-service's own internal listener.
Confirm the Internal Port Serves Everything Operational, and Prometheus Actually Scrapes It
Applied exercise
Reproduce a pool-exhaustion reading and confirm the metric actually moves
The lesson explains what hoa_organization_postgres_pool_acquired_connections should show under pressure. Prove it moves, rather than trusting the explanation.
- Temporarily set the pool's MaxConns configuration (wherever cfg or db.Connect currently sets it) to 2.
- Write a short throwaway script or use a load tool (even 5 concurrent curl loops against a slow endpoint) to hold more than 2 requests in flight simultaneously.
- While that load runs, curl :9090/metrics and confirm hoa_organization_postgres_pool_acquired_connections reads at or near 2, the configured max.
- Stop the load, confirm the value drops back down, and revert MaxConns to its original value.
Deliverable
The two /metrics readings (under load and at rest) plus the reverted configuration.
Completion checks
- The under-load reading shows acquired connections at or very near the configured max, not below it.
- MaxConns is restored to its original value at the end of the exercise.
Why does pgxmetrics implement prometheus.Collector directly instead of running a goroutine that periodically calls pool.Stat() and sets Gauges?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.