Stage 7 · Master
Phase 8 — Maintenance Service
Monitoring
Batch-job metrics via Pushgateway, and a reconciler that catches ledger drift
Why a Short-Lived Job Cannot Be Scraped
Every service so far exposed a /metrics endpoint for Prometheus to scrape at its own interval — that works because the process stays up between scrapes. cmd/billing-runner does not: it starts, bills a tenant, and exits, often in under a second. A scrape interval of even 15 seconds would frequently miss the entire process lifetime. Short-lived jobs push their metrics to a Pushgateway instead, which holds the last-pushed values until Prometheus scrapes the gateway itself on its normal schedule.
package observability
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/push"
)
// InvoicesGeneratedTotal, InvoiceGenerationDuration, InvoicesOverdue, and
// LedgerStatusDrift are registered into a dedicated registry (not the
// default global one) so PushBatchMetrics can push exactly this set and
// nothing accidentally pulled in from an unrelated package's init().
var (
Registry = prometheus.NewRegistry()
InvoicesGeneratedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "invoices_generated_total",
Help: "Invoices generated by a billing run, labeled by outcome.",
}, []string{"outcome"}) // outcome: created | skipped_existing | failed
InvoiceGenerationDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "invoice_generation_duration_seconds",
Help: "Time to generate and persist a single flat's invoice.",
Buckets: prometheus.DefBuckets,
})
InvoicesOverdue = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "invoices_overdue",
Help: "Current count of overdue invoices, labeled by tenant.",
}, []string{"tenant_id"})
LedgerStatusDrift = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "ledger_status_drift_total",
Help: "Invoices whose stored status did not match the status re-derived from payment_records.",
}, []string{"tenant_id"})
)
func init() {
Registry.MustRegister(InvoicesGeneratedTotal, InvoiceGenerationDuration, InvoicesOverdue, LedgerStatusDrift)
}
// PushBatchMetrics pushes the current values of every metric in Registry
// to the given Pushgateway URL under a fixed job name. It is called once,
// right before cmd/billing-runner and cmd/reconciler exit — there is no
// ongoing scrape loop to rely on, so this single push is the job's only
// chance to report anything.
func PushBatchMetrics(pushgatewayURL, jobName string) error {
return push.New(pushgatewayURL, jobName).Gatherer(Registry).Push()
}Organization Service's metrics lesson owns low-cardinality label design; this lesson applies it by keeping outcome to created, skipped_existing, and failed while focusing on Pushgateway reporting for jobs that exit.
BillingRunner.processFlat (Transactions lesson) already sits at exactly the right seam to record InvoicesGeneratedTotal and InvoiceGenerationDuration — no restructuring needed, just instrumentation added around the existing call.
package service
import (
"context"
"time"
"github.com/google/uuid"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
"github.com/hoa-platform/backend/services/maintenance-service/internal/observability"
)
// recordOutcome wraps CreateIfAbsent's result and any error into the
// outcome label observability.InvoicesGeneratedTotal expects, and times
// the whole per-flat attempt. It replaces the direct call to
// r.invoices.CreateIfAbsent inside processFlat.
func (r *BillingRunner) instrumentedCreateIfAbsent(ctx context.Context, invoice domain.Invoice, lineItems []domain.InvoiceLineItem) (bool, error) {
start := time.Now()
created, err := r.invoices.CreateIfAbsent(ctx, invoice, lineItems)
observability.InvoiceGenerationDuration.Observe(time.Since(start).Seconds())
switch {
case err != nil:
observability.InvoicesGeneratedTotal.WithLabelValues("failed").Inc()
case created:
observability.InvoicesGeneratedTotal.WithLabelValues("created").Inc()
default:
observability.InvoicesGeneratedTotal.WithLabelValues("skipped_existing").Inc()
}
return created, err
}
var _ = uuid.NilprocessFlat in the Transactions lesson calls r.invoices.CreateIfAbsent directly; wiring this instrumented wrapper in its place is this lesson's one-line change to that existing function, not a rewrite of the runner.
Detecting Drift Between Stored and Derived Status
An invoice's status field is a cached conclusion — it should always equal what you'd derive fresh from its payment_records rows plus its due date. If those two ever disagree, something upstream (a manual SQL fix, a bug, a partially-applied migration) corrupted the ledger. The reconciler's job is to notice, not to blindly trust either side.
package postgres
import (
"context"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
)
type ReconciliationRepository struct {
pool *pgxpool.Pool
}
func NewReconciliationRepository(pool *pgxpool.Pool) *ReconciliationRepository {
return &ReconciliationRepository{pool: pool}
}
type LedgerRow struct {
InvoiceID uuid.UUID
TenantID uuid.UUID
StoredStatus domain.InvoiceStatus
TotalPaise domain.Paise
PaidPaise domain.Paise
IsPastDueDate bool
}
// ListForReconciliation returns every non-terminal invoice (not yet paid
// or written off) along with the raw numbers needed to re-derive its
// status independently of whatever is currently stored.
func (r *ReconciliationRepository) ListForReconciliation(ctx context.Context) ([]LedgerRow, error) {
const q = `
SELECT id, tenant_id, status, total_paise, paid_paise,
(billing_period + INTERVAL '30 days') < now() AS is_past_due_date
FROM invoices
WHERE status NOT IN ('paid', 'written_off')`
rows, err := r.pool.Query(ctx, q)
if err != nil {
return nil, err
}
defer rows.Close()
var out []LedgerRow
for rows.Next() {
var lr LedgerRow
if err := rows.Scan(&lr.InvoiceID, &lr.TenantID, &lr.StoredStatus, &lr.TotalPaise, &lr.PaidPaise, &lr.IsPastDueDate); err != nil {
return nil, err
}
out = append(out, lr)
}
return out, rows.Err()
}
// PromoteToOverdue transitions one invoice to overdue via a direct
// UPDATE. Doing this with raw SQL rather than routing back through
// PaymentRepository (which only knows how to apply a payment or a
// write-off) is flagged explicitly in this lesson's quiz as a spot where
// the state-machine rule and this query could drift apart if
// domain.legalTransitions ever changes without this query being revisited.
func (r *ReconciliationRepository) PromoteToOverdue(ctx context.Context, invoiceID uuid.UUID) error {
_, err := r.pool.Exec(ctx, `UPDATE invoices SET status = 'overdue' WHERE id = $1`, invoiceID)
return err
}IsPastDueDate uses a fixed 30-day grace period from billing_period as a simplification — a per-tenant configurable due date is out of scope for this lesson and is exactly the kind of extension the exercise below asks for.
package service
import (
"context"
"log"
"github.com/google/uuid"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
"github.com/hoa-platform/backend/services/maintenance-service/internal/observability"
"github.com/hoa-platform/backend/services/maintenance-service/internal/repository/postgres"
)
type Reconciler struct {
repo *postgres.ReconciliationRepository
}
func NewReconciler(repo *postgres.ReconciliationRepository) *Reconciler {
return &Reconciler{repo: repo}
}
// Run re-derives what each non-terminal invoice's status ought to be from
// its raw numbers, using the exact same domain.CanTransition rules the
// payment path is bound by, and compares that to what is actually stored.
// A mismatch increments LedgerStatusDrift and is logged for investigation
// — it is never silently auto-corrected, because a drift is evidence of a
// bug somewhere else, not routine housekeeping.
func (rec *Reconciler) Run(ctx context.Context) error {
rows, err := rec.repo.ListForReconciliation(ctx)
if err != nil {
return err
}
overdueByTenant := map[uuid.UUID]int{}
for _, row := range rows {
derived := deriveStatus(row)
if derived != row.StoredStatus {
observability.LedgerStatusDrift.WithLabelValues(row.TenantID.String()).Inc()
log.Printf("ledger drift: invoice %s tenant %s stored=%s derived=%s",
row.InvoiceID, row.TenantID, row.StoredStatus, derived)
if derived == domain.InvoiceStatusOverdue && domain.CanTransition(row.StoredStatus, domain.InvoiceStatusOverdue) {
if err := rec.repo.PromoteToOverdue(ctx, row.InvoiceID); err != nil {
log.Printf("failed to promote invoice %s to overdue: %v", row.InvoiceID, err)
}
}
}
if derived == domain.InvoiceStatusOverdue {
overdueByTenant[row.TenantID]++
}
}
for tenantID, count := range overdueByTenant {
observability.InvoicesOverdue.WithLabelValues(tenantID.String()).Set(float64(count))
}
return nil
}
func deriveStatus(row postgres.LedgerRow) domain.InvoiceStatus {
switch {
case row.PaidPaise >= row.TotalPaise:
return domain.InvoiceStatusPaid
case row.IsPastDueDate:
return domain.InvoiceStatusOverdue
case row.PaidPaise > 0:
return domain.InvoiceStatusPartiallyPaid
default:
return domain.InvoiceStatusIssued
}
}The only automatic correction Run performs is promoting to overdue, and only when CanTransition confirms that promotion is legal from the invoice's current stored status — every other kind of drift is reported, never rewritten, because auto-fixing a paid/paid mismatch, say, could paper over a real bug in the payment path.
reconciler := service.NewReconciler(postgres.NewReconciliationRepository(pool))
if err := reconciler.Run(context.Background()); err != nil {
log.Fatalf("reconciliation run failed: %v", err)
}
if err := observability.PushBatchMetrics(os.Getenv("PUSHGATEWAY_URL"), "maintenance_reconciler"); err != nil {
log.Printf("failed to push metrics: %v", err)
}
log.Println("reconciliation run complete")The pool startup is the same batch composition root shown in cmd/billing-runner. The reconciler-specific work is deriving ledger status, pushing one final metric set, and exiting; its CronJob runs daily at 03:00 UTC.
Manufacturing Ledger Drift and Watching Pushgateway
| Observation | Metric | Operator action |
|---|---|---|
| billing job generated invoices | invoices_generated_total | confirm the run finished and inspect failures |
| stored status disagrees with derived ledger state | ledger_status_drift_total | investigate corruption before changing data |
| tenant has many overdue invoices | invoices_overdue | contact the committee or tune the tenant's collection process |
Applied exercise
Alert when overdue invoices exceed a per-tenant threshold
InvoicesOverdue is a gauge with no alerting behavior attached yet — a society accumulating a large number of overdue invoices should trigger operational attention before it becomes a financial crisis for the HOA.
- Add a Prometheus alerting rule (a YAML rule file, evaluated by Alertmanager) that fires when invoices_overdue for any tenant_id exceeds 5 for more than 24 hours.
- Extend Reconciler.Run to log a structured warning (not just the existing drift log line) whenever a single tenant's overdue count crosses the same threshold within one run.
- Write a unit test for the threshold-crossing log using a fake ReconciliationRepository seeded with 6 overdue invoices for one tenant.
- Document, in a comment above the alerting rule, why this is an alert (needs a human) rather than an automatic action (like auto-write-off).
Deliverable
A Prometheus alerting rule file, an updated Reconciler.Run with threshold-crossing structured logging, and a passing unit test for the logging behavior.
Completion checks
- go test ./services/maintenance-service/internal/service/... -run TestReconciler passes.
- The alerting rule references invoices_overdue and a threshold of 5, evaluated over a 24h window.
- The comment explains why this remains a human-in-the-loop alert rather than an automatic write-off or escalation.
Why does cmd/billing-runner push its metrics to a Pushgateway instead of exposing a /metrics endpoint for Prometheus to scrape?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.