Stage 7 · Master
Phase 8 — Maintenance Service
Transactions
Orchestrating a tenant-wide billing run without losing a single flat to a partial failure
One Billing Run, Many Independent Flats
Generating one invoice is a pure function; running billing for an entire society is an orchestration problem. Every flat's invoice must be attempted even if another flat's generation fails midway — a stale cross-service dependency should not cost every other resident their invoice for the month. billing_runs tracks the overall attempt, billing_run_failures records exactly which flats failed and why, and every individual invoice write is isolated in its own transaction.
CREATE TABLE billing_runs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
billing_period DATE NOT NULL,
status TEXT NOT NULL CHECK (status IN ('running', 'completed', 'completed_with_errors')),
flats_total INT NOT NULL DEFAULT 0,
flats_succeeded INT NOT NULL DEFAULT 0,
flats_failed INT NOT NULL DEFAULT 0,
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
completed_at TIMESTAMPTZ,
-- Rerunning billing for a period that already ran updates this same
-- row rather than creating a second one — see StartRun below.
UNIQUE (tenant_id, billing_period)
);
CREATE TABLE billing_run_failures (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
billing_run_id UUID NOT NULL REFERENCES billing_runs (id) ON DELETE CASCADE,
flat_id UUID NOT NULL,
reason TEXT NOT NULL,
occurred_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX billing_run_failures_run_id_idx ON billing_run_failures (billing_run_id);billing_run_failures rows are never deleted or overwritten between reruns of the same period — they are an append-only audit trail. A period that failed twice for the same flat, for different reasons, shows both attempts.
DROP TABLE IF EXISTS billing_run_failures;
DROP TABLE IF EXISTS billing_runs;The Counting Bug a Naive Rerun Design Hides
A tempting first design fetches the existing billing_runs row for a period if one exists, or inserts a new one otherwise, then increments flats_succeeded/flats_failed as it goes. Rerunning a period that already succeeded would fetch the old row — with its already-populated counters — and add to them, reporting far more flats processed than actually exist. The fix below resets the counters on every invocation instead of accumulating into whatever was left from last time.
package domain
import (
"time"
"github.com/google/uuid"
)
type BillingRunStatus string
const (
BillingRunStatusRunning BillingRunStatus = "running"
BillingRunStatusCompleted BillingRunStatus = "completed"
BillingRunStatusCompletedWithErrors BillingRunStatus = "completed_with_errors"
)
type BillingRun struct {
ID uuid.UUID
TenantID uuid.UUID
BillingPeriod time.Time
Status BillingRunStatus
FlatsTotal int
FlatsSucceeded int
FlatsFailed int
StartedAt time.Time
CompletedAt *time.Time
}
// FlatSummary is the minimal shape maintenance-service needs from
// flat-service to bill a flat: its identity and the area used for
// per-sqft proration. Declared here, in domain, for the same reason as
// ActiveCharge — client.FlatClient and service.BillingRunner both need
// the exact same type, not two structurally-identical lookalikes.
type FlatSummary struct {
ID uuid.UUID
AreaSqft float64
}package postgres
import (
"context"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
)
type BillingRunRepository struct {
pool *pgxpool.Pool
}
func NewBillingRunRepository(pool *pgxpool.Pool) *BillingRunRepository {
return &BillingRunRepository{pool: pool}
}
// StartRun (re)initializes the billing_runs row for this tenant and
// period, resetting every counter to zero and status to "running" even if
// a row already exists from a previous attempt. This is what makes
// rerunning a period safe to observe: the reported counts always describe
// only the current invocation, never an accumulation across attempts.
func (r *BillingRunRepository) StartRun(ctx context.Context, tenantID uuid.UUID, billingPeriod time.Time) (domain.BillingRun, error) {
const q = `
INSERT INTO billing_runs (tenant_id, billing_period, status, flats_total, flats_succeeded, flats_failed, started_at, completed_at)
VALUES ($1, $2, 'running', 0, 0, 0, now(), NULL)
ON CONFLICT (tenant_id, billing_period) DO UPDATE
SET status = 'running', flats_total = 0, flats_succeeded = 0, flats_failed = 0,
started_at = now(), completed_at = NULL
RETURNING id, started_at`
run := domain.BillingRun{TenantID: tenantID, BillingPeriod: billingPeriod, Status: domain.BillingRunStatusRunning}
err := r.pool.QueryRow(ctx, q, tenantID, billingPeriod).Scan(&run.ID, &run.StartedAt)
return run, err
}
func (r *BillingRunRepository) RecordFlatSuccess(ctx context.Context, runID uuid.UUID) error {
_, err := r.pool.Exec(ctx, `UPDATE billing_runs SET flats_total = flats_total + 1, flats_succeeded = flats_succeeded + 1 WHERE id = $1`, runID)
return err
}
func (r *BillingRunRepository) RecordFlatFailure(ctx context.Context, runID, flatID uuid.UUID, reason string) error {
if _, err := r.pool.Exec(ctx, `UPDATE billing_runs SET flats_total = flats_total + 1, flats_failed = flats_failed + 1 WHERE id = $1`, runID); err != nil {
return err
}
_, err := r.pool.Exec(ctx, `INSERT INTO billing_run_failures (billing_run_id, flat_id, reason) VALUES ($1, $2, $3)`, runID, flatID, reason)
return err
}
func (r *BillingRunRepository) Complete(ctx context.Context, runID uuid.UUID, status domain.BillingRunStatus) error {
_, err := r.pool.Exec(ctx, `UPDATE billing_runs SET status = $1, completed_at = now() WHERE id = $2`, status, runID)
return err
}flats_total is incremented alongside flats_succeeded or flats_failed, never separately — there is no code path where a flat is counted in the total without also being counted as either a success or a failure.
Reading Flats and Owners Over HTTP
The billing run needs two facts this service does not own: which flats exist and their area (flat-service), and who currently occupies each one (resident-service). The first step below adds a narrow internal current-owner read contract because Resident Service intentionally exposed only resident-facing ownership commands. Resident Service's ownership lesson owns the cross-service HTTP-client pattern; Maintenance reuses it for read-only billing inputs.
func (h *OwnershipHandler) CurrentOwner(c *gin.Context) {
tenantID := tenantctx.MustFromContext(c.Request.Context())
flatID, err := uuid.Parse(c.Param("flat_id"))
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid flat id"})
return
}
owner, err := h.service.CurrentOwner(c.Request.Context(), tenantID, flatID)
if errors.Is(err, domain.ErrNoCurrentOwner) {
c.JSON(http.StatusNotFound, gin.H{"error": "no current owner"})
return
}
if err != nil {
c.Error(err)
return
}
c.JSON(http.StatusOK, gin.H{"resident_id": owner.ResidentID})
}
// Mounted only on the cluster-internal router; tenant context is still
// required and the repository query filters by tenant_id and flat_id.
internal.GET("/internal/flats/:flat_id/current-owner", ownershipHandler.CurrentOwner)This is an explicit cross-service contract introduced when the first consumer needs it, not a hidden assumption. Resident remains the only service that reads its ownership table directly.
package tenantctx
import (
"context"
"github.com/google/uuid"
)
type key struct{}
func WithTenant(ctx context.Context, tenantID uuid.UUID) context.Context {
return context.WithValue(ctx, key{}, tenantID)
}
func FromContext(ctx context.Context) (uuid.UUID, bool) {
tenantID, ok := ctx.Value(key{}).(uuid.UUID)
return tenantID, ok
}
func MustFromContext(ctx context.Context) uuid.UUID {
tenantID, ok := FromContext(ctx)
if !ok {
panic("tenantctx: tenant middleware did not populate context")
}
return tenantID
}Transport middleware validates the tenant claim before calling WithTenant. MustFromContext is reserved for code already behind that middleware, turning missing scope into a loud programming defect rather than an unscoped query.
package client
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"time"
"github.com/google/uuid"
"github.com/hoa-platform/backend/pkg/tenantctx"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
)
var ErrFlatServiceUnavailable = errors.New("flat_client: flat-service unavailable")
type FlatClient struct {
baseURL string
http *http.Client
}
func NewFlatClient(baseURL string) *FlatClient {
return &FlatClient{baseURL: baseURL, http: &http.Client{Timeout: 3 * time.Second}}
}
type flatDTO struct {
ID uuid.UUID
AreaSqft float64
Status string
}
// ListActive returns every occupied flat for the tenant. flat-service has
// no server-side status filter yet, so this filters client-side after
// fetching the full list — acceptable for the society sizes this
// platform targets today, and called out explicitly in this lesson's
// exercise as the first thing to fix if a tenant's flat count grows large.
func (c *FlatClient) ListActive(ctx context.Context) ([]domain.FlatSummary, error) {
tenantID := tenantctx.MustFromContext(ctx)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/flats", nil)
if err != nil {
return nil, err
}
req.Header.Set("X-Tenant-Id", tenantID.String())
resp, err := c.http.Do(req)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrFlatServiceUnavailable, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%w: status %d", ErrFlatServiceUnavailable, resp.StatusCode)
}
var flats []flatDTO
if err := json.NewDecoder(resp.Body).Decode(&flats); err != nil {
return nil, fmt.Errorf("%w: decoding response: %v", ErrFlatServiceUnavailable, err)
}
out := make([]domain.FlatSummary, 0, len(flats))
for _, f := range flats {
if f.Status == "occupied" {
out = append(out, domain.FlatSummary{ID: f.ID, AreaSqft: f.AreaSqft})
}
}
return out, nil
}flatDTO has no json tags because flat-service's Flat struct (Module 06) has none either — its fields serialize under their exact Go names, so this DTO's field names must match verbatim.
type OwnerResolver interface {
CurrentOwner(ctx context.Context, flatID uuid.UUID) (uuid.UUID, error)
}
var (
ErrNoCurrentOwner = errors.New("resident_client: flat has no current owner")
ErrResidentServiceUnavailable = errors.New("resident_client: resident-service unavailable")
)
func (c *ResidentClient) CurrentOwner(ctx context.Context, flatID uuid.UUID) (uuid.UUID, error) {
// Request construction, tenant header propagation, timeout, and decoding follow
// Resident Service's FlatClient pattern. The mapping below is the new contract.
switch resp.StatusCode {
case http.StatusOK:
return dto.ResidentID, nil
case http.StatusNotFound:
return uuid.Nil, ErrNoCurrentOwner
default:
return uuid.Nil, fmt.Errorf("%w: status %d", ErrResidentServiceUnavailable, resp.StatusCode)
}
}Resident Service's ownership lesson owns request construction and unavailable-vs-not-found mapping. Maintenance only adds the current-owner meaning: 404 is a billable flat with no owner right now, not a missing flat.
The Billing Runner
package service
import (
"context"
"log"
"time"
"github.com/google/uuid"
"github.com/hoa-platform/backend/pkg/tenantctx"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
)
// The runner depends on narrow interfaces, not concrete client or
// repository types, so unit tests substitute hand-rolled fakes and only
// the integration test in the Testing lesson wires real Postgres and HTTP
// implementations together.
type FlatLister interface {
ListActive(ctx context.Context) ([]domain.FlatSummary, error)
}
type OwnerResolver interface {
CurrentOwner(ctx context.Context, flatID uuid.UUID) (uuid.UUID, error)
}
type ChargeVersionSource interface {
ListActiveOn(ctx context.Context, tenantID uuid.UUID, billingDate time.Time) ([]domain.ActiveCharge, error)
}
type BillingRunRepo interface {
StartRun(ctx context.Context, tenantID uuid.UUID, billingPeriod time.Time) (domain.BillingRun, error)
RecordFlatSuccess(ctx context.Context, runID uuid.UUID) error
RecordFlatFailure(ctx context.Context, runID, flatID uuid.UUID, reason string) error
Complete(ctx context.Context, runID uuid.UUID, status domain.BillingRunStatus) error
}
type InvoiceRepo interface {
// CreateIfAbsent persists an invoice and its line items in one
// transaction and reports whether it actually inserted a new row —
// created is false when a prior run already billed this flat for this
// period, which the runner treats as a success, not a failure.
CreateIfAbsent(ctx context.Context, invoice domain.Invoice, lineItems []domain.InvoiceLineItem) (created bool, err error)
}
type BillingRunner struct {
flats FlatLister
owners OwnerResolver
charges ChargeVersionSource
runs BillingRunRepo
invoices InvoiceRepo
}
func NewBillingRunner(flats FlatLister, owners OwnerResolver, charges ChargeVersionSource, runs BillingRunRepo, invoices InvoiceRepo) *BillingRunner {
return &BillingRunner{flats: flats, owners: owners, charges: charges, runs: runs, invoices: invoices}
}
// GenerateForPeriod bills every active flat for the tenant found in ctx.
// A failure on one flat is recorded and the loop continues — the run only
// stops early if listing flats or charges (both tenant-wide prerequisites)
// fails, since there is nothing meaningful left to bill without them.
func (r *BillingRunner) GenerateForPeriod(ctx context.Context, billingPeriod time.Time) (domain.BillingRun, error) {
tenantID := tenantctx.MustFromContext(ctx)
run, err := r.runs.StartRun(ctx, tenantID, billingPeriod)
if err != nil {
return domain.BillingRun{}, err
}
flats, err := r.flats.ListActive(ctx)
if err != nil {
_ = r.runs.Complete(ctx, run.ID, domain.BillingRunStatusCompletedWithErrors)
return run, err
}
charges, err := r.charges.ListActiveOn(ctx, tenantID, billingPeriod)
if err != nil {
_ = r.runs.Complete(ctx, run.ID, domain.BillingRunStatusCompletedWithErrors)
return run, err
}
anyFailed := false
for _, flat := range flats {
if err := r.processFlat(ctx, run.ID, tenantID, flat, billingPeriod, charges); err != nil {
anyFailed = true
log.Printf("billing run %s: flat %s failed: %v", run.ID, flat.ID, err)
if recErr := r.runs.RecordFlatFailure(ctx, run.ID, flat.ID, err.Error()); recErr != nil {
log.Printf("billing run %s: failed to record failure for flat %s: %v", run.ID, flat.ID, recErr)
}
continue
}
if err := r.runs.RecordFlatSuccess(ctx, run.ID); err != nil {
log.Printf("billing run %s: failed to record success for flat %s: %v", run.ID, flat.ID, err)
}
}
finalStatus := domain.BillingRunStatusCompleted
if anyFailed {
finalStatus = domain.BillingRunStatusCompletedWithErrors
}
return run, r.runs.Complete(ctx, run.ID, finalStatus)
}
func (r *BillingRunner) processFlat(ctx context.Context, runID, tenantID uuid.UUID, flat domain.FlatSummary, billingPeriod time.Time, charges []domain.ActiveCharge) error {
residentID, err := r.owners.CurrentOwner(ctx, flat.ID)
if err != nil {
return err
}
invoice, lineItems := GenerateMonthlyInvoice(tenantID, flat.ID, residentID, billingPeriod, flat.AreaSqft, charges)
_, err = r.invoices.CreateIfAbsent(ctx, invoice, lineItems)
return err
}processFlat's only responsibility is one flat; every error it returns is caught by the loop in GenerateForPeriod and recorded against that one flat, never propagated in a way that would abort billing for the rest of the society.
package postgres
import (
"context"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/hoa-platform/backend/services/maintenance-service/internal/domain"
)
type InvoiceRepository struct {
pool *pgxpool.Pool
}
func NewInvoiceRepository(pool *pgxpool.Pool) *InvoiceRepository {
return &InvoiceRepository{pool: pool}
}
// CreateIfAbsent inserts the invoice and every line item in one
// transaction. The ON CONFLICT DO NOTHING on the (tenant_id, flat_id,
// billing_period) unique index is the actual once-only guarantee — if a
// prior run already billed this flat this period, the INSERT ... RETURNING
// returns zero rows, no line items are inserted, and created is false.
func (r *InvoiceRepository) CreateIfAbsent(ctx context.Context, invoice domain.Invoice, lineItems []domain.InvoiceLineItem) (bool, error) {
tx, err := r.pool.Begin(ctx)
if err != nil {
return false, err
}
defer tx.Rollback(ctx)
var insertedID = invoice.ID
err = tx.QueryRow(ctx, `
INSERT INTO invoices (id, tenant_id, flat_id, resident_id, billing_period, status, total_paise)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (tenant_id, flat_id, billing_period) DO NOTHING
RETURNING id`,
invoice.ID, invoice.TenantID, invoice.FlatID, invoice.ResidentID, invoice.BillingPeriod, invoice.Status, invoice.TotalPaise,
).Scan(&insertedID)
if err == pgx.ErrNoRows {
return false, tx.Commit(ctx)
}
if err != nil {
return false, err
}
for _, li := range lineItems {
if _, err := tx.Exec(ctx, `
INSERT INTO invoice_line_items (id, invoice_id, charge_type_id, description, amount_paise)
VALUES ($1, $2, $3, $4, $5)`,
li.ID, li.InvoiceID, li.ChargeTypeID, li.Description, li.AmountPaise,
); err != nil {
return false, err
}
}
if err := tx.Commit(ctx); err != nil {
return false, err
}
return true, nil
}The pgx.ErrNoRows branch commits the (empty) transaction rather than rolling it back — there is nothing to undo, and committing keeps the connection's transaction state clean for the pool.
package main
import (
"context"
"log"
"os"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/hoa-platform/backend/pkg/tenantctx"
"github.com/hoa-platform/backend/services/maintenance-service/internal/client"
"github.com/hoa-platform/backend/services/maintenance-service/internal/repository/postgres"
"github.com/hoa-platform/backend/services/maintenance-service/internal/service"
)
// billing-runner is a one-shot batch job, not a server: it runs to
// completion and exits. It is scoped to a single tenant per invocation via
// BILLING_TENANT_ID — running it for every tenant in the platform is a
// simplification left as this lesson's exercise, and is revisited again
// as a scheduling concern in the Deployment lesson.
func main() {
pool, err := pgxpool.New(context.Background(), os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatalf("connecting to postgres: %v", err)
}
defer pool.Close()
tenantID, err := uuid.Parse(os.Getenv("BILLING_TENANT_ID"))
if err != nil {
log.Fatalf("BILLING_TENANT_ID must be a valid UUID: %v", err)
}
billingPeriod := resolveBillingPeriod()
ctx := tenantctx.WithTenantID(context.Background(), tenantID)
flatClient := client.NewFlatClient(os.Getenv("FLAT_SERVICE_URL"))
residentClient := client.NewResidentClient(os.Getenv("RESIDENT_SERVICE_URL"))
chargeVersionRepo := postgres.NewChargeVersionRepository(pool)
billingRunRepo := postgres.NewBillingRunRepository(pool)
invoiceRepo := postgres.NewInvoiceRepository(pool)
runner := service.NewBillingRunner(flatClient, residentClient, chargeVersionRepo, billingRunRepo, invoiceRepo)
run, err := runner.GenerateForPeriod(ctx, billingPeriod)
if err != nil {
log.Fatalf("billing run failed to start: %v", err)
}
log.Printf("billing run %s completed: status=%s total=%d succeeded=%d failed=%d",
run.ID, run.Status, run.FlatsTotal, run.FlatsSucceeded, run.FlatsFailed)
}
// resolveBillingPeriod defaults to the first of the current month unless
// BILLING_PERIOD (YYYY-MM-DD) is set, which the exercise and testing
// lessons both use to generate invoices for arbitrary past or future periods.
func resolveBillingPeriod() time.Time {
if v := os.Getenv("BILLING_PERIOD"); v != "" {
t, err := time.Parse("2006-01-02", v)
if err != nil {
log.Fatalf("BILLING_PERIOD must be YYYY-MM-DD: %v", err)
}
return t
}
now := time.Now().UTC()
return time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC)
}ChargeVersionSource, FlatLister, and OwnerResolver are satisfied structurally by postgres.ChargeVersionRepository, client.FlatClient, and client.ResidentClient respectively — main.go is the only place that wires concrete types into the runner's narrow interfaces.
Rerunning a Period Without Duplicate Invoices
| Failure | Run behavior | Invariant preserved |
|---|---|---|
| one flat has no current owner | record one billing_run_failures row and continue | other flats still receive invoices |
| same period is run twice | StartRun resets counters; CreateIfAbsent skips existing invoices | one invoice per flat per period |
| flat-service is unavailable | mark the run completed_with_errors and stop before per-flat work | no partial tenant-wide prerequisite is trusted |
Applied exercise
Bill every tenant in one invocation
cmd/billing-runner currently bills exactly one tenant per process invocation via BILLING_TENANT_ID — acceptable for a handful of societies, not for a platform with hundreds.
- Add a domain.TenantRepository-style interface (or reuse an existing tenant listing mechanism) to fetch all active tenant IDs.
- Change cmd/billing-runner/main.go to loop over every tenant, calling GenerateForPeriod once per tenant with that tenant's ID placed in ctx.
- Ensure one tenant's billing failure (e.g., its flat-service being unreachable) does not stop the loop from attempting the remaining tenants.
- Add a summary log line at the end reporting how many tenants completed successfully versus with errors.
Deliverable
An updated cmd/billing-runner/main.go that bills every tenant in one invocation, isolates per-tenant failures, and logs a final cross-tenant summary.
Completion checks
- A simulated failure for one tenant (e.g., an invalid FLAT_SERVICE_URL override for that tenant only) does not prevent other tenants' runs from completing.
- go build ./services/maintenance-service/... succeeds.
- BILLING_TENANT_ID is no longer required for the multi-tenant path (single-tenant mode may remain as a fallback if BILLING_TENANT_ID is still set).
Why does BillingRunner.GenerateForPeriod continue to the next flat when processFlat returns an error for one flat, instead of returning immediately?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.