Course outline

01 · Phase 1 — Project Foundation16/16

02 · Phase 2 — Organization Service20/20

03 · Phase 3 — User Service10/10

04 · Phase 4 — Authentication Service10/10

05 · Phase 5 — API Gateway9/9

06 · Phase 6 — Flat Service6/6

07 · Phase 7 — Resident Service7/7

08 · Phase 8 — Maintenance Service7/7

09 · Phase 9 — Payment Service7/7

10 · Phase 10 — Complaint Service7/7

11 · Phase 11 — Notice Service7/7

12 · Phase 12 — Visitor Service6/6

13 · Phase 13 — Staff Service6/6

14 · Phase 14 — Notification Service7/7

15 · Phase 15 — Redis Integration6/6

16 · Phase 16 — Kafka Integration8/8

17 · Phase 17 — Observability7/7

18 · Phase 18 — Performance Engineering6/6

19 · Phase 19 — Production Deployment11/11

20 · Phase 20 — Production Readiness8/8

Stage 7 · Master

Phase 6 — Flat Service

Monitoring

Counting flats, not just requests

14 min readHOA Backend Engineering CourseMaster

What Generic HTTP Metrics Miss

Organization Service's metrics lesson owns the private registry, route-template HTTP histogram, and /metrics wiring; User Service already re-applied that contract. Flat Service adds a domain signal for abnormal inventory creation, labeled by tenant_id because the operator needs to know which HOA is creating units unusually fast.

Business Counter for Inventory Creation

Terminal — flat-service

$mkdir -p services/flat-service/internal/observability

$touch services/flat-service/internal/observability/metrics.go

Create services/flat-service/internal/observability/metrics.go before adding the code

$mkdir -p services/flat-service/internal/observability && touch services/flat-service/internal/observability/metrics.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/flat-service/internal/observability/metrics.go (excerpt — flat business signal)

package observability

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

var FlatsCreatedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
	Name: "flat_service_flats_created_total",
	Help: "Total flats created, labeled by tenant.",
}, []string{"tenant_id"})

tenant_id is intentionally present on this single counter: the time series count grows with tenants only for the inventory action an operator investigates.

Create services/flat-service/internal/handler/http/flat_handler.go before adding the code

$mkdir -p services/flat-service/internal/handler/http && touch services/flat-service/internal/handler/http/flat_handler.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/flat-service/internal/handler/http/flat_handler.go (excerpt — Create instrumented)

func (h *FlatHandler) Create(c *gin.Context) {
	tenantID := tenantctx.MustFromContext(c.Request.Context())
	// ... binding and validation unchanged ...

	flat, err := h.svc.Create(c.Request.Context(), domain.Flat{ /* ... */ })
	if err != nil {
		// ... error handling unchanged ...
		return
	}

	observability.FlatsCreatedTotal.WithLabelValues(tenantID.String()).Inc()
	c.JSON(http.StatusCreated, flat)
}

Structured Logs Carry the Same Identifiers

Create services/flat-service/internal/handler/http/flat_handler.go before adding the code

$mkdir -p services/flat-service/internal/handler/http && touch services/flat-service/internal/handler/http/flat_handler.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/flat-service/internal/handler/http/flat_handler.go (excerpt — logging on error)

case err != nil:
	log.Error().
		Str("tenant_id", tenantID.String()).
		Str("building_id", req.BuildingID.String()).
		Err(err).
		Msg("could not create flat")
	c.JSON(http.StatusInternalServerError, gin.H{"error": "could not create flat"})

Every log line an operator reads after an incident should carry tenant_id — without it, 'flat-service errored for someone' is not actionable.

Inventory Counter Drill

Terminal — confirm the counter moves

$curl -s localhost:8081/metrics | grep flat_service_flats_created_total

$curl -s -H 'X-Tenant-Id: 11111111-1111-1111-1111-111111111111' -X POST localhost:8081/flats -d '{"building_id":"$BID","unit_number":"A-104","floor_number":1,"area_sqft":600}'

$curl -s localhost:8081/metrics | grep flat_service_flats_created_total

// counter for tenant 11111111... incremented by exactly 1

Deliberately not solved here

This counter does not explain why inventory is being created. It gives the on-call engineer the tenant and rate needed to decide whether to inspect an import job, a retry loop, or a legitimate onboarding event.

Applied exercise

Alert on a tenant hammering the create endpoint

30 min

One tenant's integration accidentally retries every create request on any non-2xx response, including 409 Conflict, generating 40x the normal request volume for that tenant alone. Nothing today would page anyone about it.

Write a PromQL expression that computes rate(flat_service_flats_created_total[5m]) per tenant_id.
Define an alerting rule that fires when one tenant's creation rate exceeds 10x the recent tenant median, not a fixed absolute threshold.
Document, in one paragraph, why an absolute per-tenant threshold is the wrong choice for a multi-tenant platform with HOAs of very different sizes.

Deliverable

A PromQL alerting rule (as YAML) plus the explanatory paragraph.

Completion checks

The rule is relative (ratio to median or similar), not a fixed absolute number.
The rule uses flat_service_flats_created_total rather than the generic HTTP histogram.
The explanation names the specific failure mode an absolute threshold would cause: false alerts for large tenants, or missed alerts for small ones.

Quiz & Practice

Question 1 / 2

Why is tenant_id a label on FlatsCreatedTotal but not on RequestDuration?

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Course outline

01 · Phase 1 — Project Foundation16/16

02 · Phase 2 — Organization Service20/20

03 · Phase 3 — User Service10/10

04 · Phase 4 — Authentication Service10/10

05 · Phase 5 — API Gateway9/9

06 · Phase 6 — Flat Service6/6

07 · Phase 7 — Resident Service7/7

08 · Phase 8 — Maintenance Service7/7

09 · Phase 9 — Payment Service7/7

10 · Phase 10 — Complaint Service7/7

11 · Phase 11 — Notice Service7/7

12 · Phase 12 — Visitor Service6/6

13 · Phase 13 — Staff Service6/6

14 · Phase 14 — Notification Service7/7

15 · Phase 15 — Redis Integration6/6

16 · Phase 16 — Kafka Integration8/8

17 · Phase 17 — Observability7/7

18 · Phase 18 — Performance Engineering6/6

19 · Phase 19 — Production Deployment11/11

20 · Phase 20 — Production Readiness8/8

Stage 7 · Master

Phase 6 — Flat Service

Monitoring

Counting flats, not just requests

14 min readHOA Backend Engineering CourseMaster

What Generic HTTP Metrics Miss

Business Counter for Inventory Creation

Terminal — flat-service

$mkdir -p services/flat-service/internal/observability

$touch services/flat-service/internal/observability/metrics.go

Create services/flat-service/internal/observability/metrics.go before adding the code

$mkdir -p services/flat-service/internal/observability && touch services/flat-service/internal/observability/metrics.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/flat-service/internal/observability/metrics.go (excerpt — flat business signal)

package observability

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

var FlatsCreatedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
	Name: "flat_service_flats_created_total",
	Help: "Total flats created, labeled by tenant.",
}, []string{"tenant_id"})

tenant_id is intentionally present on this single counter: the time series count grows with tenants only for the inventory action an operator investigates.

Create services/flat-service/internal/handler/http/flat_handler.go before adding the code

$mkdir -p services/flat-service/internal/handler/http && touch services/flat-service/internal/handler/http/flat_handler.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/flat-service/internal/handler/http/flat_handler.go (excerpt — Create instrumented)

func (h *FlatHandler) Create(c *gin.Context) {
	tenantID := tenantctx.MustFromContext(c.Request.Context())
	// ... binding and validation unchanged ...

	flat, err := h.svc.Create(c.Request.Context(), domain.Flat{ /* ... */ })
	if err != nil {
		// ... error handling unchanged ...
		return
	}

	observability.FlatsCreatedTotal.WithLabelValues(tenantID.String()).Inc()
	c.JSON(http.StatusCreated, flat)
}

Structured Logs Carry the Same Identifiers

Create services/flat-service/internal/handler/http/flat_handler.go before adding the code

$mkdir -p services/flat-service/internal/handler/http && touch services/flat-service/internal/handler/http/flat_handler.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/flat-service/internal/handler/http/flat_handler.go (excerpt — logging on error)

case err != nil:
	log.Error().
		Str("tenant_id", tenantID.String()).
		Str("building_id", req.BuildingID.String()).
		Err(err).
		Msg("could not create flat")
	c.JSON(http.StatusInternalServerError, gin.H{"error": "could not create flat"})

Every log line an operator reads after an incident should carry tenant_id — without it, 'flat-service errored for someone' is not actionable.

Inventory Counter Drill

Terminal — confirm the counter moves

$curl -s localhost:8081/metrics | grep flat_service_flats_created_total

$curl -s -H 'X-Tenant-Id: 11111111-1111-1111-1111-111111111111' -X POST localhost:8081/flats -d '{"building_id":"$BID","unit_number":"A-104","floor_number":1,"area_sqft":600}'

$curl -s localhost:8081/metrics | grep flat_service_flats_created_total

// counter for tenant 11111111... incremented by exactly 1

Deliberately not solved here

Applied exercise

Alert on a tenant hammering the create endpoint

30 min

Write a PromQL expression that computes rate(flat_service_flats_created_total[5m]) per tenant_id.
Define an alerting rule that fires when one tenant's creation rate exceeds 10x the recent tenant median, not a fixed absolute threshold.
Document, in one paragraph, why an absolute per-tenant threshold is the wrong choice for a multi-tenant platform with HOAs of very different sizes.

Deliverable

A PromQL alerting rule (as YAML) plus the explanatory paragraph.

Completion checks

The rule is relative (ratio to median or similar), not a fixed absolute number.
The rule uses flat_service_flats_created_total rather than the generic HTTP histogram.
The explanation names the specific failure mode an absolute threshold would cause: false alerts for large tenants, or missed alerts for small ones.

Quiz & Practice

Question 1 / 2

Why is tenant_id a label on FlatsCreatedTotal but not on RequestDuration?

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.