Course outline

01 · Phase 1 — Project Foundation16/16

02 · Phase 2 — Organization Service20/20

03 · Phase 3 — User Service10/10

04 · Phase 4 — Authentication Service10/10

05 · Phase 5 — API Gateway9/9

06 · Phase 6 — Flat Service6/6

07 · Phase 7 — Resident Service7/7

08 · Phase 8 — Maintenance Service7/7

09 · Phase 9 — Payment Service7/7

10 · Phase 10 — Complaint Service7/7

11 · Phase 11 — Notice Service7/7

12 · Phase 12 — Visitor Service6/6

13 · Phase 13 — Staff Service6/6

14 · Phase 14 — Notification Service7/7

15 · Phase 15 — Redis Integration6/6

16 · Phase 16 — Kafka Integration8/8

17 · Phase 17 — Observability7/7

18 · Phase 18 — Performance Engineering6/6

19 · Phase 19 — Production Deployment11/11

20 · Phase 20 — Production Readiness8/8

Stage 7 · Master

Phase 7 — Resident Service

Monitoring

Watching a dependency you don't own

14 min readHOA Backend Engineering CourseMaster

The Metric This Service Actually Needs

resident-service's riskiest failure mode is not its own bug — it is flat-service becoming slow or unreachable and every ownership or occupancy request silently degrading. Generic request-duration metrics would show resident-service itself as 'slow,' which is misleading; the real signal is how often calls to FlatClient fail or time out.

Terminal — resident-service

$mkdir -p services/resident-service/internal/observability

$touch services/resident-service/internal/observability/metrics.go

Create services/resident-service/internal/observability/metrics.go before adding the code

$mkdir -p services/resident-service/internal/observability && touch services/resident-service/internal/observability/metrics.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/resident-service/internal/observability/metrics.go

package observability

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

var FlatServiceCallsTotal = promauto.NewCounterVec(
	prometheus.CounterOpts{
		Name: "resident_service_flat_client_calls_total",
		Help: "Calls from resident-service to flat-service, labeled by outcome",
	},
	[]string{"outcome"},
)

Flat Service already implemented HTTP middleware, so Resident does not copy it. This package adds only the new cross-service signal. outcome is bounded to ok, not_found, and unavailable; resident_id, flat_id, and household_id are deliberately excluded because entity identifiers create unbounded cardinality.

Create services/resident-service/internal/client/flat_client.go before adding the code

$mkdir -p services/resident-service/internal/client && touch services/resident-service/internal/client/flat_client.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/resident-service/internal/client/flat_client.go (modify — four counter increments, no other change)

func (c *FlatClient) Exists(ctx context.Context, tenantID, flatID uuid.UUID) (bool, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet,
		fmt.Sprintf("%s/flats/%s", c.baseURL, flatID), nil)
	if err != nil {
		return false, err
	}
	req.Header.Set("X-Tenant-Id", tenantID.String())

	resp, err := c.http.Do(req)
	if err != nil {
		observability.FlatServiceCallsTotal.WithLabelValues("unavailable").Inc()
		return false, ErrFlatServiceUnavailable
	}
	defer resp.Body.Close()

	switch resp.StatusCode {
	case http.StatusOK:
		observability.FlatServiceCallsTotal.WithLabelValues("ok").Inc()
		return true, nil
	case http.StatusNotFound:
		observability.FlatServiceCallsTotal.WithLabelValues("not_found").Inc()
		return false, nil
	default:
		observability.FlatServiceCallsTotal.WithLabelValues("unavailable").Inc()
		var body map[string]any
		_ = json.NewDecoder(resp.Body).Decode(&body)
		return false, fmt.Errorf("flat_client: unexpected status %d: %v", resp.StatusCode, body)
	}
}

Structured Logs for a Cross-Service Failure

Create services/resident-service/internal/service/ownership_service.go before adding the code

$mkdir -p services/resident-service/internal/service && touch services/resident-service/internal/service/ownership_service.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/resident-service/internal/service/ownership_service.go (excerpt — logging on downstream failure)

if !exists {
	log.Warn().
		Str("tenant_id", o.TenantID.String()).
		Str("flat_id", o.FlatID.String()).
		Str("resident_id", o.ResidentID.String()).
		Msg("ownership creation rejected: flat not found or unreachable")
	return domain.FlatOwnership{}, domain.ErrFlatNotFoundForOwnership
}

Logging flat_id alongside tenant_id and resident_id here means an on-call engineer can immediately cross-reference flat-service's own logs for the same flat_id during the same time window.

Separating Dependency Failure From Latency

Terminal — confirm the outcome counter distinguishes failure modes

$curl -s localhost:8082/metrics | grep resident_service_flat_client_calls_total

$kill %1

// stop flat-service to simulate an outage

$curl -s -H 'X-Tenant-Id: 11111111-1111-1111-1111-111111111111' -X POST localhost:8082/ownerships -d '{"flat_id":"$FLAT_ID","resident_id":"$RID","share_percentage":100,"deed_reference":"DEED-003","effective_from":"2024-01-01"}'

$curl -s localhost:8082/metrics | grep resident_service_flat_client_calls_total

// outcome="unavailable" incremented, distinct from outcome="not_found"

Applied exercise

Alert on flat-service degradation from resident-service's perspective

30 min

flat-service's own monitoring (module 06) would eventually catch an outage, but resident-service is often the first place a degraded flat-service becomes visible to actual users, since ownership and occupancy requests fail immediately.

Write a PromQL expression for the ratio of outcome="unavailable" to total resident_service_flat_client_calls_total over a 5-minute window.
Define an alert that fires when that ratio exceeds 10% for more than 2 minutes, distinct from any alert flat-service's own team owns.
Explain in one paragraph why this alert belongs to resident-service's on-call rotation even though the root cause is in a different service.

Deliverable

A PromQL alerting rule (as YAML) plus the explanatory paragraph.

Completion checks

The rule is a ratio, not a raw count, so it scales with traffic volume.
The rule only fires on sustained degradation (2+ minutes), not a single transient blip.
The explanation distinguishes 'who gets paged' from 'who fixes the root cause.'

Quiz & Practice

Question 1 / 2

Why does FlatServiceCallsTotal use a bounded 'outcome' label (ok, not_found, unavailable) instead of labeling by the raw HTTP status code returned?

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Course outline

01 · Phase 1 — Project Foundation16/16

02 · Phase 2 — Organization Service20/20

03 · Phase 3 — User Service10/10

04 · Phase 4 — Authentication Service10/10

05 · Phase 5 — API Gateway9/9

06 · Phase 6 — Flat Service6/6

07 · Phase 7 — Resident Service7/7

08 · Phase 8 — Maintenance Service7/7

09 · Phase 9 — Payment Service7/7

10 · Phase 10 — Complaint Service7/7

11 · Phase 11 — Notice Service7/7

12 · Phase 12 — Visitor Service6/6

13 · Phase 13 — Staff Service6/6

14 · Phase 14 — Notification Service7/7

15 · Phase 15 — Redis Integration6/6

16 · Phase 16 — Kafka Integration8/8

17 · Phase 17 — Observability7/7

18 · Phase 18 — Performance Engineering6/6

19 · Phase 19 — Production Deployment11/11

20 · Phase 20 — Production Readiness8/8

Stage 7 · Master

Phase 7 — Resident Service

Monitoring

Watching a dependency you don't own

14 min readHOA Backend Engineering CourseMaster

The Metric This Service Actually Needs

Terminal — resident-service

$mkdir -p services/resident-service/internal/observability

$touch services/resident-service/internal/observability/metrics.go

Create services/resident-service/internal/observability/metrics.go before adding the code

$mkdir -p services/resident-service/internal/observability && touch services/resident-service/internal/observability/metrics.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/resident-service/internal/observability/metrics.go

package observability

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

var FlatServiceCallsTotal = promauto.NewCounterVec(
	prometheus.CounterOpts{
		Name: "resident_service_flat_client_calls_total",
		Help: "Calls from resident-service to flat-service, labeled by outcome",
	},
	[]string{"outcome"},
)

Create services/resident-service/internal/client/flat_client.go before adding the code

$mkdir -p services/resident-service/internal/client && touch services/resident-service/internal/client/flat_client.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/resident-service/internal/client/flat_client.go (modify — four counter increments, no other change)

func (c *FlatClient) Exists(ctx context.Context, tenantID, flatID uuid.UUID) (bool, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet,
		fmt.Sprintf("%s/flats/%s", c.baseURL, flatID), nil)
	if err != nil {
		return false, err
	}
	req.Header.Set("X-Tenant-Id", tenantID.String())

	resp, err := c.http.Do(req)
	if err != nil {
		observability.FlatServiceCallsTotal.WithLabelValues("unavailable").Inc()
		return false, ErrFlatServiceUnavailable
	}
	defer resp.Body.Close()

	switch resp.StatusCode {
	case http.StatusOK:
		observability.FlatServiceCallsTotal.WithLabelValues("ok").Inc()
		return true, nil
	case http.StatusNotFound:
		observability.FlatServiceCallsTotal.WithLabelValues("not_found").Inc()
		return false, nil
	default:
		observability.FlatServiceCallsTotal.WithLabelValues("unavailable").Inc()
		var body map[string]any
		_ = json.NewDecoder(resp.Body).Decode(&body)
		return false, fmt.Errorf("flat_client: unexpected status %d: %v", resp.StatusCode, body)
	}
}

Structured Logs for a Cross-Service Failure

Create services/resident-service/internal/service/ownership_service.go before adding the code

$mkdir -p services/resident-service/internal/service && touch services/resident-service/internal/service/ownership_service.go

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

services/resident-service/internal/service/ownership_service.go (excerpt — logging on downstream failure)

if !exists {
	log.Warn().
		Str("tenant_id", o.TenantID.String()).
		Str("flat_id", o.FlatID.String()).
		Str("resident_id", o.ResidentID.String()).
		Msg("ownership creation rejected: flat not found or unreachable")
	return domain.FlatOwnership{}, domain.ErrFlatNotFoundForOwnership
}

Logging flat_id alongside tenant_id and resident_id here means an on-call engineer can immediately cross-reference flat-service's own logs for the same flat_id during the same time window.

Separating Dependency Failure From Latency

Terminal — confirm the outcome counter distinguishes failure modes

$curl -s localhost:8082/metrics | grep resident_service_flat_client_calls_total

$kill %1

// stop flat-service to simulate an outage

$curl -s localhost:8082/metrics | grep resident_service_flat_client_calls_total

// outcome="unavailable" incremented, distinct from outcome="not_found"

Applied exercise

Alert on flat-service degradation from resident-service's perspective

30 min

Write a PromQL expression for the ratio of outcome="unavailable" to total resident_service_flat_client_calls_total over a 5-minute window.
Define an alert that fires when that ratio exceeds 10% for more than 2 minutes, distinct from any alert flat-service's own team owns.
Explain in one paragraph why this alert belongs to resident-service's on-call rotation even though the root cause is in a different service.

Deliverable

A PromQL alerting rule (as YAML) plus the explanatory paragraph.

Completion checks

The rule is a ratio, not a raw count, so it scales with traffic volume.
The rule only fires on sustained degradation (2+ minutes), not a single transient blip.
The explanation distinguishes 'who gets paged' from 'who fixes the root cause.'

Quiz & Practice

Question 1 / 2

Why does FlatServiceCallsTotal use a bounded 'outcome' label (ok, not_found, unavailable) instead of labeling by the raw HTTP status code returned?

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.