Course outline

01 · Phase 1 — Project Foundation16/16

02 · Phase 2 — Organization Service20/20

03 · Phase 3 — User Service10/10

04 · Phase 4 — Authentication Service10/10

05 · Phase 5 — API Gateway9/9

06 · Phase 6 — Flat Service6/6

07 · Phase 7 — Resident Service7/7

08 · Phase 8 — Maintenance Service7/7

09 · Phase 9 — Payment Service7/7

10 · Phase 10 — Complaint Service7/7

11 · Phase 11 — Notice Service7/7

12 · Phase 12 — Visitor Service6/6

13 · Phase 13 — Staff Service6/6

14 · Phase 14 — Notification Service7/7

15 · Phase 15 — Redis Integration6/6

16 · Phase 16 — Kafka Integration8/8

17 · Phase 17 — Observability7/7

18 · Phase 18 — Performance Engineering6/6

19 · Phase 19 — Production Deployment11/11

20 · Phase 20 — Production Readiness8/8

Stage 7 · Master

Phase 19 — Production Deployment

Kubernetes

Write a Namespace, a migration Job, and a Deployment/Service pair that reference an image by immutable digest, enforce the non-root read-only container Lesson 1 already proved works, and never race a migration against the API it precedes.

14 min readHOA Backend Engineering CourseMaster

Compose Proved the Lifecycle Works; Kubernetes Runs It for Real Traffic

Lesson 2's docker-compose.yml proved migrate-then-serve ordering, non-root, and read-only constraints on one laptop with one replica. Staging runs three replicas across nodes Compose has no concept of, and Kubernetes has no built-in depends_on — the migrate-then-serve guarantee has to be rebuilt using primitives Kubernetes actually provides: a Job that must complete, applied and awaited before the Deployment.

Organization Service already wrote a ConfigMap, Secret, Deployment and Service, and User Service already wrote the migration Job that must finish before a rollout begins. None of that is re-derived here. What changes at staging scale is narrower and worth being precise about: every image becomes a digest, the Job gets a name that cannot collide across deploys, and the operator — not the manifest — becomes responsible for ordering.

Terminal — repository root

$mkdir -p deploy/organization && cp services/organization/deploy/k8s/*.yaml deploy/organization/

// ConfigMap, Secret, Deployment and Service come from Phase 2 rather than a second set that will drift from them.

$cp services/user-service/deploy/migration-job.yaml deploy/organization/migration-job.yaml

// The Job pattern is User Service's; this copy is what the diff below edits, so it must exist first.

$kubectl create namespace hoa-staging --dry-run=client -o yaml > deploy/organization/namespace.yaml

// Generate the Namespace as a file rather than applying it imperatively — every object this environment owns must be reviewable in the repository.

$ls deploy/organization

// Expect namespace.yaml, migration-job.yaml, configmap.yaml, secret.yaml, deployment.yaml and service.yaml before editing anything.

Create deploy/organization/namespace.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/namespace.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/namespace.yaml (generated above — one label added by hand)

apiVersion: v1
kind: Namespace
metadata:
  name: hoa-staging
  labels:
    environment: staging

The generated file carries only name; the environment label is added because the NetworkPolicy in a later lesson selects namespaces by label rather than by name, and a policy that silently matches nothing is worse than no policy at all.

Create deploy/organization/migration-job.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/migration-job.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/migration-job.yaml (modify — from the Phase 3 migration Job)

 metadata:
-  name: organization-migrate
+  name: organization-migrate-${GIT_SHA}
   namespace: hoa-staging
 spec:
   backoffLimit: 2
+  ttlSecondsAfterFinished: 3600
   template:
     spec:
       restartPolicy: Never
       containers:
         - name: migrate
-          image: ghcr.io/hoa-platform/organization-migrate:latest
+          image: ghcr.io/hoa-platform/organization-migrate@${MIGRATE_IMAGE_DIGEST}

The name suffix is not cosmetic. Kubernetes Jobs are immutable once created, so re-applying the same name with a changed image is rejected rather than updated — every deploy needs its own Job object, and ttlSecondsAfterFinished is what stops those objects accumulating until someone runs a cleanup script nobody wrote.

@sha256:..., never :latest or even a commit-SHA tag

Every image reference in this lesson is a digest, resolved once in CI and substituted at deploy time. A tag reference — even an immutable-looking commit SHA tag — still requires trusting the registry not to have repointed it. A digest is cryptographically exact regardless of what the registry currently claims a tag means.

The Deployment Runs Only After the Job Has Actually Completed

Create deploy/organization/deployment.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/deployment.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/deployment.yaml (modify — from the Phase 2 Deployment)

 spec:
-  replicas: 1
+  replicas: 3
   template:
     spec:
       containers:
         - name: api
-          image: ghcr.io/hoa-platform/organization-api:latest
+          image: ghcr.io/hoa-platform/organization-api@${API_IMAGE_DIGEST}
           ports:
             - { name: http, containerPort: 8081 }
+            - { name: internal, containerPort: 9090 }
           readinessProbe:
             httpGet:
               path: /readyz
-              port: http
+              port: internal
           livenessProbe:
             httpGet:
               path: /healthz
-              port: http
+              port: internal
           resources:
             requests: { cpu: "100m", memory: "128Mi" }
-            limits: { cpu: "500m", memory: "256Mi" }
+            limits: { memory: "256Mi" }
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 65532
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]

The probe *semantics* are unchanged — the /readyz and /healthz split, and the reasoning for keeping them separate, still belong to the Health Checks lesson. Their target does change, and it has to: Observability moved both routes onto the internal listener, so a probe still pointing at the http port would now get a 404 from the public router. Kubernetes would read that as a failed liveness check and restart a perfectly healthy pod, over and over. Naming the port internal rather than writing 9090 twice means the port number lives in exactly one place in this manifest.

A CPU limit on a Go service can throttle it even when the node has spare CPU

Kubernetes enforces CPU limits using the kernel's CFS bandwidth controller in discrete time slices; a Go process with GOMAXPROCS set to the node's full core count can be throttled mid-slice even while idle CPU exists elsewhere on the node. Setting a CPU request for scheduling fairness without a CPU limit, combined with GOMAXPROCS detecting the container's real cgroup quota, avoids that specific failure mode. Memory keeps a hard limit, because an OOM kill is a clean fast failure and an unbounded leak is not.

Create deploy/organization/service.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/service.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/service.yaml (modify — one port stays deliberately absent)

 spec:
   selector: { app: organization-api }
   ports:
     - { name: http, port: 8081, targetPort: http }
+  # No port 9090 entry. The internal metrics and pprof listener is reachable
+  # only by kubectl port-forward to a specific pod — never through this
+  # Service, and never through any Ingress that selects it.

A comment explaining an omission earns its place here: the next engineer to add a port to this Service needs to know that 9090's absence is a decision, not an oversight. The NetworkPolicy in a later lesson enforces the same rule independently, because a comment is not a control.

kubectl Has No depends_on — the Order Is Enforced by the Operator, Not the Manifests

Terminal — repository root, with kubectl context set to staging

$kubectl apply -f deploy/organization/namespace.yaml

// The namespace must exist before anything else in it can be created.

$kubectl apply -f deploy/organization/configmap.yaml -f deploy/organization/secret.yaml

// Both the Job and the Deployment read their environment from these; applying them later means the Job's first pod starts with no DATABASE_URL and fails on connect, which reads like a broken migration rather than a missing Secret.

$kubectl get configmap organization-config secret organization-db-credentials -n hoa-staging

// Cheap confirmation that both exist before anything consumes them — kubectl apply succeeding is not the same as the objects being present under the names the manifests reference.

$kubectl apply -f deploy/organization/migration-job.yaml

// Creates the Job; does not wait for it to finish on its own.

$kubectl wait --for=condition=complete --timeout=120s job/organization-migrate-${GIT_SHA} -n hoa-staging

// Blocks until the Job reports Complete — this explicit wait is what Compose's service_completed_successfully did automatically; Kubernetes requires it to be scripted.

$kubectl apply -f deploy/organization/deployment.yaml -f deploy/organization/service.yaml

// Only reached if the wait above succeeded — a failed migration job means this line never runs, matching Lesson 2's 'API never starts against a broken schema' guarantee.

$kubectl rollout status deployment/organization-api -n hoa-staging --timeout=120s

// Waits until all 3 replicas are Available, not just Created — the actual signal a deploy script should treat as success.

A Deployment Reporting Available Is Not the Same as the API Actually Answering

Terminal — repository root

$kubectl port-forward svc/organization-api 8081:8081 -n hoa-staging

// Creates a temporary local tunnel to the Service, exercising the same path real traffic would take through it.

$curl -sf http://localhost:8081/api/v1/organizations -H "Authorization: ******" && echo 'organization-api is answering through its Service'

// A real API route, because the Service deliberately publishes only port 8081 — health lives on the internal port and is not reachable this way, which is the whole point of omitting 9090 from the Service.

$kubectl port-forward deploy/organization-api 9090:9090 -n hoa-staging

// Health and metrics need a tunnel straight to a pod. That kubelet-only reachability is exactly what the probes rely on.

$curl -sf http://localhost:9090/readyz && echo 'readiness agrees with the rollout status above'

// Confirms the endpoint the readinessProbe actually polls returns 200, rather than trusting the Deployment's own Available condition.

A migration failure surfaced correctly

$ kubectl wait --for=condition=complete --timeout=120s job/organization-migrate-a1b2c3d -n hoa-staging
error: timed out waiting for the condition on jobs/organization-migrate-a1b2c3d

$ kubectl logs job/organization-migrate-a1b2c3d -n hoa-staging
error: migration failed in line 0: CREATE INDEX CONCURRENTLY occupancies_current_gist ... (details: pq: constraint "occupancies_current_gist" already exists)

The wait command's non-zero exit code stops the deploy script before it ever reaches kubectl apply -f deployment.yaml — the same ordering guarantee Lesson 2 enforced with Compose, now enforced explicitly by the script instead of implicitly by depends_on.

Applied exercise

Write the Deployment, Service, and migration Job for services/gateway-service

22 minutes

services/gateway-service has no migrations of its own but does need the same non-root, digest-pinned, readiness/liveness discipline, plus a Service that is the only one of the platform's Services intended to eventually receive external traffic via Ingress.

Write deploy/gateway/deployment.yaml with the same securityContext and probe structure as organization's.
Write deploy/gateway/service.yaml, naming it clearly as the platform's external entry point.
Justify in one sentence why gateway's Deployment needs no equivalent of organization-migrate's Job.
Apply both manifests to a local kind or minikube cluster, then verify /healthz on the internal port via a pod port-forward and a real route through the Service.

Deliverable

Committed gateway manifests, a written justification for skipping a migration Job, and a verified port-forward request against the running Service.

Completion checks

gateway's containers run with runAsNonRoot, readOnlyRootFilesystem, and no CPU limit, matching organization's rationale.
The justification correctly identifies that gateway owns no database schema of its own.
A route through the Service returns 200, and /healthz answers only through a direct pod port-forward on the internal port.

Quiz & Practice

Question 1 / 2

Why does deploy/organization/deployment.yaml set a memory limit but deliberately omit a CPU limit?

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Course outline

01 · Phase 1 — Project Foundation16/16

02 · Phase 2 — Organization Service20/20

03 · Phase 3 — User Service10/10

04 · Phase 4 — Authentication Service10/10

05 · Phase 5 — API Gateway9/9

06 · Phase 6 — Flat Service6/6

07 · Phase 7 — Resident Service7/7

08 · Phase 8 — Maintenance Service7/7

09 · Phase 9 — Payment Service7/7

10 · Phase 10 — Complaint Service7/7

11 · Phase 11 — Notice Service7/7

12 · Phase 12 — Visitor Service6/6

13 · Phase 13 — Staff Service6/6

14 · Phase 14 — Notification Service7/7

15 · Phase 15 — Redis Integration6/6

16 · Phase 16 — Kafka Integration8/8

17 · Phase 17 — Observability7/7

18 · Phase 18 — Performance Engineering6/6

19 · Phase 19 — Production Deployment11/11

20 · Phase 20 — Production Readiness8/8

Stage 7 · Master

Phase 19 — Production Deployment

Kubernetes

14 min readHOA Backend Engineering CourseMaster

Compose Proved the Lifecycle Works; Kubernetes Runs It for Real Traffic

Terminal — repository root

$mkdir -p deploy/organization && cp services/organization/deploy/k8s/*.yaml deploy/organization/

// ConfigMap, Secret, Deployment and Service come from Phase 2 rather than a second set that will drift from them.

$cp services/user-service/deploy/migration-job.yaml deploy/organization/migration-job.yaml

// The Job pattern is User Service's; this copy is what the diff below edits, so it must exist first.

$kubectl create namespace hoa-staging --dry-run=client -o yaml > deploy/organization/namespace.yaml

// Generate the Namespace as a file rather than applying it imperatively — every object this environment owns must be reviewable in the repository.

$ls deploy/organization

// Expect namespace.yaml, migration-job.yaml, configmap.yaml, secret.yaml, deployment.yaml and service.yaml before editing anything.

Create deploy/organization/namespace.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/namespace.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/namespace.yaml (generated above — one label added by hand)

apiVersion: v1
kind: Namespace
metadata:
  name: hoa-staging
  labels:
    environment: staging

Create deploy/organization/migration-job.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/migration-job.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/migration-job.yaml (modify — from the Phase 3 migration Job)

 metadata:
-  name: organization-migrate
+  name: organization-migrate-${GIT_SHA}
   namespace: hoa-staging
 spec:
   backoffLimit: 2
+  ttlSecondsAfterFinished: 3600
   template:
     spec:
       restartPolicy: Never
       containers:
         - name: migrate
-          image: ghcr.io/hoa-platform/organization-migrate:latest
+          image: ghcr.io/hoa-platform/organization-migrate@${MIGRATE_IMAGE_DIGEST}

@sha256:..., never :latest or even a commit-SHA tag

The Deployment Runs Only After the Job Has Actually Completed

Create deploy/organization/deployment.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/deployment.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/deployment.yaml (modify — from the Phase 2 Deployment)

 spec:
-  replicas: 1
+  replicas: 3
   template:
     spec:
       containers:
         - name: api
-          image: ghcr.io/hoa-platform/organization-api:latest
+          image: ghcr.io/hoa-platform/organization-api@${API_IMAGE_DIGEST}
           ports:
             - { name: http, containerPort: 8081 }
+            - { name: internal, containerPort: 9090 }
           readinessProbe:
             httpGet:
               path: /readyz
-              port: http
+              port: internal
           livenessProbe:
             httpGet:
               path: /healthz
-              port: http
+              port: internal
           resources:
             requests: { cpu: "100m", memory: "128Mi" }
-            limits: { cpu: "500m", memory: "256Mi" }
+            limits: { memory: "256Mi" }
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 65532
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]

A CPU limit on a Go service can throttle it even when the node has spare CPU

Create deploy/organization/service.yaml before adding the code

$mkdir -p deploy/organization && touch deploy/organization/service.yaml

// touch preserves an existing file, so the same command is safe when revisiting the lesson.

deploy/organization/service.yaml (modify — one port stays deliberately absent)

 spec:
   selector: { app: organization-api }
   ports:
     - { name: http, port: 8081, targetPort: http }
+  # No port 9090 entry. The internal metrics and pprof listener is reachable
+  # only by kubectl port-forward to a specific pod — never through this
+  # Service, and never through any Ingress that selects it.

kubectl Has No depends_on — the Order Is Enforced by the Operator, Not the Manifests

Terminal — repository root, with kubectl context set to staging

$kubectl apply -f deploy/organization/namespace.yaml

// The namespace must exist before anything else in it can be created.

$kubectl apply -f deploy/organization/configmap.yaml -f deploy/organization/secret.yaml

$kubectl get configmap organization-config secret organization-db-credentials -n hoa-staging

// Cheap confirmation that both exist before anything consumes them — kubectl apply succeeding is not the same as the objects being present under the names the manifests reference.

$kubectl apply -f deploy/organization/migration-job.yaml

// Creates the Job; does not wait for it to finish on its own.

$kubectl wait --for=condition=complete --timeout=120s job/organization-migrate-${GIT_SHA} -n hoa-staging

// Blocks until the Job reports Complete — this explicit wait is what Compose's service_completed_successfully did automatically; Kubernetes requires it to be scripted.

$kubectl apply -f deploy/organization/deployment.yaml -f deploy/organization/service.yaml

// Only reached if the wait above succeeded — a failed migration job means this line never runs, matching Lesson 2's 'API never starts against a broken schema' guarantee.

$kubectl rollout status deployment/organization-api -n hoa-staging --timeout=120s

// Waits until all 3 replicas are Available, not just Created — the actual signal a deploy script should treat as success.

A Deployment Reporting Available Is Not the Same as the API Actually Answering

Terminal — repository root

$kubectl port-forward svc/organization-api 8081:8081 -n hoa-staging

// Creates a temporary local tunnel to the Service, exercising the same path real traffic would take through it.

$curl -sf http://localhost:8081/api/v1/organizations -H "Authorization: ******" && echo 'organization-api is answering through its Service'

$kubectl port-forward deploy/organization-api 9090:9090 -n hoa-staging

// Health and metrics need a tunnel straight to a pod. That kubelet-only reachability is exactly what the probes rely on.

$curl -sf http://localhost:9090/readyz && echo 'readiness agrees with the rollout status above'

// Confirms the endpoint the readinessProbe actually polls returns 200, rather than trusting the Deployment's own Available condition.

A migration failure surfaced correctly

$ kubectl wait --for=condition=complete --timeout=120s job/organization-migrate-a1b2c3d -n hoa-staging
error: timed out waiting for the condition on jobs/organization-migrate-a1b2c3d

$ kubectl logs job/organization-migrate-a1b2c3d -n hoa-staging
error: migration failed in line 0: CREATE INDEX CONCURRENTLY occupancies_current_gist ... (details: pq: constraint "occupancies_current_gist" already exists)

Applied exercise

Write the Deployment, Service, and migration Job for services/gateway-service

22 minutes

Write deploy/gateway/deployment.yaml with the same securityContext and probe structure as organization's.
Write deploy/gateway/service.yaml, naming it clearly as the platform's external entry point.
Justify in one sentence why gateway's Deployment needs no equivalent of organization-migrate's Job.
Apply both manifests to a local kind or minikube cluster, then verify /healthz on the internal port via a pod port-forward and a real route through the Service.

Deliverable

Committed gateway manifests, a written justification for skipping a migration Job, and a verified port-forward request against the running Service.

Completion checks

gateway's containers run with runAsNonRoot, readOnlyRootFilesystem, and no CPU limit, matching organization's rationale.
The justification correctly identifies that gateway owns no database schema of its own.
A route through the Service returns 200, and /healthz answers only through a direct pod port-forward on the internal port.

Quiz & Practice

Question 1 / 2

Why does deploy/organization/deployment.yaml set a memory limit but deliberately omit a CPU limit?

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.