Stage 7 · Master
Phase 3 — User Service
Deployment
Deploy user-service to Kubernetes with a Job that runs golang-migrate migrations to completion before the Deployment's rolling update begins, and readiness probes that keep half-started pods out of rotation.
Why Migrations Cannot Run Inside the App's Startup Path
It is tempting to run migrate up inside main() before the HTTP server starts. That works for a single replica. The moment user-service scales to three replicas, a rolling update can start all three new pods at nearly the same instant, and two of them will race to apply the same migration — one wins, one gets a Postgres advisory-lock timeout or a duplicate-column error, and the rollout looks flaky for a reason that has nothing to do with the application code.
Kubernetes's Job resource runs to completion and is a separate lifecycle object from the Deployment's replica set. Sequencing 'Job completes' before 'Deployment begins its rollout' — via a deploy pipeline step, not application code — removes the race entirely: migrations run once, from one pod, before any replica of the new version is created.
A Migration Job for This Release
apiVersion: batch/v1
kind: Job
metadata:
name: user-service-migrate
labels:
app: user-service
spec:
backoffLimit: 2
ttlSecondsAfterFinished: 300
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: hoa/user-service-migrate:latest
command: ["migrate", "-path", "/migrations", "-database", "$(DATABASE_URL)", "up"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: user-service-db
key: migrateUrlttlSecondsAfterFinished cleans up the completed Job automatically so a deploy pipeline that runs many times a day doesn't accumulate hundreds of finished Job objects. Kubernetes substitutes $(DATABASE_URL) into command itself, with no shell involved, so the secret key it reads must already hold the exact string golang-migrate expects -- see the callout below.
user-service-db carries two keys pointing at the identical hoa_user database: url (postgres://..., the scheme pgxpool.New expects, mounted into the Deployment) and migrateUrl (pgx5://..., the scheme golang-migrate's pgx/v5 driver requires, mounted only into this Job). Kubernetes's $(VAR) substitution in a command array is literal -- it cannot strip or rewrite a scheme the way the Makefile's shell-based targets do -- so the pgx5:// form has to already exist as its own secret value.
The Deployment, Gated on the Job
spec:
+ replicas: 3
+ strategy:
+ type: RollingUpdate
+ rollingUpdate:
+ maxUnavailable: 0
+ maxSurge: 1
template:
spec:
containers:
- name: user-service
+ readinessProbe:
+ httpGet: { path: /readyz, port: 8082 }
+ periodSeconds: 5
+ livenessProbe:
+ httpGet: { path: /healthz, port: 8082 }
+ periodSeconds: 10Organization Service already taught Deployment, ClusterIP, ConfigMap, and Secret syntax. The only new decision here is rollout capacity: maxUnavailable: 0 prevents an old pod from leaving until a replacement has passed the database-aware readiness check after the migration Job completes.
Organization Service's health-checks lesson owns the readyz/healthz split; here the probes are present only because maxUnavailable: 0 needs a trustworthy readiness gate after migrations finish.
Confirming the Sequence Locally With kind
| Failure point | Pipeline response | Why rollout stays safe |
|---|---|---|
| Migration Job fails | Stop before applying Deployment | New code never starts against an unknown schema state. |
| New pod fails readiness | Rollout pauses | maxUnavailable: 0 leaves old ready replicas serving traffic. |
| Kind verification fails | Fix locally before registry push | The sequence is proven without touching a shared cluster. |
Applied exercise
Design the failure path
The migration Job's backoffLimit of 2 is exhausted because migration 00004 has a syntax error that only manifests against production data volume.
- Describe what a deploy pipeline should do when kubectl wait for the Job times out or reports failure.
- Decide whether the Deployment's rollout should proceed, and justify your answer using the maxUnavailable setting.
- Propose one pre-deploy check that could have caught this specific migration error earlier.
Deliverable
A short runbook entry: pipeline behavior on Job failure, the rollout decision, and the earlier-catch proposal.
Completion checks
- The rollout decision explicitly blocks progression on Job failure, not just 'investigate later'.
- The earlier-catch proposal references running the migration against a realistic data volume before production.
Why does the migration run as a separate Kubernetes Job instead of inside the application's main() at startup?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.