Stage 7 · Master
Containerization and Deployment
Zero-Downtime Deploys as an Application Contract
Why rolling updates alone are not enough, how readiness and graceful shutdown actually prevent customer-facing errors, and how Meridian should extend its current /healthz-only scaffold.
What Zero-Downtime Really Means
A deployment is zero-downtime only if new traffic keeps succeeding while old processes are replaced. That outcome depends on four cooperating pieces: a new pod must start before capacity is removed, readiness must stay false until the pod can really serve, terminating pods must stop receiving new work before exit, and the process must finish in-flight work within a bounded grace period. A Kubernetes rolling update can orchestrate this handoff, but it cannot invent truthful application behavior on its own.
| Technique | What it solves | What it does not solve |
|---|---|---|
| RollingUpdate | Replaces pods gradually instead of deleting everything at once | Does not stop a pod from claiming readiness too early or exiting too abruptly. |
| Readiness probe | Prevents new traffic from reaching an unready or draining pod | Does not finish requests already in progress. |
| Graceful shutdown | Lets in-flight requests complete before process exit | Does not by itself remove the pod from Service endpoints quickly enough. |
What the Repo Has Today
Today every Meridian service exposes only GET /healthz and starts directly from cmd/<service>/main.go. apps/gateway/cmd/gateway/main.go sets read and write timeouts, and apps/gateway/internal/router/router.go returns a simple health response. That is enough for a scaffolding phase. It is not yet enough for zero-downtime rollouts, because the health endpoint does not express drain state and the processes do not yet coordinate SIGTERM, readiness withdrawal, and graceful server shutdown.
A static health endpoint answers 'the process exists.' Zero-downtime deployment needs a richer answer: 'the process should still receive new traffic right now.' Those are different questions.
Request-Path Draining
The fundamental pattern is drain-before-exit. As soon as shutdown begins, a request-path service flips an internal draining flag. The readiness handler starts returning failure, so Kubernetes removes the pod from Service endpoints. Only after that removal begins should the HTTP server wait for active requests to finish. If the order is reversed, the pod keeps attracting new requests while trying to shut down. If the first step never happens, external traffic continues to hit a dying process until the kernel closes the socket.
Planned Code and Manifest Changes
apps/gateway/cmd/gateway/main.gomodifyResponsibility: Add signal handling and bounded graceful shutdown for the gateway HTTP server.
Why now: The public entrypoint is the first place where abrupt process exit becomes customer-visible.
apps/gateway/internal/router/router.gomodifyResponsibility: Differentiate liveness and readiness or make /healthz drain-aware until separate endpoints exist.
Why now: Probe truthfulness is the control-plane input that determines whether Kubernetes should keep routing traffic.
apps/identity-service/cmd/identity-service/main.gomodifyResponsibility: Apply the same graceful-shutdown discipline to identity-service.
Why now: Internal services can still create gateway-visible errors if they disappear mid-request.
apps/community-service/cmd/community-service/main.gomodifyResponsibility: Add signal handling and drain-aware readiness.
Why now: Community routes will eventually serve real traffic and must roll safely like the gateway.
apps/billing-service/cmd/billing-service/main.gomodifyResponsibility: Add the same shutdown behavior to billing-service.
Why now: Billing correctness is especially sensitive to interrupted requests and partial responses.
deploy/k8s/base/gateway/deployment.yamlmodifyResponsibility: Define maxUnavailable, maxSurge, terminationGracePeriodSeconds, and readiness probes that match the new drain behavior.
Why now: Application shutdown logic is ineffective if Kubernetes is not configured to give it time and endpoint removal semantics.
Rolling Update Math
For request-path services, the safest starting rule is maxUnavailable: 0 and a small maxSurge value. maxUnavailable: 0 preserves existing capacity while new pods come online. A small surge avoids creating an artificial thundering herd against downstream services during rollout. This matters especially in Meridian because gateway fans out to identity-service, community-service, and billing-service; a very aggressive rollout can create connection spikes that are not normal customer traffic.
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
spec:
terminationGracePeriodSeconds: 30
containers:
- name: gateway
readinessProbe:
httpGet:
path: /healthz
port: 8080
lifecycle:
preStop:
exec:
command: ["/busybox/sh", "-c", "sleep 5"]The preStop delay is not the shutdown solution by itself. It merely gives endpoint removal a head start. The process still needs real drain logic.
Reference Specification
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.