Zero-Downtime Deploys

What Caused Visible Deploys

A deployment becomes visible to users long before the whole system goes down. In Fieldwork's early staging rollouts, the most common symptom was a short burst of 502s from api-gateway during image updates. Nothing catastrophic happened. A few old pods terminated before replacement pods were actually ready, and some in-flight requests lost the race. That was enough to prove the point: "Kubernetes supports rolling updates" is not the same thing as "your application is currently configured for zero-downtime deploys." The platform gives you mechanisms. You still have to connect them to real application behavior.

We rejected recreate-style updates immediately because Fieldwork is an always-on API serving task dashboards, membership checks, and mobile clients across multiple tenants. Even short windows of no healthy gateway pods are externally visible. The real work, then, was making rolling updates truthful. A new pod should enter service only when it is genuinely ready, and an old pod should leave only after it has stopped accepting new traffic and finished the work it already owns.

Readiness Had to Gate Traffic

Readiness was the first half of invisible deploys. Every service already exposed /health/ready, but for zero-downtime rollouts that endpoint had to track one more thing: whether the process should still receive new requests. When Kubernetes begins terminating a pod, there is a period where the process is alive but should be considered draining. We made readiness flip to false as soon as shutdown began. That caused the pod to be removed from Service endpoints before the process exited, which gave Ingress and kube-proxy time to stop routing new traffic toward it.

Shutdown Had to Drain Cleanly

The second half of invisible deploys lived inside the Go process. Once SIGTERM arrived, the service stopped advertising readiness, but it also had to stop accepting new HTTP connections and give in-flight requests a bounded grace period. We used net/http graceful shutdown with context deadlines, and we kept handlers cancellation-aware so a long-running task export or membership sync could respect shutdown instead of fighting it. A pod that flips readiness false but exits immediately still drops requests. A pod that ignores shutdown forever blocks rollout progress. Draining means neither of those things happens.

The subtle but important choice was where to hold the draining state. We kept it in-process and made the readiness endpoint depend on it, rather than trying to infer termination indirectly from Kubernetes events. That kept the rule close to the code that actually serves traffic. When the process knows it is draining, it can stop claiming readiness immediately and begin orderly shutdown in the same place.

The Rollout Strategy We Standardized

On the Kubernetes side, we standardized on RollingUpdate with maxUnavailable: 0 and a small maxSurge so capacity only moved forward during deploys. We also added a terminationGracePeriodSeconds value long enough for normal requests to finish and a short preStop hook to give endpoint removal a head start before process shutdown. The point was not to pile on knobs. The point was to make the cluster's timing line up with the application's timing. If kubelet sends SIGTERM and kills the container before upstream routing has reacted, you still get dropped requests even with graceful shutdown code present.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: fieldwork
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api-gateway
          image: ghcr.io/thesyscoder/fieldwork/api-gateway:sha-9f2a6c1
          lifecycle:
            preStop:
              exec:
                command: ["/busybox/sh", "-c", "sleep 5"]
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080

• Flip readiness false as soon as shutdown starts, not at process exit.
• Use graceful HTTP shutdown with explicit timeouts for in-flight requests.
• Keep maxUnavailable at zero for public API services unless you deliberately accept visible degradation.
• Give endpoint removal and connection draining a few seconds of head start before the process dies.

What We Actually Shipped

Fieldwork shipped rolling updates backed by real readiness gates, in-process draining state, graceful HTTP shutdown, and Kubernetes rollout settings that preserved capacity while pods rotated. New pods received traffic only after startup completed. Old pods stopped receiving new work before exit and had time to finish the requests they already owned. That combination, not any single flag, is what made deploys boring.

The lesson here is that zero-downtime deploys are an end-to-end property. Kubernetes can orchestrate the handoff, but only if the application tells the truth about readiness and honors shutdown correctly. Fieldwork did not assume that behavior came for free. We wired it deliberately, tested it during rollouts, and treated a quiet deploy as a feature worth engineering on purpose.