Platform Engineering

Graceful Shutdown Isn't Optional: The Rolling Deploy That Dropped 3% of Requests

Every deploy, every day, for months, we silently dropped a small percentage of real requests — and nobody noticed until we actually measured it, because 3% doesn't look like an outage.

July 10, 20267 min read

Kubernetesgraceful shutdownrolling deploysSIGTERM

Nobody filed an incident. There was no page, no red dashboard, no angry customer email that anyone connected to a deploy. And yet every single rolling deploy — dozens per week — was silently dropping around 3% of in-flight requests, for months, because our application never handled SIGTERM correctly. It took someone deliberately measuring request success rate *during* a deploy window, not just before and after, to notice it at all.

How we even found this

A teammate was investigating unrelated flaky end-to-end tests and noticed the failures clustered suspiciously close to deploy timestamps. Pulling error-rate metrics filtered to a tight 30-second window around each rolling deploy (rather than the usual hourly aggregate, which completely smooths this out) showed a small, consistent, unmistakable spike: roughly 3% of requests returning connection-reset or 502 errors, every single time, for the duration of each pod replacement.

What Kubernetes actually does when it terminates a pod

During a rolling update, Kubernetes sends SIGTERM to the container, then waits up to terminationGracePeriodSeconds (default 30s) before force-killing it with SIGKILL. Separately, and concurrently, it removes the pod's IP from the Service's endpoint list. The critical detail almost nobody accounts for: these two things happen in parallel, not in a guaranteed safe order. There is a real window where a pod has received SIGTERM (and, in the naive case, immediately stops accepting connections) while some in-flight requests are still being routed to it, or new connections are still arriving because the endpoint removal hasn't fully propagated to every kube-proxy / load balancer yet.

The race between SIGTERM and endpoint-list propagation

sequenceDiagram
    participant K8s as Kubernetes control plane
    participant Pod as Terminating pod
    participant Proxy as kube-proxy / LB
    participant Client

    par SIGTERM sent
        K8s->>Pod: SIGTERM
        Pod->>Pod: process exits immediately\n(no graceful drain)
    and endpoint removal (parallel, not sequenced)
        K8s->>Proxy: remove pod IP from endpoints
        Note over Proxy: takes time to propagate\nacross every node
    end
    Client->>Proxy: new request (endpoint not yet removed)
    Proxy->>Pod: routes to terminating pod
    Pod--xClient: connection reset (pod already exited)

The fix: a deliberate three-part shutdown sequence

The fix has three parts, and all three matter — most guides only mention the second one, which alone isn't sufficient.

**A pre-stop delay before anything else happens.** Give the endpoint-removal propagation time to actually complete across the cluster before the app stops accepting new connections.
**Stop accepting new connections, but let in-flight ones finish.** This is what `http.Server.Shutdown()` actually does in Go — it's the part everyone thinks is the whole fix.
**A termination grace period long enough for the slowest realistic in-flight request**, not just the average one.

Part 1: preStop hook — the part most teams skip

containers:
  - name: api
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "5"]
    terminationGracePeriodSeconds: 30

This alone fixed roughly 60% of the drops we measured. SIGTERM isn't sent until after this sleep completes, giving the Service's endpoint list time to propagate the pod's removal to every kube-proxy/load balancer before the app even starts shutting down. Requests stop arriving before the app stops accepting them.

Part 2: graceful HTTP shutdown (the part everyone remembers)

srv := &http.Server{Addr: ":8080", Handler: router}

go func() {
    if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
        log.Fatalf("server error: %v", err)
    }
}()

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM)
<-sigCh // block until SIGTERM

ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()

// Stops accepting new connections immediately, but waits for
// in-flight requests to complete (up to the context deadline)
// before returning.
if err := srv.Shutdown(ctx); err != nil {
    log.Printf("graceful shutdown timed out: %v", err)
}

srv.Shutdown() closes the listener immediately (no new connections accepted) but lets already-accepted requests finish naturally, up to the given deadline. Without this, the naive default is the process exiting the instant SIGTERM arrives, killing every in-flight request mid-response.

Part 3: grace period sized for the slowest real request, not the average

terminationGracePeriodSeconds: 35
# 5s preStop delay + up to 20s for Shutdown() to drain + 10s buffer.
# If your p99 request latency is 8s, don't set this to 15 total —
# a burst of slow requests right as a deploy starts will get killed anyway.

We initially set this too low based on average latency and still saw occasional drops — the requests being cut off were specifically the slow-tail ones (p99+), which is exactly the population most likely to still be in-flight when a pod terminates.

The one more thing: readiness probes need to fail fast on shutdown too

We also updated the app to fail its readiness probe the instant it receives SIGTERM, before even starting the drain sequence. This gives Kubernetes an additional, faster signal (beyond just the endpoint removal from pod deletion) to stop routing new traffic to a terminating pod — belt-and-suspenders on top of the preStop delay.

Result: 3% dropped requests per deploy → effectively 0%

After all three changes, the same tight-window measurement around deploy timestamps showed no measurable increase in error rate during rolling updates — deploys became genuinely invisible to real traffic, not just invisible to our hourly-aggregated dashboards.