Project challenges / verified progress
Beacon: turn a program into a service

The engineering notebook

Shut down politely

What should happen between SIGTERM and the process actually exiting?

Loading statusStage 8 of 10

  • Workspace not ready
  • Agent not ready
Focus25:00
A small focus ritual

0 focus sessions completed. Every fourth session offers a longer break. Start each phase when you are ready.

Study time never unlocks verified lesson progress.

Loading...

Loading verified progress...

Loading GitHub account...
Phase 4 — Process behaviour

Step 01 of 06

Learn the concept

Production rarely ends with Ctrl-C and a clean conscience. Supervisors send SIGTERM, wait a deadline, then use SIGKILL if the process refuses to leave. Kubernetes calls that deadline terminationGracePeriodSeconds; course four will make it very real.

THIRTY SECOND EXITsecondsSIGTERMstop requested0Drainfinish work25sDeadlineforce exit5s left
A graceful shutdown is a budget. Stop accepting new work immediately, give current work a bounded window, and exit before the supervisor reaches for SIGKILL.
Step 01

The ideas this is made of

Signals are process-level messages

SIGTERM is the standard polite stop signal. SIGINT usually comes from Ctrl-C. SIGKILL cannot be caught, delayed or cleaned up after. Your code only gets a chance to close databases and drain requests if it responds to catchable signals before the outside system loses patience.

NotifyContext connects signals to Go cancellation

signal.NotifyContext returns a context cancelled when selected signals arrive. Passing that context to the scheduler, server and checks creates one stop line through the process. Remember to call the returned stop function so signal handling is released during cleanup.

HTTP shutdown is not server close

Server.Close drops active connections. Server.Shutdown(ctx) first closes listeners so new requests stop, then waits for active handlers until the context deadline. That is the behaviour users expect during deploys: no new work, but current requests get a fair chance.

Kubernetes will not wait forever

In Kubernetes, a pod receives SIGTERM and has terminationGracePeriodSeconds, often 30s, before SIGKILL. If Beacon needs 45s to drain, Kubernetes will cut it off. Designing a 25s shutdown deadline inside a 30s grace period leaves room for runtime and network slop.

Cancel on SIGTERM
package main

import (
	"context"
	"fmt"
	"os"
	"os/signal"
	"syscall"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	<-ctx.Done()
	fmt.Println("shutdown requested")
}

The program waits until a signal cancels the context. Beacon will use the same cancellation to stop the scheduler and begin HTTP server shutdown.

Stop signals

SignalCatchableUse

SIGINT

Yes

Terminal interrupt

SIGTERM

Yes

Supervisor stop

SIGKILL

No

Forced kill

SIGHUP

Yes

Reload in some daemons

What these are called on the job

  • Drain — Let in-flight work finish while refusing new work.

  • SIGTERM — A catchable request for a process to terminate.

  • SIGKILL — An uncatchable forced process kill.