Stage 7 · Master
Background Processing
Graceful Worker Shutdown
A billing-service worker must drain in-flight tenant jobs on SIGTERM instead of abandoning them during pod reschedules.
Why Shutdown Matters
A worker that processes billing jobs for invoices or payments will eventually run on Kubernetes, where pods are stopped routinely for rollouts, node drains, and scaling decisions. If the process exits immediately on SIGTERM, the platform risks duplicating payment work, abandoning long-running receipt generation, or losing observability about which tenant job failed mid-flight. Shutdown behavior is therefore part of correctness, not a deployment afterthought.
The current repo has no worker binary yet. This lesson defines the shutdown contract that apps/billing-service/cmd/worker/main.go should implement when the background-processing phase begins.
Signal-Handling Shape
Draining In-Flight Jobs
Graceful shutdown has two phases. First, the worker stops claiming new jobs. Second, it allows in-flight jobs to finish within a bounded grace period. For payment and invoice workflows, the worker should log tenant_id, job ID, and job type when shutdown begins so operators can see which work is still draining. If the grace period expires, the queue lease should lapse cleanly so another worker can retry the job later.
Workflow
Signal: SIGTERM cancels the shared worker context and begins a bounded shutdown path.
Step 1 / 4 — Signal
Kubernetes and Leases
| Concern | Required behavior | Reason |
|---|---|---|
| SIGTERM from pod eviction | Stop claiming and drain | Normal rollout path |
| Queue lease / visibility timeout | Expire or extend explicitly | Prevents duplicate long-running work |
| readiness during shutdown | Report not ready before exit | Stops new traffic or job assignment |
| structured logs | Include tenant_id and job_id | Makes replay and incident review possible |
The queue adapter should treat a claimed job as leased, not owned forever. If a worker dies after taking a payment-webhook job, another worker must eventually be able to retry it. If a worker is merely draining on shutdown, it may extend the lease for a long-running receipt-render step. That distinction is what keeps graceful shutdown from turning into silent work loss.
apps/billing-service/cmd/worker/main.gocreateResponsibility: Owns signal handling, shutdown deadlines, and process exit semantics.
Why now: Lifecycle orchestration belongs at the binary entrypoint, not deep inside a queue adapter.
Connects to: Uses libs/platform/logging and the billing-service config loader already present in the module.
apps/billing-service/internal/queue/lease.gocreateResponsibility: Tracks claim, ack, retry, and lease-expiry behavior for in-flight jobs.
Why now: Graceful shutdown only works if the queue contract understands temporary ownership.
Connects to: Complements the retry and scheduling rules from the previous lesson.
The Decision
- billing-service workers should live inside a planned cmd/worker binary and shut down through context cancellation on SIGTERM.
- Shutdown starts by stopping new job claims, then draining in-flight jobs within a bounded grace period.
- Queue leases must expire or be extended explicitly so shutdown never silently loses tenant work.
- Logs and metrics for shutdown paths must include tenant_id and job_id because those values drive replay and incident response.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.