Stage 3 · Build
Long-Running Daemons & Services
Background Workers
errgroup, bounded queues, backoff, jitter, and draining worker pools safely.
8 min readGo for SREBuild
Worker Patterns
Background workers process tasks asynchronously. They are essential for retrying failed operations, batch processing, and scheduled work. Choose the right pattern based on your needs.
| Pattern | Use Case | Complexity |
|---|---|---|
| errgroup | Parallel tasks with error handling | Low |
| Worker pool | Fixed concurrency with job queue | Medium |
| Semaphore | Dynamic concurrency limiting | Low |
| Pipeline | Multi-stage processing | High |
errgroup Workers
Goerrgroup-worker-pattern.go
38 linesLn 1, Col 1Go
errgroup provides bounded concurrency with SetLimit. Context cancellation stops all workers on first error. Collecting errors with mutex allows processing all items even when some fail.
Bounded Queues
Gobounded-work-queue.go
56 linesLn 1, Col 1Go
Bounded queues prevent memory exhaustion. Submit returns false when the queue is full — this provides backpressure. Shutdown closes the jobs channel and waits for workers to finish. Results channel collects outcomes.
Backoff and Jitter
Gobackoff-strategies.go
55 linesLn 1, Col 1Go
Exponential backoff doubles the delay each retry. Jitter prevents thundering herd — many clients retrying at the same time. Full jitter is the most effective at preventing collisions. Always cap the maximum backoff.
Safe Draining
Goworker-pool-with-safe-draining.go
65 linesLn 1, Col 1Go
Safe draining closes the done channel to stop new jobs, then drains remaining jobs from the queue. The shutdown timeout prevents hanging workers from blocking the process. Workers finish current jobs before stopping.
Error Handling Strategies
Goworker-error-handling.go
70 linesLn 1, Col 1Go
Retry with exponential backoff handles transient failures. ErrorHandler logs or alerts on permanent failures. Dead letter queue stores failed jobs for later retry. This provides resilience without losing work.
Monitor worker pool metrics
Track queue length, worker count, processing time, and error rate. Alert on growing queue length — it indicates workers cannot keep up. Monitor error rates to detect upstream failures.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.