Stage 3 · Build

Observability & Networking

Advanced Concurrency

errgroup, semaphore, worker pools, and pipeline patterns for parallel processing.

7 min readGo for SREBuild

errgroup

errgroup manages a group of goroutines that return errors. It provides synchronized startup, error collection, and context cancellation.

Goerrgroup-usage.go

16 linesLn 1, Col 1Go

errgroup.WithContext creates a group tied to a context. When any goroutine returns an error, the context is canceled, signaling other goroutines to stop. SetLimit bounds concurrency to prevent resource exhaustion.

Semaphore Pattern

A semaphore limits concurrent access to a resource. Use it when errgroup is not flexible enough — for example, when you need to track which goroutines are running.

Gochannel-based-semaphore.go

30 linesLn 1, Col 1Go

The semaphore is a buffered channel. Sending to the channel acquires a slot. Receiving from it releases a slot. Select with ctx.Done() allows cancellation while waiting for a slot. errors.Join combines multiple errors.

Worker Pools

Worker pools run a fixed number of goroutines processing items from a channel. This is efficient for processing large volumes of work with bounded resources.

Goworker-pool-pattern.go

47 linesLn 1, Col 1Go

jobs channel feeds work to workers. results channel collects outcomes. Closing jobs signals workers to stop. WaitGroup ensures all workers finish before closing results. This pattern is predictable and testable.

Pipeline Patterns

Pipelines connect processing stages through channels. Each stage reads from its input channel, processes, and sends to its output channel. This decomposes complex processing into composable steps.

Gopipeline-stages.go

37 linesLn 1, Col 1Go

Each stage is a function that returns a read channel. Stages run concurrently and communicate through channels. The pipeline is assembled by chaining channel inputs to outputs. Context cancellation stops all stages.

Fan-in / Fan-out

Fan-out distributes work across multiple goroutines. Fan-in merges results from multiple goroutines into a single channel. This scales processing across available CPU cores.

Gofan-in/fan-out.go

53 linesLn 1, Col 1Go

Fan-out creates N workers reading from the same channel. Fan-in uses a merge function to combine multiple output channels. The WaitGroup ensures the merged channel closes after all sources are done.

Rate Limiting

Rate limiting controls how fast you send requests. Use it when calling external APIs, databases, or any resource with rate limits.

Gotoken-bucket-rate-limiter.go

29 linesLn 1, Col 1Go

rate.NewLimiter creates a token bucket limiter. rate.Limit is requests per second. Burst is the maximum burst size. Wait blocks until a token is available or the context is canceled. This prevents overwhelming external services.

Use golang.org/x/sync for concurrency primitives

errgroup, semaphore, and other concurrency patterns are available in golang.org/x/sync. These are well-tested, production-ready implementations. Use them instead of writing your own concurrency primitives.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.