TopolearnUnbounded goroutines can exhaust memory, file descriptors or a downstream service; pools keep load predictable.
Starting one goroutine per item is fine for a hundred items, but not for a million URLs or rows, and it can overwhelm whatever you're calling. A worker pool fixes concurrency at N:
1. Create a jobs channel and a results channel. 2. Start N worker goroutines, each running for j := range jobs { results <- process(j) }. 3. Feed jobs from another goroutine and close(jobs) when done — this is what ends the workers' loops. 4. When all workers have returned (a WaitGroup), close results so the collector's range ends.
Choose N based on the bottleneck: around runtime.GOMAXPROCS(0) for CPU-bound work; for I/O-bound work, the limit the remote system tolerates (often tens).
Carry errors in the result type (type result struct { id int; err error }) rather than dropping them, and decide whether one failure should cancel the rest — if so, use a context or errgroup with SetLimit(n), which is effectively a worker pool with error handling built in.
To limit rate rather than concurrency, use a time.Ticker to pace job submission, or the token-bucket golang.org/x/time/rate.Limiter whose Wait(ctx) blocks until a request is allowed.
Size channel buffers deliberately. A buffered jobs channel lets the producer run ahead; an unbuffered one gives natural back-pressure. Large buffers mostly hide problems.
Remember results arrive in completion order, not submission order; include an id if the caller needs to match them up.
Example
package main
import (
"fmt"
"sync"
)
func main() {
jobs := make(chan int)
results := make(chan int)
var wg sync.WaitGroup
for range 3 { // fixed pool of 3 workers
wg.Add(1)
go func() {
defer wg.Done()
for j := range jobs {
results <- j * j
}
}()
}
go func() {
for j := 1; j <= 9; j++ {
jobs <- j
}
close(jobs) // ends the workers' range loops
}()
go func() { wg.Wait(); close(results) }()
sum := 0
for r := range results {
sum += r
}
fmt.Println(sum) // 285
}Remember
- N workers ranging over a jobs channel bound concurrency to N
- Close jobs to stop workers; close results after wg.Wait()
- Use rate.Limiter or a Ticker when the limit is requests per second
Watch out for
- Closing the results channel from a worker, or before all workers finish — causes a send-on-closed panic.