Worker Pools

In this lesson, explore worker pools through one small Go program, and connect its output to the language rules behind it.

A worker pool bounds the number of goroutines processing a stream of jobs. Feed jobs through a channel, run a fixed number of workers, and collect results with enough metadata to reconstruct order.

Choose worker counts using expected workload, latency, CPU and I/O capacity; a pool is not automatically the best choice for every job.

Follow this diagram before executing the example. Check where a value or decision changes what runs next.

This example is a complete program. Save it as main.go or paste it into the Go Playground.

Go code

package main

import (
	"fmt"
	"sync"
)

type Result struct{ index, square int }

func main() {
	jobs := make(chan int)
	results := make(chan Result, 4)
	var wg sync.WaitGroup
	for w := 0; w < 2; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for n := range jobs {
				results <- Result{n - 1, n * n}
			}
		}()
	}
	for n := 1; n <= 4; n++ {
		jobs <- n
	}
	close(jobs)
	wg.Wait()
	close(results)
	ordered := make([]int, 4)
	for r := range results {
		ordered[r.index] = r.square
	}
	fmt.Println(ordered)
}

Output

[1 4 9 16]

Two workers receive four jobs. The buffered results channel prevents the main producer from getting stuck while it waits. Indexed results reconstruct [1 4 9 16].

• An unbuffered results channel plus waiting before consuming can deadlock.
• Close jobs after the producer finishes; close results after all workers finish.

Increase jobs to six while leaving only two workers; adjust result capacity and ordered output.

Tip: Edit the example above, predict the output, and run the modified code to test your reasoning.

You have practiced Worker Pools in a small, self-contained program. The important habit is to predict what runs, inspect the actual output and distinguish the language rule from the particular example.

Before continuing, make sure you can explain the program to someone who has not seen it, including the line that changes its result.