Go Concurrency Without Leaks: Workers, Backpressure, and Ownership

Starting a goroutine is easy; deciding who waits for it, who can cancel it, and what happens when the consumer leaves is the real work. I have found that useful Go concurrency is usually smaller than the first design. A fixed amount of parallel work, a visible queue policy, and one owner for each channel are enough for many APIs, importers, and background jobs.

Start by proving concurrency helps

I keep the first implementation synchronous when it already meets the required latency or throughput. Concurrency adds scheduling, cancellation, ordering, and partial-failure behaviour. It earns that cost when independent work spends time waiting for I/O or when CPU work can use more than one core, not simply because Go makes goroutines cheap.

Before adding workers, I name the constrained resource: database connections, an upstream rate limit, memory per job, CPU, or file descriptors. The worker count should follow that constraint. Fifty goroutines competing for a ten-connection database pool usually create waiting and noisier tail latency, not more useful throughput.

Every goroutine needs an owner and an exit

For each go statement, I want three answers: who waits for it, what tells it to stop, and what happens if its send or receive cannot complete. A goroutine blocked forever on an unconsumed channel is a leak even if it uses little CPU. Over time it can retain request data, sockets, timers, or other objects that make the leak expensive.

Channel ownership should also be explicit. The goroutine that produces the values normally closes the channel after its final send; receivers do not close a channel just to announce that they are done. Cancellation travels separately through context. When sending or receiving may block after cancellation, a select on ctx.Done gives the goroutine a way out.

func produce(ctx context.Context, values []Item) <-chan Item {
    out := make(chan Item)

    go func() {
        defer close(out)
        for _, item := range values {
            select {
            case out <- item:
            case <-ctx.Done():
                return
            }
        }
    }()

    return out
}

Bound both active work and waiting work

A fixed worker count bounds active work, but a huge buffered channel can still move the problem into memory. Queue capacity is part of the service contract. Once the buffer is full, the producer must block, reject, shed, combine, or durably store work. Each choice is valid in a different system; an unlimited in-memory backlog is not a choice so much as a delayed outage.

Blocking is natural when the caller can safely slow down, such as a batch reader feeding a parser. Rejection fits a live API that can return 429 or 503 and let the caller retry. Durable storage fits work that must survive a process restart. I make the choice visible in code and metrics rather than hiding it behind a very large channel.

• Workers bound simultaneous resource use
• The channel buffer bounds in-memory waiting
• The enqueue deadline bounds how long producers wait
• A rejected or persisted job needs an explicit caller contract
• Queue depth and oldest-job age reveal pressure early

Use errgroup when sibling tasks share a fate

The golang.org/x/sync/errgroup package is useful when several goroutines belong to one operation. WithContext cancels the derived context when one function returns an error, and Wait returns the first non-nil error after the goroutines finish. SetLimit bounds the number of active goroutines, which is often simpler than maintaining a custom worker pool for a finite slice of tasks.

The functions still have to observe the derived context. errgroup cannot interrupt a database driver or HTTP request that was given a different context. I also avoid launching a goroutine that waits on the same group’s Wait from inside that group; ownership should flow from the caller that created it.

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8)

for _, accountID := range accountIDs {
    g.Go(func() error {
        return refreshAccount(ctx, accountID)
    })
}

if err := g.Wait(); err != nil {
    return fmt.Errorf("refresh accounts: %w", err)
}

A long-running worker pool has a different lifecycle

For a process-wide pool, I start the workers in one component, close its input only when no producer can send again, cancel its context during shutdown, and wait for every worker. Jobs receive the pool context so blocking calls can stop. If jobs have individual deadlines, they can derive shorter child contexts without breaking the process-wide cancellation.

Shutdown policy must be honest. A service can drain accepted jobs, stop immediately and retry them elsewhere, or finish only jobs that have crossed a durable boundary. It cannot promise to drain arbitrary in-memory work within a fixed deployment grace period. If losing an accepted job is unacceptable, the queue belongs in durable storage before a worker acknowledges it.

Test termination, not only results

A concurrency test should cover the point where the consumer stops early, one worker fails, the parent context is cancelled, the queue is full, and shutdown begins with work in flight. I use bounded test deadlines so a regression fails instead of hanging the whole suite. Go’s race detector is also worth running for packages whose concurrency paths changed, although a clean race run does not prove that goroutines terminate.

I prefer observable completion over counting goroutines before and after a test; the runtime and libraries may have their own goroutines. A WaitGroup, errgroup, or closed done channel lets a test verify that the goroutines it owns actually returned. This also produces a much clearer failure than a global goroutine count that changed by one.

Use this in practice

• Name the resource concurrency is meant to protect
• Give every goroutine a cancellation and join path
• Let producers close their output channels
• Bound active and queued work separately
• Choose block, reject, or persist explicitly
• Test early exit, cancellation, and shutdown
• Run the race detector for changed concurrent code