Introduction
Goroutines are one of Go's most useful concurrency features. They are lightweight, easy to create, and commonly used for HTTP requests, background workers, message processing, pipelines, and asynchronous operations.
But a goroutine can also become a production problem when it remains blocked forever.
A few leaked goroutines may not matter immediately. A service that continuously creates them, however, can gradually consume memory and increase garbage-collection work. Over time, the application can become slower or unstable.
Go 1.27 introduces the goroutineleak profile as a generally available runtime profiling feature. It identifies a class of goroutines that are permanently blocked on supported concurrency primitives and cannot become unblocked. The profile is available through runtime/pprof and through the /debug/pprof/goroutineleak endpoint provided by net/http/pprof.
This changes how production teams can investigate goroutine leaks. Instead of looking only at the total goroutine count, developers can ask the runtime to identify goroutines that appear to be permanently stuck.
What Is a Goroutine Leak?
A goroutine leak occurs when a goroutine remains alive even though the work it was created for can never complete.
Consider this example:
func startWorker() {
ch := make(chan string)
go func() {
message := <-ch
fmt.Println(message)
}()
}
The goroutine waits for a value from ch.
If nothing can ever send a value to the channel, the goroutine remains blocked indefinitely.
Repeated calls to startWorker() can therefore create an increasing number of permanently blocked goroutines.
The problem looks like this:
Request
|
+---- Start goroutine
|
v
Wait on channel
|
X
Nobody sends
|
v
Goroutine remains
This is different from a goroutine that is temporarily waiting for legitimate work.
Why Goroutine Leaks Matter
A leaked goroutine consumes resources indirectly.
Each goroutine has runtime-managed state and can retain references to objects through its stack. As leaked goroutines accumulate, they can therefore contribute to increasing memory usage and garbage-collection overhead.
A typical progression might look like:
10:00 500 goroutines
10:30 2,000 goroutines
11:00 8,000 goroutines
11:30 20,000 goroutines
The absolute numbers are not enough to prove a leak.
A busy service can legitimately have many goroutines.
The important signal is whether goroutines are accumulating without eventually completing their intended lifecycle.
What Changed in Go 1.27?
Before Go 1.27, developers commonly used the standard goroutine profile:
/debug/pprof/goroutine
That profile reports the stacks of all current goroutines.
Go 1.27 adds:
/debug/pprof/goroutineleak
The new profile specifically reports goroutines that the runtime determines are leaked according to its leak-detection model. The profile was experimental in Go 1.26 and became generally available in Go 1.27.
The distinction is important:
Profile | Purpose |
|---|---|
| Shows all current goroutines |
| Shows detected leaked goroutines |
| Shows blocking on synchronization primitives |
| Shows mutex contention |
| Shows memory allocation information |
The normal goroutine profile is still valuable. The new profile is more targeted.
How Go Detects a Leak
The Go runtime uses garbage-collector reachability to identify a particular class of permanently blocked goroutines.
Conceptually:
Runnable Goroutines
|
v
Reachable Objects
|
v
Concurrency Primitive
|
v
Blocked Goroutine
If the blocking primitive cannot be reached from runnable goroutines or goroutines that could eventually unblock it, the runtime can determine that the blocked goroutine cannot wake up.
This allows the runtime to distinguish many permanent leaks from ordinary temporary blocking.
However, this mechanism does not detect every possible goroutine leak.
What Types of Leaks Can It Detect?
The Go 1.27 goroutine leak profiler focuses on goroutines permanently blocked on supported Go concurrency primitives.
Examples include:
Channel sends
Channel receives
Blocking
selectsync.Mutexsync.RWMutexsync.WaitGroupsync.Cond
The runtime documentation specifically describes these as the supported blocking mechanisms for leak detection.
For example:
func worker(ch chan int) {
ch <- 42
}
If the send can never complete, the goroutine may be identified by the leak profiler.
A Realistic Leak Pattern
A common pattern occurs when multiple workers send results back to a coordinator.
Consider:
func processWork(items []int) error {
ch := make(chan int)
for _, item := range items {
go func() {
result := process(item)
ch <- result
}()
}
if err := validate(); err != nil {
return err
}
return nil
}
The function can return before receiving all results.
The worker goroutines may then remain blocked at:
ch <- result
The channel has no receiver because the parent function has already returned.
This produces a classic goroutine leak.
The important lesson is that creating a goroutine is not the hard part.
Designing its termination path is.
Exposing the Profile With net/http/pprof
If your service already uses net/http/pprof, Go 1.27 exposes the new profile automatically.
A simple development setup is:
package main
import (
_ "net/http/pprof"
"log"
"net/http"
)
func main() {
go func() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}()
// Application code...
}
The profiling endpoint is then available under:
/debug/pprof/
The new profile is:
/debug/pprof/goroutineleak
The Go standard library exposes this endpoint specifically for leaked goroutine stack traces.
Collect the Leak Profile
You can collect the profile using curl:
curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof
Then inspect it with:
go tool pprof leak.prof
The pprof tool can inspect and visualize profiles generated by Go applications.
A typical investigation starts with:
pprof
|
+---- top
+---- list functionName
+---- web
The objective is to identify the stack where leaked goroutines are blocked.
Use debug=2 for Human-Readable Output
The leak endpoint also supports a human-readable form:
/debug/pprof/goroutineleak?debug=2
This produces stack traces in a format similar to unrecovered panic output. The net/http/pprof implementation documents debug=2 specifically for this profile.
For a quick production investigation, this can be easier than immediately opening the profile with go tool pprof.
Finding the Blocking Operation
Suppose the profile shows:
goroutineleak
|
v
worker
|
v
ch <- result
That is much more actionable than simply seeing:
20,000 goroutines
The profile points the investigation toward the synchronization operation responsible for the blocked goroutines.
From there, inspect:
Who creates the goroutine?
Who owns the channel?
Who is supposed to receive the value?
Can that receiver exit early?
Is cancellation propagated?
Is the channel closed appropriately?
Can the goroutine terminate on context cancellation?
Fixing a Channel Leak
One possible solution is to ensure the channel can accept all results even if the consumer exits early.
For example:
ch := make(chan result, len(items))
The worker can then send without requiring an active receiver for every message.
However, buffering is not a universal fix.
If the worker performs expensive or long-running work that should be cancelled when the parent operation exits, a context-based design is usually more appropriate.
Use Context Cancellation
A more robust pattern is:
func worker(ctx context.Context, ch chan<- result) {
result := process()
select {
case ch <- result:
return
case <-ctx.Done():
return
}
}
Now the worker has an explicit termination path.
The design becomes:
Parent Operation
|
v
Context
|
+----------+
| |
v v
Worker A Worker B
| |
+----+-----+
|
v
Cancellation
|
v
Workers Exit
This is often preferable to simply increasing channel capacity.
Compare goroutine and goroutineleak Profiles
The two profiles answer different questions.
Normal Goroutine Profile
Use:
go tool pprof http://localhost:6060/debug/pprof/goroutine
This answers:
What goroutines are currently running or blocked?
Leak Profile
Use:
go tool pprof http://localhost:6060/debug/pprof/goroutineleak
This answers:
Which goroutines does the runtime identify as permanently leaked?
Both can be useful during an incident.
For example:
Goroutine count increased
|
v
Check goroutine profile
|
v
Identify suspicious stacks
|
v
Check goroutineleak profile
|
v
Confirm permanently blocked goroutines
Production Setup
The profiling endpoint should not simply be exposed publicly.
A safer architecture is:
Internet
|
X
|
Application
|
Private Profiling Endpoint
|
Operations Network
Protect profiling endpoints using appropriate network controls and authentication mechanisms.
Profiling data can expose:
Internal package names
File paths
Function names
Request-related information
Runtime state
Treat profiling endpoints as operational diagnostics, not public application endpoints.
How to Detect Leaks Before They Become Large
The new profile is useful during incidents, but monitoring should still track goroutine behavior over time.
A simple signal is:
goroutine count
|
v
Time series
|
v
Unexpected sustained growth
However, a rising goroutine count alone is not proof of a leak.
For example, traffic increases may legitimately create more concurrent goroutines.
Use the goroutine count as an investigation trigger, then use profiles to determine what those goroutines are doing.
Limitations of the Goroutine Leak Profiler
The new profiler is useful but not universal.
Network and File I/O
A goroutine blocked on network or file I/O is not automatically considered a leaked goroutine by this mechanism.
For example:
data, err := conn.Read(buffer)
The goroutine may be waiting for legitimate external activity.
The leak profiler is focused on supported Go concurrency primitives rather than every operation that can block.
Reachable Synchronization Objects
The runtime can miss certain leaks when the relevant concurrency primitive remains reachable through global variables or runnable goroutines.
This is a consequence of the reachability-based detection model.
Leaks Must Occur First
The profiler detects leaks after they exist.
It does not predict that a particular code path will leak a goroutine before the problematic execution occurs.
Common Mistakes
Looking Only at Goroutine Count
A high count does not automatically mean a leak.
Assuming Every Blocked Goroutine Is Leaked
Some goroutines are expected to wait for work.
Using Buffering as the Only Fix
A larger channel can hide a lifecycle problem rather than solve it.
Ignoring Context Cancellation
Background work should have an explicit termination path when the parent operation ends.
Exposing pprof Publicly
Profiling endpoints should be protected like other operational interfaces.
Fixing the Symptom Instead of the Lifecycle
Deleting goroutines or forcing shutdown may hide the underlying ownership problem.
Best Practices
Give every long-lived goroutine a clearly defined owner.
Define how each goroutine terminates.
Propagate
context.Contextthrough cancellable operations.Use the
goroutineleakprofile when investigating permanent synchronization leaks.Compare leak profiles before and after a suspected fix.
Use the normal goroutine profile to understand overall runtime behavior.
Monitor goroutine-count trends, but do not treat them as proof of a leak.
Protect
pprofendpoints from unauthorized access.Test cancellation and early-return paths explicitly.
Combine production profiling with concurrency-focused tests.
Production Investigation Workflow
A practical incident workflow can be:
1. Observe sustained goroutine growth
|
v
2. Capture normal goroutine profile
|
v
3. Capture goroutine leak profile
|
v
4. Identify common blocking stack
|
v
5. Inspect channel / sync lifecycle
|
v
6. Reproduce the execution path
|
v
7. Add cancellation or correct ownership
|
v
8. Deploy the fix
|
v
9. Capture profile again
|
v
10. Verify leaked goroutines no longer accumulate
This provides a measurable before-and-after process instead of relying only on application restarts.
Production Checklist
[ ] Application runs on Go 1.27 or later
[ ] pprof is available through a protected endpoint
[ ] Goroutine count is monitored
[ ] Normal goroutine profile can be collected
[ ] goroutineleak profile can be collected
[ ] Goroutine ownership is documented
[ ] Background operations have termination paths
[ ] Context cancellation is used where appropriate
[ ] Early-return paths are tested
[ ] Channel lifecycle is reviewed
[ ] Profiling endpoints are not publicly exposed
[ ] Leak fixes are validated with another profile
Summary
Go 1.27 makes goroutine leak profiling a standard runtime capability rather than an experimental feature. The goroutineleak profile can identify a significant class of permanently blocked goroutines, particularly those waiting indefinitely on channels and supported sync primitives.
The most useful distinction is between goroutine count and goroutine leaks.
A high goroutine count tells you that many goroutines exist. The leak profile can help identify goroutines that cannot possibly become unblocked.
A practical production approach is:
Monitor
|
v
Detect unusual growth
|
v
Profile
|
v
Find blocking operation
|
v
Fix lifecycle
|
v
Profile again
The best fix is usually not simply to increase resources. It is to make goroutine ownership and termination explicit.
When a goroutine starts, the code should have a clear answer to one question:
Under what condition will this goroutine stop?
That question is at the heart of reliable Go concurrency.
Join the conversation! Your thoughts help the community grow.