Go makes concurrent programming straightforward with goroutines and channels. However, when multiple goroutines access shared data, you must protect that data from concurrent modifications. A mutex is one of the most common tools for doing this.
A mutex protects shared resources by allowing only one goroutine at a time to enter a critical section. This prevents data races, but it can also introduce performance problems when many goroutines compete for the same lock.
This problem is known as mutex contention.
For example, an application might process thousands of requests concurrently, but every request needs to update the same shared map. Even if the request handlers run in separate goroutines, they may spend a significant amount of time waiting for the mutex. As traffic increases, throughput can stop improving, and response times can become unpredictable.
The solution is not to remove mutexes everywhere. Mutexes are often the right choice for protecting shared state. Instead, you need to identify which locks are causing delays, understand why goroutines are waiting, and reduce unnecessary contention without introducing data races.
In this article, you will learn how mutex contention works in Go, how to reproduce it, how to investigate it with Go's profiling tools, and how to improve lock performance in production applications.
What Is Mutex Contention in Go?
A mutex, short for mutual exclusion, protects a critical section of code from simultaneous access by multiple goroutines.
Go provides two common synchronization types in the sync package:
sync.Mutex provides exclusive access to shared data.sync.RWMutex allows multiple readers to hold read locks simultaneously, while a writer requires exclusive access.
Consider this example:
package main
import (
"fmt"
"sync"
)
type Counter struct {
mu sync.Mutex
value int
}
func (c *Counter) Increment() {
c.mu.Lock()
defer c.mu.Unlock()
c.value++
}
func (c *Counter) Value() int {
c.mu.Lock()
defer c.mu.Unlock()
return c.value
}
func main() {
var counter Counter
var wg sync.WaitGroup
for i := 0; i < 1000; i++ {
wg.Add(1)
go func() {
defer wg.Done()
counter.Increment()
}()
}
wg.Wait()
fmt.Println("Final value:", counter.Value())
}
The mutex protects the value field. Without synchronization, concurrent increments could interfere with one another, producing incorrect results.
However, all 1,000 goroutines must acquire the same mutex before incrementing the counter.
If one goroutine holds the lock, other goroutines that need it must wait. When lock acquisition becomes a significant source of waiting time, the application experiences mutex contention.
A mutex does not automatically cause a performance problem. Contention becomes important when lock requests frequently overlap and the protected work cannot be performed concurrently.
Why Mutex Contention Slows Down Applications
A mutex protects correctness, but excessive contention can reduce the benefits of concurrency.
Too Many Goroutines Competing for One Lock
Suppose an HTTP service receives requests from hundreds of clients. Each request updates a shared statistics map protected by a single mutex.
Even if the application has many available CPU cores, only one goroutine at a time can execute the critical section protected by that mutex.
As the number of concurrent requests increases, more goroutines may wait for access.
This can result in:
- Higher request latency.
- Lower throughput than expected.
- Increased goroutine waiting time.
- Poor scaling when additional CPU cores are available.
- Delayed execution of unrelated work that depends on the same shared resource.
The problem is not necessarily the number of goroutines. It is the amount of work serialized behind the lock.
Holding a Lock for Too Long
A critical section should generally contain only the operations that require exclusive access.
Consider this example:
func (s *Store) UpdateUser(id string, name string) error {
s.mu.Lock()
defer s.mu.Unlock()
s.users[id] = name
if err := s.writeToDatabase(id, name); err != nil {
return err
}
return nil
}
The mutex remains locked while the application writes to the database.
Database operations may involve network communication, connection-pool waiting, or disk activity. During that time, other goroutines that need the same mutex cannot proceed.
A better approach is to separate the protected in-memory update from the database operation when the application's consistency requirements allow it.
func (s *Store) UpdateUser(id string, name string) error {
s.mu.Lock()
s.users[id] = name
s.mu.Unlock()
return s.writeToDatabase(id, name)
}
This reduces the time spent holding the mutex.
However, this change is not automatically correct for every application. If the database write must be atomic with the in-memory update, or if other goroutines depend on the two operations occurring together, a different design is necessary.
The goal is to minimize the critical section without weakening the required consistency guarantees.
Performing Unnecessary Work Inside a Critical Section
Logging, formatting, large loops, and calculations can increase lock duration when they do not need protected access to shared data.
For example, if a goroutine acquires a mutex, copies a shared value into a local variable, and then performs expensive formatting while still holding the lock, other goroutines must wait unnecessarily.
Where possible, copy the required state while holding the lock, release the lock, and perform independent work afterward.
This approach is useful only when the copied data remains valid for the intended operation and no additional synchronization is required.
Reproduce Mutex Contention with a Benchmark
Before optimizing a mutex, create a reproducible benchmark. This gives you a baseline and helps you verify whether a change actually improves performance.
The following example simulates concurrent updates to a shared counter.
Create a file named mutex_test.go:
package mutexdemo
import (
"sync"
"testing"
)
type SharedCounter struct {
mu sync.Mutex
value int64
}
func (c *SharedCounter) Increment() {
c.mu.Lock()
c.value++
c.mu.Unlock()
}
func BenchmarkSharedCounter(b *testing.B) {
var counter SharedCounter
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
counter.Increment()
}
})
}
Run the benchmark:
go test -run '^$' -bench BenchmarkSharedCounter -benchmem -count=5
The command runs the benchmark multiple times and reports measurements such as execution time per operation and memory allocations.
The RunParallel method allows the benchmark to execute operations concurrently. All workers update the same counter, so they compete for the same mutex.
For a more controlled comparison, you can also run the benchmark with different CPU settings:
go test -run '^$' -bench BenchmarkSharedCounter -cpu=1,2,4,8 -count=5
The -cpu option changes the values used for GOMAXPROCS during the test runs.
If performance improves with one CPU but stops improving as concurrency increases, the mutex may be limiting scalability. However, benchmark results alone do not prove that mutex contention is the primary problem. CPU scheduling, benchmark overhead, memory behavior, and other bottlenecks can also affect the results.
Use profiling to investigate the cause.
Use Go's Mutex Profile to Find Contention
Go provides a mutex profile that helps identify code locations associated with mutex contention.
The profile focuses on time spent waiting for contended mutexes, attributed to the code responsible for the corresponding lock operations. It is useful when you want to find which critical sections contribute most to contention.
There are several ways to collect a mutex profile. One practical approach is to expose Go's net/http/pprof endpoints in a controlled profiling environment.
Enable Profiling in a Test Application
Create a small HTTP server:
package main
import (
"log"
"net/http"
_ "net/http/pprof"
"sync"
)
type Counter struct {
mu sync.Mutex
value int64
}
func (c *Counter) Increment() {
c.mu.Lock()
c.value++
c.mu.Unlock()
}
func main() {
counter := &Counter{}
http.HandleFunc("/increment", func(w http.ResponseWriter, r *http.Request) {
counter.Increment()
w.WriteHeader(http.StatusNoContent)
})
log.Println("Application listening on :8080")
log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}
The blank import registers the standard profiling handlers. The server binds to the loopback interface, making the example suitable for local testing.
In a production deployment, expose profiling endpoints only through an appropriately restricted administrative interface or a protected internal network. Profiling endpoints can reveal implementation details and application activity.
Collect a Mutex Profile
Start the application and generate concurrent traffic against the /increment endpoint.
Then collect the profile:
go tool pprof \
http://127.0.0.1:8080/debug/pprof/mutex
The interactive pprof prompt allows you to inspect the profile.
To display the functions with the largest recorded contention costs, run:
top
To inspect the call paths, run:
top -cum
To view the profile in a browser, use:
web
The browser command requires Graphviz to be installed and available in your environment.
The exact profile output depends on the workload and the mutex profile sampling settings. A profile collected before the application experiences meaningful contention may not provide useful results.
Enable Mutex Profiling When Necessary
For an application that needs explicit control over mutex profile sampling, Go provides runtime.SetMutexProfileFraction.
For example:
package main
import (
"log"
"net/http"
_ "net/http/pprof"
"runtime"
)
func main() {
runtime.SetMutexProfileFraction(5)
log.Println("Profiling endpoint listening on 127.0.0.1:6060")
log.Fatal(http.ListenAndServe("127.0.0.1:6060", nil))
}
A positive value enables sampling. A value of 5 requests sampling of approximately one in every five contended mutex events, although profiling is sampled and should not be interpreted as an exact record of every event.
A value of 1 requests sampling of every contended event, which can increase profiling overhead.
The mutex profile can reveal where contention is attributed, but you should also investigate why the lock is held for a long time. A function near the top of the profile may be a symptom of a broader design issue rather than the entire cause.
Understand the Difference Between Mutex and Block Profiles
Go provides more than one profiling option for investigating synchronization problems.
| Profile | Primary purpose | Useful when |
|---|
| CPU profile | Shows where CPU execution time is spent | The application consumes substantial CPU |
| Mutex profile | Identifies code associated with contended mutexes | Shared locks are suspected of limiting performance |
| Block profile | Identifies time goroutines spend blocked on synchronization and related operations | Goroutines appear to wait on channels, mutexes, or other blocking operations |
| Goroutine profile | Shows current goroutine stacks and states | You need to investigate waiting goroutines or suspected deadlocks |
| Heap profile | Shows memory allocation and retention | Memory usage or garbage collection is suspected of affecting performance |
The mutex profile and block profile answer different questions.
The mutex profile helps identify contention attributed to lock holders. The block profile helps identify where goroutines experience blocking.
When diagnosing a performance problem, collect the profile that matches the observed symptom. If a service has high latency but low CPU usage, investigating blocking and synchronization may be more useful than examining only the CPU profile.
Reduce Mutex Contention Without Breaking Correctness
Once you identify a contended lock, consider the following optimization techniques.
Keep Critical Sections Small
Protect only the shared state that requires synchronization.
For example, instead of performing formatting while holding a lock, copy the necessary value and release the lock first.
type Statistics struct {
mu sync.Mutex
counts map[string]int
}
func (s *Statistics) Increment(name string) {
s.mu.Lock()
s.counts[name]++
s.mu.Unlock()
}
func (s *Statistics) Get(name string) int {
s.mu.Lock()
value := s.counts[name]
s.mu.Unlock()
return value
}
This example protects both map access and updates. Go maps are not safe for concurrent reads and writes without synchronization.
If the application uses this structure, initialize the map before calling these methods, and ensure every access follows the same synchronization rules.
Reduce Shared Mutable State
Sometimes the best way to reduce contention is to avoid sharing mutable data.
For example, independent requests can calculate intermediate results locally and combine them later. If every goroutine updates a single global counter repeatedly, the shared counter becomes a coordination point.
Depending on the requirements, alternatives include:
- Per-worker counters with periodic aggregation.
- Sharded maps that divide keys across multiple locks.
- Immutable data structures for read-heavy operations.
- Message passing when a single goroutine can safely own a resource.
These approaches involve trade-offs. Per-worker state may delay visibility of updated totals, while sharding adds complexity to key distribution and aggregation.
Choose the design based on the consistency and latency requirements of the application.
Consider sync.RWMutex for Read-Heavy Workloads
A sync.RWMutex allows multiple goroutines to hold read locks simultaneously. A writer must acquire exclusive access.
It can help when reads are frequent, writes are relatively infrequent, and read operations are sufficiently substantial to benefit from concurrent access.
For example:
type Configuration struct {
mu sync.RWMutex
values map[string]string
}
func (c *Configuration) Get(key string) string {
c.mu.RLock()
value := c.values[key]
c.mu.RUnlock()
return value
}
func (c *Configuration) Set(key, value string) {
c.mu.Lock()
c.values[key] = value
c.mu.Unlock()
}
The read method uses RLock and RUnlock, while the write method uses Lock and Unlock.
However, RWMutex is not always faster than Mutex. It has additional coordination costs, and write activity can limit its benefits. Benchmark both implementations under a workload that resembles the real application.
Use Sharded Locks for Highly Contended Maps
If many goroutines update different keys in the same map, one mutex can serialize unrelated operations.
Sharding divides the data into multiple maps, each with its own mutex. A key determines which shard receives the operation.
For example:
package main
import (
"hash/fnv"
"sync"
)
const shardCount = 32
type shard struct {
mu sync.Mutex
values map[string]int
}
type ShardedCounter struct {
shards [shardCount]shard
}
func NewShardedCounter() *ShardedCounter {
counter := &ShardedCounter{}
for i := range counter.shards {
counter.shards[i].values = make(map[string]int)
}
return counter
}
func shardIndex(key string) uint32 {
hash := fnv.New32a()
_, _ = hash.Write([]byte(key))
return hash.Sum32() % shardCount
}
func (c *ShardedCounter) Increment(key string) {
s := &c.shards[shardIndex(key)]
s.mu.Lock()
s.values[key]++
s.mu.Unlock()
}
func (c *ShardedCounter) Get(key string) int {
s := &c.shards[shardIndex(key)]
s.mu.Lock()
value := s.values[key]
s.mu.Unlock()
return value
}
In this example, each shard protects its own map. Operations on keys assigned to different shards can proceed concurrently.
The hash function distributes keys across the shards, but distribution depends on the workload. If many operations target the same key, that key remains a contention point.
The example uses a fixed number of shards to keep the implementation simple. Production code should also consider how to enumerate all entries, aggregate values, handle memory growth, and test the behavior under concurrent access.
Use Atomic Operations for Simple Counters
If the shared state is a simple numeric counter, an atomic operation may be more suitable than a mutex.
For example:
package main
import (
"fmt"
"sync"
"sync/atomic"
)
func main() {
var counter atomic.Int64
var wg sync.WaitGroup
for i := 0; i < 1000; i++ {
wg.Add(1)
go func() {
defer wg.Done()
counter.Add(1)
}()
}
wg.Wait()
fmt.Println("Final value:", counter.Load())
}
The atomic counter provides synchronized updates without requiring an application-managed mutex.
Atomic operations are useful for simple values, but they do not replace mutexes for protecting complex invariants across multiple fields or data structures. If an operation requires several values to change together, a mutex may still be the clearer and safer choice.
Common Mistakes When Investigating Mutex Contention
Removing a Mutex Without Protecting the Data
Removing a mutex may appear to improve benchmark results, but concurrent access to a regular Go map or other mutable data can introduce data races.
Run the race detector during development and testing:
go test -race ./...
The race detector can identify many data races exercised by the test workload. It does not prove that the entire application is race-free, especially if important execution paths are not tested.
Optimizing Before Collecting a Profile
A shared mutex may look suspicious in code, but that does not mean it is responsible for a production slowdown.
Measure the application under a representative workload, collect the relevant profiles, and establish a baseline before changing synchronization.
Replacing Every Mutex with an RWMutex
Read-write locks are not universally faster. Their performance depends on the read-to-write ratio, critical-section duration, and contention pattern.
Benchmark the alternatives instead of assuming that concurrent readers automatically produce better performance.
Ignoring Lock Ordering
When an application uses multiple mutexes, inconsistent lock acquisition order can cause deadlocks.
For example, one goroutine might acquire lock A and then lock B, while another acquires lock B and then lock A.
Establish a consistent lock ordering where multiple locks must be acquired together, and keep the synchronization design as simple as possible.
Collecting Profiles Without Enough Traffic
A mutex profile collected during an idle period may not reveal the bottleneck that appears under load.
Collect profiles during a representative workload and interpret the results alongside request latency, throughput, CPU usage, and application logs.
A Practical Workflow for Production Troubleshooting
When mutex contention is suspected in a Go service, use a repeatable process.
- Establish a baseline by measuring request latency, throughput, CPU utilization, and memory usage.
- Reproduce the problem with a benchmark or a representative load test.
- Collect a mutex profile while the application experiences the slowdown.
- Inspect the highest-cost entries and identify the relevant critical sections.
- Review the code for long lock durations, unnecessary shared state, and unrelated work performed while holding locks.
- Make one targeted change, such as reducing the critical section, introducing sharding, or using an atomic counter.
- Run the same benchmark and compare the results with the baseline.
- Run tests with the race detector and validate correctness under concurrent workloads.
- Deploy the change gradually and monitor latency, throughput, and error rates.
This workflow helps distinguish genuine contention problems from unrelated performance issues and makes optimization decisions easier to verify.
Conclusion
Mutex contention occurs when multiple goroutines compete for the same lock and spend time waiting for exclusive access. Although mutexes are essential for protecting shared data, poorly designed critical sections can limit concurrency and reduce application throughput.
Go provides practical tools for diagnosing these problems, including mutex profiles, block profiles, goroutine profiles, and parallel benchmarks. Use them to identify the locks that contribute most to waiting time before changing the synchronization design.
Once you understand the bottleneck, reduce unnecessary work inside critical sections, reconsider shared mutable state, evaluate sync.RWMutex for read-heavy workloads, and consider sharding or atomic operations when they fit the problem.
The objective is not to eliminate mutexes. It is to protect shared data correctly while allowing independent work to proceed concurrently. Measure first, make targeted changes, and verify both performance and correctness before deploying an optimization.
Join the conversation! Your thoughts help the community grow.