A .NET application can appear to be running normally while many requests are actually stuck because of a deadlock. The process may still be alive, the application may still respond to some endpoints, and CPU usage may even remain low. From the outside, it can look like a slow application or a temporary infrastructure problem. In reality, two or more threads may be waiting for each other to release resources, so none of them can continue.
Production deadlocks are difficult to investigate because the problem may disappear when the application is restarted. Logs usually show that requests timed out, but they often do not explain why the requests became stuck. A memory dump gives a snapshot of the running process at the time of the problem. By examining thread stacks and synchronization information, you can often identify which threads are blocked and what they are waiting for.
This article explains how to investigate a .NET deadlock using a production memory dump, how to recognize a real deadlock, how to distinguish it from normal lock contention or thread pool starvation, and how to prevent the same problem from returning.
What Is a Deadlock?
A deadlock happens when two or more threads are waiting for resources held by each other.
Consider two locks named LockA and LockB. One thread takes LockA and then waits for LockB. Another thread takes LockB and then waits for LockA.
Neither thread can continue.
The first thread cannot continue until the second thread releases LockB, but the second thread cannot continue until the first thread releases LockA.
A simple example looks like this:
private readonly object _lockA = new();
private readonly object _lockB = new();
public void ProcessA()
{
lock (_lockA)
{
Thread.Sleep(100);
lock (_lockB)
{
// Work
}
}
}
public void ProcessB()
{
lock (_lockB)
{
Thread.Sleep(100);
lock (_lockA)
{
// Work
}
}
}
The problem is not that either lock is inherently wrong. The problem is that the application uses a different locking order in the two methods.
ProcessA uses:
LockA -> LockB
ProcessB uses:
LockB -> LockA
If both methods execute at the same time, they can become permanently blocked.
How a Production Deadlock Usually Appears
A deadlock does not always produce an obvious error message. In many cases, the application simply becomes unresponsive.
You may notice symptoms such as:
- API requests taking much longer than normal.
- Requests eventually returning timeout errors.
- Several threads waiting inside lock-related operations.
- CPU usage dropping even though request processing has stopped.
- Some endpoints continuing to work while others are stuck.
- Background workers appearing to stop processing.
- Increasing request queues.
- Thread pool starvation symptoms.
- No obvious exception in the application logs.
The important point is that a deadlock can happen without crashing the process.
This is why restarting the application often appears to fix the issue. Restarting destroys the blocked threads and releases all process-level locks. The application starts again with a clean state, but the underlying code problem remains.
Why a Memory Dump Helps
A memory dump captures the state of the process at a specific point in time. For deadlock investigation, the most useful information is not just the managed objects in memory. The thread state and call stacks are especially important.
A dump can help answer questions such as:
- Which threads are currently blocked?
- What method is each thread executing?
- Is a thread waiting for a monitor or lock?
- Which thread currently owns a synchronization object?
- Are multiple threads waiting for the same resource?
- Are the threads waiting on each other?
- Is the problem actually a deadlock or just slow work?
This makes a dump much more useful than simply looking at an exception log after the incident.
The dump should ideally be captured while the application is still showing the problem. If the process is restarted first, the evidence may be lost.
Capturing a Production Dump
The .NET diagnostic tools can be used to collect a dump from a running process.
First, identify the application process:
dotnet-dump ps
You can then collect a dump using the process ID:
dotnet-dump collect -p 12345 -o production-deadlock.dmp
The actual process ID will be different in your environment.
Dump collection should be treated as a production diagnostic operation. A dump can contain sensitive information from the running application, including strings, request-related data, configuration values, and other process state.
Store the dump securely and restrict access to the people who need it for troubleshooting.
If the application is running inside a container, make sure the diagnostic tool is available in the container or use an appropriate diagnostic workflow for the hosting environment.
Opening the Dump
Once the dump has been collected, open it using the .NET dump analyzer:
dotnet-dump analyze production-deadlock.dmp
You will then get an interactive diagnostic prompt.
The exact output can vary depending on the .NET runtime and the application, but the investigation generally starts with thread information and managed call stacks.
Start With the Threads
The threads command provides information about managed threads in the process.
threads
This gives you an overview of the threads and their current states.
When investigating a suspected deadlock, look for threads that are blocked, waiting, or spending time inside synchronization-related operations.
Do not assume that every waiting thread represents a deadlock. A production application normally has threads waiting for work, I/O, timers, database responses, and other operations.
The goal is to find a pattern.
If several threads are waiting and their stacks point toward the same synchronization mechanism, the situation becomes more interesting.
Inspect the Managed Call Stacks
After identifying suspicious threads, inspect their managed stacks using:
clrstack
For more information about local variables and arguments, you can use:
clrstack -a
The call stack can show where the thread entered the locking code.
For example, you might find a stack similar to this conceptually:
MyService.ProcessA()
System.Threading.Monitor.Enter()
Another thread may show:
MyService.ProcessB()
System.Threading.Monitor.Enter()
The important part is not the exact stack output. What matters is finding the relationship between the threads.
One thread may be holding one lock while waiting for another. Another thread may be holding the second lock while waiting for the first.
That is the classic deadlock pattern.
Inspect Synchronization Objects
The syncblk command is particularly useful when investigating monitor-based synchronization.
syncblk
It can provide information about synchronization blocks associated with managed objects, including lock ownership and waiting threads where that information is available.
The output should be read together with the thread stacks rather than in isolation.
Suppose the dump indicates this relationship:
Thread 10 owns LockA
Thread 10 waits for LockB
Thread 11 owns LockB
Thread 11 waits for LockA
This is strong evidence of a circular wait.
The key concept is circular dependency.
Thread 10 cannot proceed because Thread 11 owns the resource it needs.
Thread 11 cannot proceed because Thread 10 owns the resource it needs.
Neither thread can release its current resource because neither can reach the code that releases it.
Understanding the Deadlock Through a Simple Example
Consider the following code:
public void UpdateCustomerAndOrder()
{
lock (_customerLock)
{
UpdateCustomer();
lock (_orderLock)
{
UpdateOrder();
}
}
}
public void UpdateOrderAndCustomer()
{
lock (_orderLock)
{
UpdateOrder();
lock (_customerLock)
{
UpdateCustomer();
}
}
}
If both methods execute concurrently, the following sequence can happen:
Thread A:
Owns customerLock
Waiting for orderLock
Thread B:
Owns orderLock
Waiting for customerLock
The application is now deadlocked.
The problem becomes much easier to understand when looking at the dump because the dump captures the blocked execution state instead of forcing you to reconstruct the situation from logs.
Deadlock Versus Lock Contention
Not every lock problem is a deadlock.
Lock contention occurs when one thread holds a lock and other threads have to wait for it. Once the owner releases the lock, the waiting threads continue.
For example:
Thread A:
Owns LockA
Performing database-related work
Thread B:
Waiting for LockA
Thread C:
Waiting for LockA
This can make the application slow, but it is not necessarily a deadlock.
A deadlock looks different:
Thread A:
Owns LockA
Waiting for LockB
Thread B:
Owns LockB
Waiting for LockA
The distinction is important because the solution is different.
For contention, you may need to reduce lock duration or change the synchronization design.
For a deadlock, you need to remove the circular dependency.
Deadlock Versus Thread Pool Starvation
Thread pool starvation can produce symptoms that look similar to a deadlock.
For example, if many thread pool threads are blocked by synchronous I/O or long-running work, new requests may have difficulty getting threads.
The application may appear frozen even though there is no circular lock dependency.
A dump helps separate these situations.
If the stacks show many threads waiting on the same lock and the lock ownership forms a circular dependency, investigate a deadlock.
If many threads are blocked on synchronous operations without a circular dependency, investigate thread pool starvation or blocking I/O instead.
This distinction prevents teams from changing synchronization code when the real problem is excessive synchronous work.
Async Code Can Have Deadlock Problems Too
Modern .NET applications use asynchronous programming heavily, so deadlock analysis should not stop at lock.
An async deadlock can happen when asynchronous operations depend on each other and neither operation can complete.
For example:
Task taskA = ProcessAAsync();
Task taskB = ProcessBAsync();
await Task.WhenAll(taskA, taskB);
This code is not itself a deadlock. The problem occurs when the operations internally create a circular dependency.
Another common problem is mixing asynchronous code with synchronous blocking:
var result = SomeOperationAsync().Result;
or:
SomeOperationAsync().Wait();
This can cause thread starvation and, in some application environments, deadlock behavior.
In ASP.NET Core, there is no classic request SynchronizationContext like the one found in older application models, so the traditional "async deadlock" explanation should not automatically be applied to every .Result or .Wait() call. However, synchronous blocking still consumes threads and can create serious scalability and dependency problems.
The general rule is to keep asynchronous call chains asynchronous.
Use:
var result = await SomeOperationAsync();
instead of:
var result = SomeOperationAsync().Result;
Do Not Hold a Lock Across Await
The standard lock statement cannot be used across an await.
For asynchronous mutual exclusion, SemaphoreSlim is often a better fit.
For example:
private readonly SemaphoreSlim _semaphore = new(1, 1);
public async Task ProcessAsync()
{
await _semaphore.WaitAsync();
try
{
await DoWorkAsync();
}
finally
{
_semaphore.Release();
}
}
The finally block is important because the semaphore must be released even if the operation throws an exception.
Forgetting to release it can create another form of application-wide blocking.
Be Careful With Database Deadlocks
A .NET process dump can show that application threads are waiting for database operations, but that does not automatically mean the database itself has a deadlock.
There are two different problems.
A CLR deadlock occurs inside the application process. Threads can be waiting for managed locks or other synchronization resources.
A database deadlock occurs inside the database engine. Two or more database transactions can hold locks that the other transactions need.
For example:
Transaction A:
Locks Row 1
Waiting for Row 2
Transaction B:
Locks Row 2
Waiting for Row 1
The application threads may simply be waiting for the database to return.
If a dump shows several threads blocked inside database calls, investigate the database's own deadlock detection and diagnostic tools as well.
Do not automatically classify every blocked database call as a .NET deadlock.
Compare Multiple Dumps
One dump provides a snapshot.
Two or more dumps collected during the incident can provide a much stronger picture.
Suppose you collect three dumps several seconds apart and observe that the same threads remain blocked on the same locks.
That makes a persistent deadlock much more likely.
On the other hand, if the thread stacks change between dumps and the waiting threads eventually move forward, you may be dealing with temporary contention or slow external work.
A useful production investigation can therefore look like this:
Dump 1
|
v
Identify blocked threads
|
v
Inspect call stacks
|
v
Inspect synchronization state
|
v
Collect another dump
|
v
Compare thread and lock behavior
This gives you evidence instead of relying on assumptions from a single snapshot.
Fixing the Locking Problem
The most common solution for lock-order deadlocks is to establish a consistent locking order.
For example, if the application has two locks:
CustomerLock
OrderLock
always acquire them in the same order:
CustomerLock -> OrderLock
Do not allow another code path to use:
OrderLock -> CustomerLock
A corrected implementation could look like this:
public void UpdateCustomerAndOrder()
{
lock (_customerLock)
{
lock (_orderLock)
{
UpdateCustomer();
UpdateOrder();
}
}
}
public void UpdateOrderAndCustomer()
{
lock (_customerLock)
{
lock (_orderLock)
{
UpdateOrder();
UpdateCustomer();
}
}
}
The methods may perform different business operations, but they acquire the shared resources in the same order.
Another useful approach is to reduce the amount of work performed while holding a lock.
Avoid doing database calls, HTTP calls, file operations, or expensive calculations while holding a lock unless the design specifically requires it.
The longer a lock is held, the greater the opportunity for contention and dependency problems.
Common Mistakes During Deadlock Investigation
Restarting the Application Too Quickly
Restarting usually removes the immediate symptoms, but it also destroys the evidence.
If the issue is reproducible and the application is safe to diagnose, capture a dump before restarting.
Looking Only at Exceptions
A deadlock may not generate a managed exception.
A timeout exception may be the result of a deadlock, but the exception itself may not identify the thread holding the required resource.
Use thread and synchronization information from the dump.
Assuming Every Waiting Thread Is a Deadlock
Waiting is normal in server applications.
Threads wait for locks, I/O, timers, database responses, queues, and other resources.
Look for a circular dependency rather than simply counting waiting threads.
Holding Locks During Slow Operations
This is a common design problem:
lock (_lock)
{
CallExternalService();
}
If the external service becomes slow, every thread that needs the same lock can become blocked.
A better design is often to avoid holding the lock while performing slow external work.
Ignoring Cancellation and Timeouts
Long-running operations should have appropriate cancellation and timeout behavior where possible.
Timeouts do not fix a deadlock, but they can prevent a blocked operation from holding resources indefinitely in other failure scenarios.
Best Practices for Preventing Deadlocks
Use a consistent lock ordering strategy throughout the application.
Keep critical sections small and predictable.
Avoid performing database, network, or file operations while holding application locks when possible.
Do not mix synchronous blocking with asynchronous workflows.
Use SemaphoreSlim.WaitAsync for asynchronous mutual exclusion instead of trying to force async code into synchronous locking patterns.
Avoid unnecessary shared mutable state.
Document non-obvious synchronization rules so another developer does not accidentally introduce a conflicting lock order.
Monitor request latency and thread pool behavior so that a deadlock can be detected before it becomes a prolonged outage.
Most importantly, treat synchronization as part of application architecture rather than as a small implementation detail.
Production Deadlock Troubleshooting Checklist
When a production application appears to be stuck, use the following approach:
- Check whether requests are timing out or remaining pending.
- Check CPU, memory, thread count, and thread pool behavior.
- Capture a process dump while the problem is occurring.
- Open the dump with dotnet-dump analyze.
- Use threads to inspect thread states.
- Use clrstack to inspect suspicious managed stacks.
- Use syncblk to investigate monitor synchronization.
- Look for threads waiting on resources owned by other blocked threads.
- Determine whether the problem is a CLR deadlock, lock contention, thread pool starvation, or external I/O.
- Compare multiple dumps when the diagnosis is unclear.
- Fix the synchronization or dependency design.
- Add monitoring and tests that can detect the problem earlier.
Advantages and Disadvantages of Dump-Based Deadlock Analysis
Aspect | Advantage | Disadvantage |
|---|
Production diagnosis | Can inspect the process without immediately restarting it | Requires careful handling of sensitive dump data |
Thread analysis | Shows what threads were doing at the time of capture | Large applications can produce a lot of diagnostic information |
Lock investigation | Helps identify synchronization relationships | Interpreting synchronization output requires practice |
Root cause analysis | Provides evidence from the actual process state | A single dump may not be enough |
Incident response | Useful when logs do not explain the problem | Dump collection itself must be planned carefully |
Summary
A production deadlock is more than a slow request problem. It is usually a dependency problem where threads or asynchronous operations are waiting for resources that can never become available because of a circular dependency.
A memory dump provides a snapshot of that state. With tools such as dotnet-dump, you can inspect thread states, managed call stacks, and synchronization information to understand what the application was doing when it became stuck.
The most useful investigation starts with the threads command, moves into clrstack for suspicious threads, and uses syncblk when monitor-based synchronization is involved. The goal is not simply to find waiting threads. The goal is to identify the dependency between them.
It is also important to separate a CLR deadlock from thread pool starvation, normal lock contention, slow database operations, and database-level deadlocks. These problems can produce similar symptoms but require different fixes.
The best long-term solution is to design synchronization carefully. Keep lock ordering consistent, keep critical sections small, avoid blocking asynchronous operations, and avoid holding locks while performing slow external work.
When a production application becomes unresponsive, capturing a dump before restarting the process can turn a difficult incident into a traceable engineering problem.
Join the conversation! Your thoughts help the community grow.