Troubleshoot Crash Issues on IIS Using DebugDiag & WindDbg

The previous article, explained High CPU utilization issues and how to collect & analyze memory dumps using DebugDiag. Here, we will look into troubleshooting crashes of web sites hosted on IIS 7 and IIS 7.5. Sometimes, our ASP.NET application might not respond to our requests and shows the "This page can't be displayed" error. This might happen when w3wp.exe hosting your website crashes. W3wp.exe crashes in scenarios like unhandled exceptions, memory overflow, heap corruption and so on. We need to find the root cause for this issue. We can troubleshoot this type of issue by collecting a memory dump of the w3wp.exe at the time of crash and analyze it further in WinDbg. Let's download DebugDiag 1.2 [Available here] based on server bit ness [32 or 64 bit].

Let's create a sample MVC application that will throw a stack overflow exception [second chance exception]:

public class HomeController : Controller
{
public ActionResult Index()
{

MethodA();

return View();
}

private static void MethodA()

{

MethodA();
}

}

Now publish this app on IIS using the following procedure:

Go to Project properties and click on Web tab.
Select Local IIS Web Server as shown below:

When we browse to the application [http://localhost/MvcApplication4], w3wp.exe crashes after sometime and that will be logged in the Windows Application Log as shown below:

Let's open DebugDiag and configure a rule to collect a full user memory dump for the crash of w3wp.exe as shown below:

Click Next:

Click Next and type w3wp.exe or select the specific App Pool or select w3wp.exe, if it is already running:

Click Next & Finish.

Run the application, this will throw an unhandled exception and DebugDiag will collect a full user dump having information about the threads, stack trace, exception, locks and so on. Let's analyze it using advanced analysis.

This shows an exception occurred on thread 22 and stack trace shows recursion.

This analysis report provides the following details:

All running threads along with CPU time.
List of all running requests [under Client Connections].
Exception details.
Stack trace of each thread.
Process details like up time, process ID and so on.
Server and OS details.

Let's analyze it further in WindDbg. Download WinDbg [Available here], drag & drop the memory dump and use the following commands:

Point WinDbg to Microsoft symbols server, run command .symfix
Run .sympath to confirm its pointing tohttp://msdl.microsoft.com/download/symbols.
Load .NET extension to analyze .NET threads, exception, stack trace and so on, run .loadby sos clr & .reload comands.
When we crash dump, we will be in the crashing thread [22].
Run !ClrStack, we can see the following stack trace showing recursion on MethodA.
We can use the !analyze –v command to get exception details.

Let's dig into DebugDiag customization:

We can use Exceptions to collect a stack trace or a dump on a specific native or .NET exception.
Breakpoints are used to collect stack trace or dump on a process termination, unhandled exception and so on.
Events are used to collect stack trace or dump on a specific event like Module unload, Thread exit and so on.
PageHeap Flags are used to troubleshoot memory leak issues.

Let's complete the explanation by having an overview:

A crash can happen due to an unhandled exception, heap corruption, memory overflow and so on.
To troubleshoot the issue, create a crash rule in DebugDiag and REPRO the issue.
Analyze each dump individually using Advanced Analysis of DebugDiag.
Look at the thread that threw the exception and analyze its stack trace from the bottom to the top.
We can use SOS or PSSCor extensions to analyze .NET code within WinDbg.
Refer to here for a full list of WinDbg SOS commands.

I will end here with the previous explanation and in future articles explain troubleshooting other issues like hang, high memory and so on.