UI freezes occasionally

Hello,

We have a bunch of multi-threaded time-critical  C# applications communicating with the outside world via TCP/IP sockets. UI of any of these applications can occasionally get frozen for a few seconds (in extreme cases up to 30 seconds). This does not happen often and can perhaps be explained by some unusual conditions that occur from time to time on any Windows system. As far as we could ascertain from the logs, these delays only occur on the UI thread, which is usually running with lower priority than worker threads of the application.

Since the applications in question are time-critical as I said, even those rare delays are subject of our investigation; however their random character greatly hinders any analysis. It is possible that there is no miracle solution to the problem, but we need at least to be able to determine in each particular case, what was the cause of the delay.

Now my question: is there any way to:

1)      Detect an "abnormal" delay (i.e. determine that the application is not just idle, it's waiting for something).

2)      Determine the cause of such delay.

Obviously either some internal tracing/debugging/hooking or/and usage of an external observing tool is needed. Any ideas and suggestions will be greatly welcomed.

TIA

So you give the UI a lower priority and the UI must be responsive? I think the first thing you need to do is to decide which of those incompatible requirements you want.
 
One very simple diagnostic technique is to use the Task Manager. I keep it running. You can minimize it to the status area and still see a graph that indicates processor utilization. When the UI freezes, glance down at the Task Manager wherever it is to determine if the processor is maxed out. If the UI freezes with the processor maxed out, then you know it is futile to attempt to satisfy both incompatible requirements. If tyhe processor is not busy, then you know you must proceed in a very different direction.

A good suggestion. However since most occasions of freezes occur at our customers' machines, I'm afraid it would be hardly feasible to induce them to occasionally glance at the Task Manager's info. That's why we really need a way of dealing with this in an automated fashion.

Why is the UI thread a lower priority? Does it do a lot of processing?

The UI thread is a lower priority simply because worker threads are more critical. It's alike WM_PAINT having low priority in Windows. It's better for UI to be occasionally unresponsive than for the modules doing business logic.

Note that that does answer my previous request for clarification. Does the UI do a lot of processing? If not then giving it a higher priority won't degrade performance, unless you are concerned about the time for context switches. If it does do a lot of processing then offloading processing from the UI thread to a separate thread could help.

Yeah I missed your second question. In certain situations the UI thread uses a lot of CPU but I don't think that was the case at the times when freezes were observed. Therefore your suggestion is worth consideration.
At the moment I'm thinking about periodically noting down CPU usage with PerformanceCounter, introducing a special thread for that. That way we will perhaps be able to answer the question from your first post, whether the CPU is the bottleneck. Do you see any problems with this approach?
Also, do you know whether there are ways to log context switches and CPU usage by threads? I believe there are tools (ProcessMonitor?) that visualise this information but what we need is logging.

The performance monitor is a COM object or some type of component.