Diagnosing Hangs and Slowness: dotnet-stack, dotnet-trace and Thread Pool Starvation

Version note: The code in this post was built and run with .NET 10 (runtime 10.0.14 in the container); dotnet-stack, dotnet-trace and dotnet-counters 10.0.750501 and, for trace analysis, Microsoft.Diagnostics.Tracing.TraceEvent 3.2.8 were used.

Guess First

The order summary endpoint makes an HTTP (Hypertext Transfer Protocol) call to the inventory service, which answers every request in 200 ms. The endpoint gets the result with .Result instead of await. A single request comes back in 0.37 seconds, so all looks fine. The service runs in a container with 2 CPUs (Central Processing Units) and 200 requests arrive at once. What's the median (p50) response time for those 200 requests: 0.2 seconds, 2 seconds or 20 seconds? And what does the version that uses await do under the same load? Real measurements below; the first answer is none of the three.

What Does "The App Isn't Responding" Mean?

Three different situations can hide behind this complaint, and the tool choice depends on which one it is. If the process is completely frozen (a hang), threads are waiting somewhere: a lock, I/O (Input/Output), another thread. If the process runs but slowly, either the CPU is exhausted or work isn't progressing because threads are waiting. Third, the app may be fine and one of its dependencies is slow; then the problem isn't even in this process.

My order is: first a few seconds of metrics (CPU percentage, how many threads the thread pool has, how many work items wait in its queue), then a snapshot of every thread's stack (dotnet-stack), and last a short, time-boxed trace (dotnet-trace). A dump (post 09) is also an option, but it means a file of hundreds of MB and a short freeze; dotnet-stack answers the same question in seconds, with a few KB. Deep CPU and allocation analysis is out of scope here; that belongs to the upcoming .NET Profiling and Diagnostics series.

The Case: A Service That Chokes Itself with Sync-over-Async

The code is the most common form of sync-over-async. I ran the inventory service in a separate container; the order service is limited with --cpus 2:

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClient<InventoryClient>(client =>
    client.BaseAddress = new Uri(builder.Configuration["Inventory:BaseUrl"]!));

WebApplication app = builder.Build();

// the slow dependency: an inventory endpoint that answers in about 200 ms
app.MapGet("/inventory/{bookId:int}", async (int bookId) =>
{
    await Task.Delay(200);
    return Results.Ok(new StockInfo(bookId, 7));
});

// sync-over-async: blocks a thread pool thread for the whole call
app.MapGet("/orders/{id:int}/summary", (int id, InventoryClient inventory) =>
{
    StockInfo stock = inventory.GetStockAsync(id).Result;
    return Results.Ok(new { id, stock.Available });
});

// the fix: the thread goes back to the pool while the call is in flight
app.MapGet("/orders/{id:int}/summary-async", async (int id, InventoryClient inventory) =>
{
    StockInfo stock = await inventory.GetStockAsync(id);
    return Results.Ok(new { id, stock.Available });
});

app.Run();

public sealed record StockInfo(int BookId, int Available);

public sealed class InventoryClient(HttpClient httpClient)
{
    public async Task<StockInfo> GetStockAsync(int bookId)
    {
        StockInfo? stock = await httpClient.GetFromJsonAsync<StockInfo>($"/inventory/{bookId}");
        return stock!;
    }
}

I generated the load with a small C# console app that keeps 200 concurrent requests in flight with HttpClient for 40 seconds, sending a new one as each finishes. The result for the blocking endpoint:

$ dotnet loadgen/bin/Release/net10.0/LoadGen.dll http://localhost:5420/orders/1/summary 200 40
requests=200 errors=0 rps=4.2
p50=46993ms p95=47198ms p99=47198ms max=47198ms

The median of the 200 requests is 47 seconds. In a 40-second test each worker finished a single request. No errors, no timeouts; the service is "up" but does almost nothing for 47 seconds. If the health check uses the same thread pool, it can't answer during this time either, the orchestrator restarts the pod, and the problem appears to have "fixed itself". The theory is in the C# Async series post "C# Deadlocks, SynchronizationContext and ConfigureAwait". Here we focus on diagnosis: without knowing this, just by looking at the running process, how would you find it?

A Quick Look with dotnet-counters

First look: metrics. I installed the tools into the app image's /tools folder with a multi-stage build, as in post 09, and ran them with docker exec:

docker exec debugwork-app-10 dotnet-counters collect -p 7 \
  --counters 'System.Runtime[dotnet.thread_pool.thread.count,dotnet.thread_pool.queue.length,dotnet.thread_pool.work_item.count]' \
  --refresh-interval 1 --duration 00:00:00:55 -o /tmp/bad.csv

Opening the CSV (Comma-Separated Values) file brought the first surprise: "current value" metrics such as thread count and queue length were written in this version with type Rate, as the change per second. I noticed when the queue length column showed negative values like -5 and -4:

Timestamp,Provider,Counter Name,Counter Type,Mean/Increment
11/08/2026 09:08:00,System.Runtime,dotnet.thread_pool.thread.count ({thread} / 1 sec),Rate,10
11/08/2026 09:08:00,System.Runtime,dotnet.thread_pool.queue.length ({work_item} / 1 sec),Rate,146
11/08/2026 09:08:00,System.Runtime,dotnet.thread_pool.work_item.count ({work_item} / 1 sec),Rate,25
11/08/2026 09:08:01,System.Runtime,dotnet.thread_pool.thread.count ({thread} / 1 sec),Rate,5
11/08/2026 09:08:01,System.Runtime,dotnet.thread_pool.queue.length ({work_item} / 1 sec),Rate,-5

Summing the changes (with a small Python script) gives the absolute values. From the moment the load started, every few seconds:

This is exactly the signature of thread pool starvation: the thread count climbs at a steady rate (about 4 per second here), the queue drains slowly, and completed work per second stays on the floor. The starting value of 2 is no accident: the thread pool's default minimum equals the processor count, and with --cpus 2 the runtime sees 2 processors. Above the minimum, the runtime adds new threads gradually, measuring as it goes. 200 blocked requests need roughly 200 threads, so the wait lasted 47 seconds. The thread pool starvation tutorial on Microsoft Learn gives the same two signals: a slow, steady increase in thread count and CPU usage well below 100 percent.

dotnet-stack: A Snapshot of Every Thread's Stack

The metrics say "threads are waiting for something"; the stack says for what. dotnet-stack takes the managed stack of every thread once, over EventPipe, and prints it. Twelve seconds into the load:

docker exec debugwork-app-10 dotnet-stack report -p 7 > stack-bad.txt
grep -c '^Thread (' stack-bad.txt
grep -c 'b__0_2(int32,class InventoryClient)' stack-bad.txt

69
60

60 of the 69 threads are inside the same lambda. One of their stacks (I cut the middle, Kestrel and thread pool frames):

Thread (0x45):
  [Native Frames]
  System.Private.CoreLib!System.Threading.Monitor.Wait(class System.Object,int32)
  System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.Tasks.Task.SpinThenBlockingWait(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.Tasks.Task.InternalWaitCore(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.Tasks.Task.InternalWait(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.Tasks.Task`1[System.__Canon].GetResultCore(bool)
  Bookstore.Api!Program+<>c.<<Main>$>b__0_2(int32,class InventoryClient)
  Anonymously Hosted DynamicMethods Assembly!dynamicClass.lambda_method2(pMT: 0xffff48e58680,class System.Object,pMT: 0xffff47e5e3a0)
  System.Private.CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch()
  System.Private.CoreLib!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Read it bottom-up: WorkerThreadStart at the bottom means this is a thread pool thread. In the middle is our code, b__0_2 (the third lambda in Program.cs, the /summary handler). At the top, GetResultCore and ManualResetEventSlim.Wait: .Result puts the thread to sleep on an event. Seeing Wait at the top of a thread pool thread's stack is the clearest evidence of sync-over-async. dotnet-stack's advantage is speed: no stopping the process, no files to move, a few seconds. Its drawback is that it's a single moment; if the problem is intermittent, you may miss it.

dotnet-trace collect: A Short, Bounded Recording

Intermittent blocking needs a recording over time. Run without a profile, dotnet-trace turns on two default profiles:

$ dotnet-trace collect -p 7 --duration 00:00:00:05 -o /tmp/sampled.nettrace
No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'dotnet-sampled-thread-time'.
Provider Name                           Keywords            Level               Enabled By
Microsoft-Windows-DotNETRuntime         0x000000100003801D  Informational(4)    --profile
Microsoft-DotNETCore-SampleProfiler     0x0000F00000000000  Informational(4)    --profile
Process        : /usr/share/dotnet/dotnet
Output File    : /tmp/sampled.nettrace
Trace Duration : 00:00:00:05
Stopping the trace. This may take several minutes depending on the application being traced.
Trace completed.

Per the docs, dotnet-sampled-thread-time samples thread stacks at about 1000 Hz, and threads that are waiting rather than using CPU are sampled too. That makes it good for diagnosing waits. On Windows you open it in PerfView or Visual Studio; on Linux and macOS dotnet-trace report gives a quick summary. I copied the file to the Mac and used the local tool:

$ dotnet tool run dotnet-trace report sampled.nettrace topN -n 20 --inclusive | grep -E "Top|Program|GetResultCore|Monitor.Wait"
Top 20 Functions (Inclusive)                                                  Inclusive           Exclusive
11. Api!Program+<>c.<<Main>$>b__0_2(int32,class InventoryClient)              87.08%              0%
12. __Canon].GetResultCore(bool)                                              87.07%              0%
18. Monitor.Wait(class System.Object,int32)                                   40.39%              40.39%

In 87 percent of samples our handler and GetResultCore are on the stack. The threads spend almost all their time waiting there.

For more targeted evidence there are the WaitHandleWait events introduced in .NET 9: they're emitted when a thread blocks on something like Task.Result, Task.Wait, lock or ManualResetEventSlim.Wait. A 10-second trace with only these events:

docker exec debugwork-app-10 dotnet-trace collect -p 7 \
  --clrevents waithandle --clreventlevel verbose --duration 00:00:00:10 -o /tmp/bad.nettrace

The docs suggest opening this file in PerfView or the community-built .NET Events Viewer. There's no Windows on this machine, so I counted the events myself with the TraceEvent library. A short console app that attributes each wait event to a frame from our assembly if its stack has one, otherwise to the thread's entry point:

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Etlx;

string etlxPath = TraceLog.CreateFromEventPipeDataFile(args[0]);
using TraceLog log = new(etlxPath);

Dictionary<string, int> waitsByOrigin = [];
int total = 0;

foreach (TraceEvent e in log.Events.Where(e => e.EventName == "WaitHandleWait/Start"))
{
    total++;
    string origin = FindOrigin(e.CallStack());
    waitsByOrigin[origin] = waitsByOrigin.GetValueOrDefault(origin) + 1;
}

Console.WriteLine($"WaitHandleWait/Start events: {total}");
foreach ((string origin, int count) in waitsByOrigin.OrderByDescending(p => p.Value))
{
    Console.WriteLine($"{count,5}  {origin}");
}

static string FindOrigin(TraceCallStack? stack)
{
    string entry = "(no stack)";
    for (TraceCallStack? frame = stack; frame is not null; frame = frame.Caller)
    {
        string method = frame.CodeAddress.FullMethodName;
        // if a frame from our code is present, attribute the wait to it
        if (frame.CodeAddress.ModuleName == "Bookstore.Api")
        {
            return method;
        }

        // otherwise keep the method at the bottom of the stack that started the thread
        if (!method.StartsWith("System.Threading.Thread.StartCallback", StringComparison.Ordinal))
        {
            entry = method;
        }
    }

    return $"[thread entry] {entry}";
}

WaitHandleWait/Start events: 175
   57  [thread entry] System.Threading.PortableThreadPool+GateThread.GateThreadStart()
   39  Program+<>c.<<Main>$>b__0_2(int32,class InventoryClient)
   39  [thread entry] Microsoft.Extensions.Logging.Console.ConsoleLoggerProcessor.ProcessLogQueue()
   20  [thread entry] System.Diagnostics.Tracing.CounterGroup.PollForValues()
   10  [thread entry] Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()
   10  [thread entry] System.Diagnostics.Metrics.AggregationManager.CollectWorker()

Two details matter. First, most wait events come from dedicated threads doing their own job (the thread pool's gate thread, the console logger, the Kestrel heartbeat). The docs say this too: if you're hunting thread pool starvation, only waits on thread pool threads are relevant. The CounterGroup and AggregationManager rows are the dotnet-counters session that was open at the time; the measuring tool shows up in the measurement. Second, the 39 events from the handler: an event is written when a wait begins, so threads that were already blocked before the trace started produce no new events. 39 new waits in 10 seconds matches about 4 new threads per second joining the pool; each new thread blocks on the next /summary request as soon as it arrives.

collect-linux in .NET 10: When and When Not

The collect-linux command that came with dotnet-trace for .NET 10 uses Linux's perf_events infrastructure and user_events instead of EventPipe. The differences from the docs: it can trace all processes on the machine at once, it captures native library and kernel events, and stacks include native frames. The price is the prerequisites: Linux kernel 6.4 or later with CONFIG_USER_EVENTS=y, a mounted tracefs, root privileges (CAP_PERFMON or CAP_SYS_ADMIN), glibc 2.27 or later, and the .NET 10 runtime.

Probe mode, which needs no root, tells you whether a process supports the command. Inside the container:

$ dotnet-trace collect-linux --probe
Probing .NET processes for support of the EventPipe UserEvents IPC command used by collect-linux. Requires runtime '10.0.0' or later.
.NET processes that support the command:
7 dotnet

.NET processes that do NOT support the command:

The process supports it, but the environment doesn't. As root in the container, without extra privileges, I tried the real command:

$ dotnet-trace collect-linux -p 7 --duration 00:00:00:03 -o /tmp/linux.nettrace
No providers, profiles, ClrEvents, or PerfEvents were specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'.

Provider Name                           Keywords            Level               Enabled By
Microsoft-Windows-DotNETRuntime         0x000000100003801D  Informational(4)    --profile

Linux Perf Events                                                               Enabled By
cpu-sampling                                                                    --profile

Output File    : /tmp/linux.nettrace

Error: Tracefs is not accessible: It appears tracefs is not mounted. You can mount it by running mount -t tracefs nodev /sys/kernel/tracing.

Mounting tracefs requires a privileged container, and there's a more basic obstacle here: the kernel in Docker Desktop's VM (virtual machine), 6.12.72-linuxkit, says # CONFIG_USER_EVENTS is not set in /proc/config.gz. So I couldn't take a collect-linux trace in this environment; the feature list above comes from the docs. Rule of thumb: if the target is a Linux server or node, the kernel qualifies and you can reach tracefs as root, use collect-linux for native frames and for "what else was happening on the machine". If you're on a managed container platform, lack the privileges, or managed stacks are enough, use classic collect. For diagnosing thread pool starvation, classic collect was more than enough.

Fix and Verification

The fix is one word: await instead of .Result. Same load, same container, /summary-async:

$ dotnet loadgen/bin/Release/net10.0/LoadGen.dll http://localhost:5420/orders/1/summary-async 200 40
requests=38937 errors=0 rps=968.4
p50=204ms p95=210ms p99=222ms max=376ms

The median dropped from 47 seconds to 204 ms, which is the dependency's own 200 ms. In the same 40 seconds, 38,937 requests finished instead of 200. The counters showed the thread pool staying at 4 threads with a queue of 0; dotnet-stack showed 15 threads instead of 69, and the only frame from our code was the main thread waiting in app.Run(). Verify the fix with the same measurement you used to diagnose it, not with a feeling that "it's fast now": same load, same counters, same stack query.

"Let's set the minimum thread count to 200" (ThreadPool.SetMinThreads) is a painkiller, not a fix. Threads still block; the pool just grows faster. Every thread has its own stack memory, and when the load reaches 400 the same picture comes back.

Production Notes

Running these tools in production is safe, as long as you keep them bounded:

• Always pass --duration. A forgotten trace session fills the disk. My 5- and 10-second trace files were 4.4 MB each; on a busy service the same duration produces much more.
• Per the docs, the in-memory buffer defaults to 256 MB and the default mode is Drop: if events can't reach disk in time they're dropped and the app isn't slowed down. There's a Block mode to avoid losing events, but it requires a .NET 11 target runtime and can slow the app; I wouldn't turn it on in production.
• Stopping a trace can take minutes: the runtime sends type and method information for all managed code captured. Once the command says "Stopping the trace", be ready to wait and don't kill the process.
• Sampling isn't free: dotnet-sampled-thread-time takes every thread's stack every millisecond, which you'll feel in a process with 200 threads. Start with a narrow selection such as --clrevents waithandle.
• The tools must run as the same user as the target process (or root) and, on Linux, with the same TMPDIR; otherwise the connection times out. docker exec into the same container is the easiest route.
• Put dotnet-stack and dotnet-counters in the on-call runbook; dotnet-trace too, together with its duration and profile limits.

Common Pitfalls

Saying "it's not a performance problem" because CPU is low. In this case the CPU was idle; the problem was waiting. Low CPU together with high latency should make blocked threads your first suspect.

Taking the thread and queue values in the dotnet-counters collect CSV as absolute values. In this version they were written as per-second changes; sum them before plotting, or watch live with monitor.

Blaming every wait event in the trace. The console logger, the gate thread and the measuring tool's own threads always wait. Separate the thread pool threads and the stacks that contain your own code.

Ruling out an intermittent problem with a single dotnet-stack output. A clean snapshot at one moment doesn't mean there's no problem; for intermittent blocking, take a time-boxed trace.

Closing a pod that the health check restarted as "it recovered by itself". A restart resets the thread pool and erases the evidence. Write "dotnet-stack or a dump before the restart" into the runbook.

A Small Experiment

★☆☆ Start the same service with --cpus 8 and put the same load on the blocking endpoint. Where does the thread count start, and how much does p50 change? Relate the result to the default minimum thread count.

★★☆ Add another endpoint that uses Task.Run(() => inventory.GetStockAsync(id)).Result instead of .Result and repeat the measurements. How does the top of the stack change in the dotnet-stack output, and what happens to the number of wait events?

★★★ Extend the WaitReport app to count only waits on thread pool threads, i.e. stacks with WorkerThreadStart at the bottom, and to report the wait duration for each one (by matching WaitHandleWait/Stop).

Dumps and traces gave us the technical answer to "what happened". But at the end of every incident a harder question remains: why did it happen, and what has to change so it doesn't happen again? In the next post we move on to root cause analysis with 5 Whys, fishbone diagrams and blameless postmortems.