What Is a Thread? Kernel Threads, Stacks, Thread-Local Storage and the OS View of a .NET Thread

Version note: The code in this post was built and run with .NET 10; tests use xUnit v3 4.0.3. Linux observations were made with ps and /proc in a mcr.microsoft.com/dotnet/sdk:10.0 container based on Ubuntu 24.04, on a Linux 6.x kernel. The Windows and macOS tools are described from their official documentation.

Guess First

An old module of Bookstore, the online bookstore used as the shared example across this site's series, exports institutional orders by opening a new Thread for every request. With 1000 requests waiting at the same time, how much virtual memory do these threads add to the process, and how much real memory? If the same 1000 waits were 1000 Tasks using await, how many OS threads would they need?

Second question: at the start of a method you write "Ada" into a [ThreadStatic] field and into an AsyncLocal<string>, then await Task.Delay(50). What do you read from each after the await? In this post we answer with real measurements and look at what a .NET thread corresponds to in the kernel.

Process and Thread: What Do They Share?

In the previous post we defined a process as an address space plus resources. A thread is the unit that runs code inside that process, and what the scheduler puts on a CPU is a thread, not a process. Threads in the same process share the address space, the heap, open files and the environment. What each thread owns is small but important: its own stack, its own register set, including the program counter pointing at the next instruction and the stack pointer, its own thread-local storage and its own state in the scheduler's eyes.

The Linux kernel does not treat a thread as a separate concept but as a task that shares its resources with others. Processes and threads are created with the same clone syscall; the difference is in the flags that say which resources are shared. In the second post we saw CLONE_VM|CLONE_VFORK for a child process; when creating a thread, the runtime uses CLONE_THREAD together with flags that share the address space, the file table and the signal handlers. That is why every thread has its own ID, the TID, and the main thread's TID equals the process's PID. On Windows a thread is a separate kernel object inside the process; the logic is the same.

The Threads of a .NET Process

I added a command to the lab application that opens a named thread and sends a piece of work to the thread pool. ThreadLab.Describe prints the managed thread ID and, on Linux, the OS thread ID read from the /proc/thread-self symlink:

case "threads":
{
    ThreadLab.Describe("main");
    Thread exporter = new(() =>
    {
        ThreadLab.Describe("OrderExporter");
        Thread.Sleep(TimeSpan.FromSeconds(int.Parse(args[1])));
    }) { Name = "OrderExporter" };
    exporter.Start();
    await Task.Run(() => ThreadLab.Describe("thread pool"));
    Console.WriteLine($"pid={Environment.ProcessId}, sleeping {args[1]} s");
    exporter.Join();
    break;
}

// On Linux, /proc/thread-self is a symlink to "<pid>/task/<tid>"
public static string OsThreadId() =>
    OperatingSystem.IsLinux()
        ? Path.GetFileName(new FileInfo("/proc/thread-self").LinkTarget ?? "?")
        : "n/a";

dotnet Bookstore.OsLab.dll threads 15

main           managed id=1   os tid=7533
OrderExporter  managed id=4   os tid=7542
thread pool    managed id=5   os tid=7543
pid=7533, sleeping 15 s

Three things stand out. The main thread's OS thread ID equals the PID: 7533. The managed IDs, 1, 4 and 5, are the runtime's own numbers and have nothing to do with the OS IDs. Behind every managed thread there is an OS thread; in .NET this mapping is one to one. While the process runs, let's look at the thread list from a second terminal with ps -T:

ps -T -p 7533 -o pid,spid,stat,comm

    PID    SPID STAT COMMAND
   7533    7533 Sl   dotnet
   7533    7535 Sl   .NET SynchManag
   7533    7536 Sl   .NET EventPipe
   7533    7537 Sl   .NET DebugPipe
   7533    7538 Sl   .NET Debugger
   7533    7539 Sl   .NET Finalizer
   7533    7540 Sl   .NET Tiered Com
   7533    7541 Sl   .NET SigHandler
   7533    7542 Sl   OrderExporter
   7533    7543 Sl   .NET TP Worker
   7533    7544 Sl   .NET TP Gate
   7533    7545 Sl   .NET TP Worker

The SPID column is the TID. The only thread we opened is OrderExporter; Thread.Name was passed on to the OS too and fits within Linux's 15-character thread name limit. The others belong to the runtime: Finalizer runs finalizers, Tiered Com, its name cut at 15 characters, is the tiered compilation thread that recompiles methods with optimizations in the background, SigHandler receives Unix signals, the TP Worker threads are the thread pool's workers, and TP Gate is the thread pool helper that checks whether a new thread is needed. EventPipe, DebugPipe and Debugger are for diagnostics and debugger support, and SynchManag belongs to the PAL layer that emulates Win32 synchronization behavior on Linux. So even a "single-threaded" console app runs with twelve OS threads. That is why naming threads matters: in a thread dump or in top -H output you recognize yours by name.

On Windows, the Sysinternals PsList tool lists the process's threads in detail with pslist -d dotnet (without thread names), and Process Explorer shows them on the Threads tab of the process properties. On macOS, ps -M <pid> lists a process's threads, and the Threads column in Activity Monitor shows the count. These tools were not run for this post.

The Stack: How Big Is It, and Who Decides?

We said every thread has its own stack; local variables, return addresses of method calls and parameters live there. So how much memory does opening a thread cost? The measurement that mimics Bookstore's old module keeps 1000 threads waiting on a ManualResetEventSlim and reads the growth in the process's virtual and resident memory from the Process class:

public static ThreadCost Measure(int count, int maxStackSize = 0)
{
    using ManualResetEventSlim release = new();
    using Process self = Process.GetCurrentProcess();
    long virtualBefore = self.VirtualMemorySize64;
    long residentBefore = self.WorkingSet64;

    Stopwatch sw = Stopwatch.StartNew();
    List<Thread> threads = new(count);
    for (int i = 0; i < count; i++)
    {
        // Mimics an old-style server that opens one thread per request
        Thread t = new(() => release.Wait(), maxStackSize) { IsBackground = true, Name = $"order-{i}" };
        t.Start();
        threads.Add(t);
    }
    double elapsed = sw.Elapsed.TotalMilliseconds;

    self.Refresh();
    ThreadCost cost = new(count, self.Threads.Count, elapsed,
        (self.VirtualMemorySize64 - virtualBefore) / 1024 / 1024,
        (self.WorkingSet64 - residentBefore) / 1024 / 1024);

    release.Set();
    foreach (Thread t in threads)
    {
        t.Join();
    }
    return cost;
}

dotnet Bookstore.OsLab.dll thread-cost 1000
dotnet Bookstore.OsLab.dll thread-cost 1000 256
MALLOC_ARENA_MAX=1 dotnet Bookstore.OsLab.dll thread-cost 1000
MALLOC_ARENA_MAX=1 dotnet Bookstore.OsLab.dll thread-cost 1000 256

1000 threads (maxStackSize=default): 70.1 ms, os threads=1007, virtual +12762 MB, resident +20 MB
1000 threads (maxStackSize=256 KB): 81.8 ms, os threads=1007, virtual +5012 MB, resident +20 MB
1000 threads (maxStackSize=default): 70.7 ms, os threads=1007, virtual +8092 MB, resident +20 MB
1000 threads (maxStackSize=256 KB): 69.8 ms, os threads=1007, virtual +342 MB, resident +20 MB

The first line is alarming: 12.7 GB of virtual memory for 1000 threads. But that number is the sum of two different things. The first is stacks: according to glibc's pthread_create(3) man page on Linux, a new thread's default stack size comes from the RLIMIT_STACK limit in effect when the process started, that is ulimit -s, which is 8192 KB in the container. The .NET threading configuration documentation says the same: the default stack is 8 MB on Linux and 1.5 MB on Windows and macOS. The second is glibc's malloc: in multithreaded programs it opens extra malloc arenas so threads do not wait for each other, and each reserves a large address range; according to mallopt(3) their number can be limited with MALLOC_ARENA_MAX. With arenas reduced to one, the table becomes clear: 8092 MB with the default stack, 8 MB per thread, and 342 MB with a 256 KB stack.

The column that really matters is the last one: in all four cases resident memory grows by 20 MB, about 20 KB per thread. Virtual memory is a promise, resident memory is actual use; only the touched pages of a stack come into physical memory. We will look at this distinction in detail in the virtual memory post. If you want to change the default there are two ways: new Thread(work, maxStackSize) for a single thread, and for all runtime threads DOTNET_Thread_DefaultStackSize, which takes a hex value and came with .NET 10, or System.Threading.DefaultStackSize in runtimeconfig. The price of a smaller stack is a StackOverflowException in deep recursion; in .NET this exception cannot be caught and terminates the process.

Kernel Threads, Green Threads and M:N

The one-to-one model, one language thread per kernel thread, is simple: a blocking syscall stops only that thread, and the scheduler manages real parallelism. Its price is the memory we just saw and a trip to the kernel on every switch. Alternatives have been tried over the years. Green threads, M:1, ran all language threads on a single kernel thread with the language's own scheduler; a blocking syscall stopped everyone. The common approach today is M:N: many lightweight threads run on a small number of kernel threads.

Go does this with goroutines. According to the Go FAQ, a goroutine starts with a stack of a few kilobytes that grows and shrinks as needed, and when a goroutine blocks in a syscall the runtime moves the others on the same OS thread to another thread. Virtual threads, made permanent in Java with JDK 21 through JEP 444, are the JVM's version of the same idea: a virtual thread detaches from its carrier platform thread during a blocking operation; the JEP separately describes pinning, where this is not possible inside a synchronized block or a native call.

.NET solved the waiting problem at the language level instead: async/await. A waiting method holds no thread; the state machine generated by the compiler sits on the heap, and when the I/O completes a thread pool thread resumes it. The .NET team tried green threads as an experiment in 2023; according to the results note in dotnet/runtimelab, the experiment was put on hold because of concerns about introducing a new programming model, and the team decided to keep improving the existing async model. The same note says that a request waiting for I/O costs around 100 bytes in async C#, and kilobytes with a blocked OS thread. Let's measure it:

public static async Task<ThreadCost> MeasureTasksAsync(int count)
{
    TaskCompletionSource release = new(TaskCreationOptions.RunContinuationsAsynchronously);
    using Process self = Process.GetCurrentProcess();
    long virtualBefore = self.VirtualMemorySize64;
    long residentBefore = self.WorkingSet64;

    Stopwatch sw = Stopwatch.StartNew();
    Task[] waiters = new Task[count];
    for (int i = 0; i < count; i++)
    {
        // A waiting operation is a small state machine on the heap, not a thread
        waiters[i] = WaitAsync(release.Task);
    }
    double elapsed = sw.Elapsed.TotalMilliseconds;

    self.Refresh();
    ThreadCost cost = new(count, self.Threads.Count, elapsed,
        (self.VirtualMemorySize64 - virtualBefore) / 1024 / 1024,
        (self.WorkingSet64 - residentBefore) / 1024 / 1024);

    release.SetResult();
    await Task.WhenAll(waiters);
    return cost;
}

private static async Task WaitAsync(Task signal) => await signal;

dotnet Bookstore.OsLab.dll task-cost 1000
dotnet Bookstore.OsLab.dll task-cost 100000

1000 awaiting tasks: 0.8 ms, os threads=7, virtual +0 MB, resident +3 MB
100000 awaiting tasks: 17.6 ms, os threads=7, virtual +0 MB, resident +17 MB

Here is the answer to the first question. 1000 threads: around 70 ms, 1007 OS threads, gigabytes of virtual memory and 20 MB resident. 100,000 waiting tasks: 17.6 ms, seven OS threads, no change in virtual memory and about 170 bytes per wait. We covered the API side of async in the published C# Asynchronous Programming series; what you see here is that API's counterpart at the OS level.

Thread-Local Storage: ThreadStatic, ThreadLocal and AsyncLocal

The OS gives every thread an area to keep its own data: thread-local storage. The same variable name points to a different value on each thread. .NET has two direct counterparts: a static field with the [ThreadStatic] attribute, and ThreadLocal<T>, which produces each thread's initial value with a factory. AsyncLocal<T>, on the other hand, is tied to the logical flow rather than the thread; its value travels with the ExecutionContext to whichever thread the await resumes on:

public static class ContextDemo
{
    [ThreadStatic] private static string? s_threadStaticUser;
    private static readonly AsyncLocal<string?> s_asyncLocalUser = new();

    public static async Task<string> RunAsync()
    {
        s_threadStaticUser = "Ada";
        s_asyncLocalUser.Value = "Ada";
        int before = Environment.CurrentManagedThreadId;

        await Task.Delay(50);

        int after = Environment.CurrentManagedThreadId;
        return $"thread {before} -> {after}: ThreadStatic={s_threadStaticUser ?? "null"}, AsyncLocal={s_asyncLocalUser.Value ?? "null"}";
    }
}

dotnet Bookstore.OsLab.dll context

thread 1 -> 5: ThreadStatic=null, AsyncLocal=Ada

The answer to the second question: before the await the code was on thread 1, after it on thread pool thread 5. The [ThreadStatic] value stayed on thread 1 and is null on the new thread; the AsyncLocal value came along with the flow. These behaviors are pinned down by xUnit tests: a [ThreadStatic] value is not visible on another thread, and an AsyncLocal value flows to another thread through Task.Run. The rule is simple: if the code has an await, tie request context such as the user, tenant or correlation ID to the flow, not to the thread.

Production Notes

Watch the thread count. The Threads line in /proc/<pid>/status, the Threads column in Task Manager on Windows and the thread pool counters in dotnet-counters answer the same question. A thread count that keeps growing over time is almost always a leak: timers that are never disposed, background loops that never end, or threads opened on every request. The thread pool starvation we covered in the hang diagnosis post of the Debugging and Problem Solving series also shows up in these counters.

In a container, threads count too. On Linux a thread is a task, so the pids controller of cgroup v2 limits processes and threads together; once the container's pids.max limit is reached, no new thread can be opened. The 8 MB of virtual stack per thread is usually not a problem, because the container memory limit looks at resident memory; but on 32-bit systems or where a virtual memory limit is set, it is what decides the thread count. The mechanism behind container limits is in the container post of this series.

Common Pitfalls

Thinking threads are cheap. Code that opens a new thread per request works at low traffic; under load both memory and the context switch cost we will measure in the next post slow it down. If a long-running, CPU-heavy job really needs its own thread, there is TaskCreationOptions.LongRunning so it does not clog the thread pool; waiting work should be async.

Using [ThreadStatic] in async code. As in the output above, the value is lost after await, or worse, you see the value of another request that used the same thread earlier. Thread pool threads are reused; leaving a [ThreadStatic] field without clearing it carries one request's data into another.

Mistaking the managed thread ID for the OS thread ID. Environment.CurrentManagedThreadId is the runtime's number; it does not match the TID you see in ps -T, top -H or a native profiler. Naming threads is the easiest way to match the two.

A Small Experiment

★☆☆ Open a subshell with ulimit -s 2048 and run thread-cost 1000 in it. Explain why the virtual memory growth is smaller by running it together with MALLOC_ARENA_MAX=1.

★★☆ In the threads command, instead of Task.Run start 20 Task.Run(() => Thread.Sleep(2000)) calls that take a few seconds, and watch with ps -T how the number of .NET TP Worker threads grows. It shows what blocking code does to the thread pool.

★★★ Measure how deep your own recursive method can go on a thread with a 256 KB stack, then repeat with the default stack. Interpret the result as bytes per stack frame and watch how a StackOverflowException terminates the process.

In this post we saw that a console app has twelve threads, and that opening 1000 threads takes about a hundred times longer than 1000 tasks. All these threads share a limited number of cores. In the next post we look at the scheduler that does the sharing: which thread runs when, how long a context switch takes, what Linux's EEVDF changed, and why a CPU limit in a container turns into throttling?