What Is a Process? Address Space, PID, Exit Codes and Running Child Processes in .NET

Version note: The code in this post was built and run with .NET 10; tests use xUnit v3 4.0.3. Linux observations were made with ps, /proc and strace 6.8 in a mcr.microsoft.com/dotnet/sdk:10.0 container based on Ubuntu 24.04, on a Linux 6.x kernel. Windows behavior is based on Microsoft Learn documentation and the dotnet/runtime source code.

Guess First

Bookstore, the online bookstore used as the shared example across this site's series, does not build its sales report itself; it calls a command-line tool the team wrote, using Process.Start. The tool writes the CSV to stdout and "this many rows written" messages to stderr. In the test environment everything is fine with a 1000-row report. In production, a 5000-row report request never returns: zero CPU, no error, no log, both processes just sit there. Why?

Second question: when you cut this hang short with a timeout and kill the tool, what does Process.ExitCode return on Linux? And on Windows? The answers come from looking at how the kernel sees a process.

What Is a Process?

A program is a file on disk: Bookstore.OsLab.dll and the dotnet host that runs it. A process is a running instance of that program, and the kernel gives it three things. The first is an address space of its own: every memory address the process sees belongs to it alone, and even if another process uses the same address it lands in different physical memory. The second is resources: the list of open files and sockets, which is the file descriptor table on Linux and the handle table on Windows, environment variables, a working directory and the user it runs as. The third is at least one thread; threads are what actually run the code and what the scheduler puts on a CPU, and they are the topic of the next post. The kernel identifies a process by its PID.

On Linux you can read all of this as files under /proc/<pid>/. I added a command to the lab application that prints its PID and waits:

case "proc-info":
    Console.WriteLine($"pid={Environment.ProcessId}, sleeping {args[1]} s");
    Thread.Sleep(TimeSpan.FromSeconds(int.Parse(args[1])));
    break;

This time start the container with a name and with --init; --init runs a small init process as PID 1 in the container, and we will see why in the zombie section. Run the application in one terminal and enter the same container from a second one with docker exec -it oslab bash:

docker run --rm -it --init --name oslab --cap-add=SYS_PTRACE -v ${PWD}:/src -w /src mcr.microsoft.com/dotnet/sdk:10.0 bash
dotnet publish src/Bookstore.OsLab -c Release -o /app && cd /app
dotnet Bookstore.OsLab.dll proc-info 30

grep -E "^(Name|State|Tgid|PPid|Threads|VmSize|VmRSS)" /proc/1133/status

Name:	dotnet
State:	S (sleeping)
Tgid:	1133
PPid:	1131
VmSize:	17838308 kB
VmRSS:	   27356 kB
Threads:	8

Even for a process that has not run a line of our code yet, the table is full. PPid is the PID of the shell that started it; every process has a parent. Threads: 8 means the runtime has opened seven more threads without us writing a single one; those were the clone calls in the first post. VmSize is close to 17 GB while VmRSS is 27 MB. The difference between the address range a process has reserved and the part actually sitting in physical memory is the topic of the virtual memory post; for now, know that most of those 17 GB are an address range, not RAM. The file descriptor table is under /proc/1133/fd:

ls -l /proc/1133/fd

0 -> /dev/null
1 -> /tmp/pi.txt
2 -> pipe:[1119218]
...
10 -> socket:[1123811]
12 -> /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.15/System.Private.CoreLib.dll

There are 32 descriptors in total. 0, 1 and 2 are stdin, stdout and stderr in every process; here stdout has been redirected to a file. The socket is the IPC channel .NET opens for its diagnostics tools; dotnet-counters and dotnet-trace connect to the process through it. Most of the rest are opened assembly files.

Inside the Address Space: /proc/<pid>/maps

An address space is not an empty area; the kernel divides it into regions, and every region has permissions. /proc/<pid>/maps lists these regions. Lines I picked from the same process's 240-line list:

grep -E "Bookstore.OsLab.dll|libSystem.Native.so|\[heap\]|\[stack\]" /proc/1133/maps

aaaaff3b2000-aaaaff52a000 rw-p 00000000 00:00 0                          [heap]
ffffa5280000-ffffa528b000 r--p 00000000 00:42 217758                     /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.15/libSystem.Native.so
ffffa529a000-ffffa52a7000 r-xp 0000a000 00:42 217758                     /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.15/libSystem.Native.so
ffffa52b6000-ffffa52b8000 r--p 00016000 00:42 217758                     /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.15/libSystem.Native.so
ffffa52c7000-ffffa52c8000 rw-p 00017000 00:42 217758                     /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.15/libSystem.Native.so
ffffa5b8a000-ffffa5b90000 r--s 00000000 00:42 438454                     /app/Bookstore.OsLab.dll
fffff1ee4000-fffff1f05000 rw-p 00000000 00:00 0                          [stack]

Each line is an address range, permissions, an offset into the file and the file that is mapped. The permissions read as written: r read, w write, x execute, and the last letter is p for private, a copy belonging to this process, or s for shared. The same native library is split into four parts: the r-xp part is the code region, executable but not writable; r--p holds constant data; rw-p is the library's writable global data. The code, data, heap and stack regions of the classic textbook picture are here with real addresses. This output comes from an arm64 Linux, and the address layout depends on the CPU architecture: on x64 Linux user space ends at 0x00007fffffffefff, so shared libraries and the stack usually sit at addresses starting with 7f, while the program and [heap] start with 55 or 56; the ffff… and aaaa… addresses above are their arm64 counterparts.

[heap] is the C side's malloc heap, not the .NET managed heap. [stack] is the main thread's stack. Bookstore.OsLab.dll was not read like a file; it was mapped into the address space, which is how the runtime loads assemblies. The managed heap is in these lines of the list:

fffb7b401000-ffff4c000000 ---p 00000000 00:00 0
ffff66690000-ffffa5200000 ---p 00000000 00:00 0

Two huge regions with no permissions at all, ---p, and no file behind them: one about 15 GB, the other 1 GB. The first is the address range the GC reserved in advance for the managed heap. That is easy to verify: when I started the process with DOTNET_GCRegionRange=100000000, which is hex for 4 GB, the same region shrank to 4 GB and VmSize dropped from 17.8 GB to 5.9 GB. The GC makes pages of this range usable as it needs them. The addresses change on every run and will be different on your machine; there is a security reason for that, which we will discuss in the process memory post.

How a Process Is Born: fork, exec and CreateProcess

On Unix a new program starts in two steps. fork creates an almost exact copy of the calling process; then the child discards its address space with execve and loads the new program in its place. In the short window between the two, the child can apply settings such as file redirections and environment to itself; that is how > redirection works in a shell. Windows has no such split: CreateProcess creates a new process and loads the given program into it in one call; there is no copied parent, and parameters say what the child gets.

strace shows what .NET does on Linux. I produced the report with 10 rows and traced only the syscalls related to process management, leaving out the clone calls that create threads and a few unrelated lines:

strace -f -qq -e trace=clone,clone3,execve,wait4,waitid,exit_group dotnet Bookstore.OsLab.dll report 10

384   clone(child_stack=0xffffc082e710, flags=CLONE_VM|CLONE_VFORK|SIGCHLD <unfinished ...>
392   execve("/usr/share/dotnet/dotnet", ["dotnet", "/app/Bookstore.ReportTool.dll", "sales", "10"], 0xaaab154548c0 /* 19 vars */ <unfinished ...>
384   <... clone resumed>)              = 392
392   <... execve resumed>)             = 0
392   exit_group(0 <unfinished ...>
404   --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=392, si_uid=0, si_status=0, si_utime=0, si_stime=1 /* 0.01 s */} ---
391   waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=392, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WNOWAIT, NULL) = 0
391   wait4(392, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 392

The CLONE_VM|CLONE_VFORK flags say this is not a classic fork but a vfork: the child shares the parent's memory without copying it, and the calling thread waits until the child calls execve. According to the comment in the runtime source, the gain comes from the kernel not having to mark every writable page of the parent as copy-on-write; this path is not used on macOS. The child turns into the dotnet host with execve, does its work and exits. The kernel sends the parent SIGCHLD, and a .NET thread collects the child's exit status with wait4.

Process States, Zombies and Orphans

A process goes through several states during its life, and the STAT column of ps shows the state as a single letter: R running or waiting for a CPU, S waiting for an event, asleep but wakeable by a signal, D waiting uninterruptibly, usually for disk I/O, T stopped, and Z zombie. The proc-info process above was in S because it was inside Thread.Sleep.

A zombie is a process that has exited but whose exit code has not been read by its parent yet. Its memory and files are released; the kernel keeps only its PID and exit status, and the record disappears when the parent calls wait. A parent that never waits for its child is enough to see one:

sh -c "sleep 1 & exec sleep 20" &
sleep 3
ps -o pid,ppid,stat,args --ppid $(pgrep -f "^sleep 20") -p $(pgrep -f "^sleep 20")

    PID    PPID STAT COMMAND
    416     410 S    sleep 20
    418     416 Z    [sleep] <defunct>

The shell started sleep 1 in the background and turned itself into sleep 20 with exec. sleep 20 does not know it has a child and never waits for it, so sleep 1, which finishes after a second, stays around as Z. A single zombie is harmless; thousands of them fill the PID table. This does not happen with a .NET parent, because, as we saw in strace, the runtime reaps its children itself when SIGCHLD arrives.

An orphan is the opposite: a process whose parent died before it. The kernel attaches it to PID 1, or to an ancestor marked as a subreaper:

sh -c "sleep 15 & echo parent=\$\$ child=\$!; exit"
ps -o pid,ppid,stat,args -p $(pgrep -f "^sleep 15")

parent=422 child=423
    PID    PPID STAT COMMAND
    423       1 S    sleep 15

Since the new parent is PID 1, reaping the orphan's zombie becomes its job too. On a normal Linux machine PID 1 is systemd, and it does this. In a container, PID 1 is your application, which is why --init exists. We will discuss the signal side of this in the I/O models and IPC post. Windows does not have zombies in this form, but it has something similar: according to Microsoft's documentation, when a process ends, the process object lives until no handles to it remain open, which is how its exit code can still be read. The same documentation says something else that matters: terminating a process does not terminate the child processes it started. The same is true on Linux, and we will use that shortly.

Bookstore Calls Its Report Tool: The Pipe Deadlock

Back to the opening question. The report tool, Bookstore.ReportTool, writes data to stdout and progress messages to stderr like most real tools:

case "sales":
    int rows = int.Parse(args[1]);
    Console.Out.WriteLine("isbn,title,qty,total");
    for (int i = 1; i <= rows; i++)
    {
        Console.Out.WriteLine($"9780441172719,Dune,{i % 5 + 1},{(i % 5 + 1) * 420.00m:F2}");
        Console.Error.WriteLine($"progress: row {i} of {rows} written");
    }
    if (args.Contains("--fail"))
    {
        Console.Error.WriteLine("error: warehouse data is incomplete");
        return 2;
    }
    return 0;

The first version on the Bookstore side is the one you see in most examples on the internet:

// WRONG: reads stdout to the end first, nobody looks at stderr
public static string RunNaive(int rows)
{
    ProcessStartInfo psi = new("dotnet")
    {
        RedirectStandardOutput = true,
        RedirectStandardError = true,
    };
    foreach (string argument in Args("sales", rows.ToString()))
    {
        psi.ArgumentList.Add(argument);
    }

    using Process process = Process.Start(psi)!;
    string csv = process.StandardOutput.ReadToEnd();
    string log = process.StandardError.ReadToEnd();
    process.WaitForExit();
    return csv;
}

dotnet Bookstore.OsLab.dll report-naive 1000
time (timeout 10 dotnet Bookstore.OsLab.dll report-naive 5000; echo "exit code: $?")

rows=1000
exit code: 124

real	0m10.007s
user	0m0.021s
sys	0m0.009s

1000 rows finish instantly; timeout kills the 5000-row run after 10 seconds, and total CPU use in that time is 21 milliseconds. Looking from the second terminal while it hangs makes the situation clear:

ps -o pid,ppid,stat,wchan:12,args -p 277,285

    PID    PPID STAT WCHAN        COMMAND
    277     276 Sl   pipe_read    dotnet Bookstore.OsLab.dll report-naive 5000
    285     277 Sl   pipe_write   dotnet /app/Bookstore.ReportTool.dll sales 5000

The WCHAN column tells you which kernel function the process is waiting in. The parent is waiting to read from a pipe and the child is waiting to write to one. RedirectStandardOutput and RedirectStandardError create two separate pipes, and a pipe has a limited buffer in the kernel; according to the pipe(7) man page the default capacity on Linux is 65,536 bytes, and a write to a full pipe blocks. The sequence goes like this: the parent starts reading stdout to the end. On every row the child also writes about 35 bytes to stderr, but nobody reads stderr. After roughly 1900 rows the stderr pipe is full and the child's write call blocks. Since the child cannot make progress, it never closes stdout either, so the EOF the parent is waiting for never comes. Two processes waiting for each other, and both are right.

Why is 1000 rows fine? Because stderr adds up to 33,893 bytes, which fits in 64 KB. The bug depends on the size of the data, which is why it does not show up in the test environment. Anonymous pipes on Windows have a limited buffer too; the mechanism is the same. Other ways for processes to communicate are in the I/O models and IPC post.

The Right Way: Reading Both Pipes at Once, a Timeout and the Process Tree

The core of the fix is draining both streams at the same time. The small runner Bookstore uses:

using System.Diagnostics;

namespace Bookstore.OsLab;

public sealed record ProcessResult(int ExitCode, string StdOut, string StdErr, bool TimedOut);

public static class ProcessRunner
{
    public static async Task<ProcessResult> RunAsync(
        string fileName,
        IEnumerable<string> arguments,
        TimeSpan timeout,
        IEnumerable<string>? hiddenVariables = null)
    {
        ProcessStartInfo psi = new(fileName)
        {
            RedirectStandardOutput = true,
            RedirectStandardError = true,
        };
        foreach (string argument in arguments)
        {
            psi.ArgumentList.Add(argument); // no shell; every argument goes through as is
        }
        foreach (string name in hiddenVariables ?? [])
        {
            psi.Environment.Remove(name); // do not let the child inherit it
        }

        using Process process = Process.Start(psi)!;

        // Drain both pipes at once; neither can fill up and block the child
        Task<string> stdout = process.StandardOutput.ReadToEndAsync();
        Task<string> stderr = process.StandardError.ReadToEndAsync();

        bool timedOut = false;
        using (CancellationTokenSource cts = new(timeout))
        {
            try
            {
                await process.WaitForExitAsync(cts.Token);
            }
            catch (OperationCanceledException)
            {
                timedOut = true;
                process.Kill(entireProcessTree: true);
                await process.WaitForExitAsync();
            }
        }

        return new ProcessResult(process.ExitCode, await stdout, await stderr, timedOut);
    }
}

dotnet Bookstore.OsLab.dll report 5000
dotnet Bookstore.OsLab.dll report 20 --fail
dotnet Bookstore.OsLab.dll report-timeout

exit=0 timedOut=False stdout=143021 B stderr=173893 B in 39 ms
last stderr line: progress: row 5000 of 5000 written
exit=2 timedOut=False stdout=593 B stderr=647 B in 14 ms
last stderr line: error: warehouse data is incomplete
exit=137 timedOut=True after 2030 ms

5000 rows now finish in 39 milliseconds; stderr is 173 KB and that is fine. An exit code is the single number a process leaves to its parent, and interpreting it is your job. 0 means success; the meaning of other values is defined by the tool, and ours returns 2 for incomplete warehouse data. report-timeout cuts off a tool that sleeps for 30 seconds after 2 seconds, and the exit code is 137: on Linux, when a process is killed by a signal, .NET follows the same convention as shells and adds the signal number to 128; Kill sends SIGKILL, signal 9, so 128 + 9 = 137. On Windows, according to the dotnet/runtime source, Kill calls TerminateProcess with -1, so you see -1 in the same situation. That is why you should not take shortcuts such as "137 means OOM": 137 only says "killed by SIGKILL", not who killed it.

entireProcessTree: true is not decoration. If the tool starts a process of its own and that grandchild inherits the stdout pipe, killing only the child does not close the pipe and ReadToEndAsync waits forever. I pinned this down with an xUnit test: the test that runs /bin/sh -c "sleep 30 & sleep 30" with a 1-second timeout waited 30 seconds and failed with entireProcessTree: false, and passed with true. The runner's six tests cover large output, exit codes, timeouts, grandchild processes, arguments and environment behavior:

dotnet test --solution Bookstore.OsLab.slnx -c Release

Test run summary: Passed!
  total: 7
  failed: 0
  succeeded: 7
  skipped: 0

Production Notes

Put a timeout on every external process call. If the external tool hangs, your request hangs too, and in ASP.NET Core every waiting request holds a connection and memory. Combining the timeout with the request's own CancellationToken makes the tool stop as well when the client gives up.

ReadToEndAsync loads all output into memory. 143 KB for 5000 rows is not a problem, but if the tool can produce gigabytes, copy stdout straight to a file or to the response stream and keep only the tail of stderr.

Starting a process is an expensive chain of syscalls: a new address space, loading the program and, for a .NET tool, a runtime startup. Do not start a process on every request of an endpoint that serves hundreds of requests per second; if possible, do the work in-process as a library, or limit the number of concurrently running tools with SemaphoreSlim. Make sure the tool is really installed in the container image and on the PATH; the image side is a topic for the upcoming Docker series.

Common Pitfalls

Shell injection. If you embed a value coming from the user into a shell command as a string, a ;, && or $( ) in that value becomes a command to the shell. Picture an endpoint that builds a report by author name; I gave the same input to the tool in two ways:

// WRONG: user input is embedded in a shell command
ProcessResult viaShell = await ProcessRunner.RunAsync("/bin/sh",
    ["-c", $"dotnet {ReportTool.DllPath} args {author}"], TimeSpan.FromSeconds(10));

// RIGHT: no shell; author goes through as a single argument
ProcessResult direct = await ProcessRunner.RunAsync("dotnet",
    ReportTool.Args("args", author), TimeSpan.FromSeconds(10));

dotnet Bookstore.OsLab.dll shell-demo "Ada; echo INJECTED: the shell ran a second command"

via /bin/sh -c:
[Ada]
INJECTED: the shell ran a second command
via ArgumentList:
[Ada; echo INJECTED: the shell ran a second command]

In the first path the shell ran the input as two commands; instead of a harmless echo it could have been any command. With ArgumentList no shell is involved, and the value reaches the child's argv in one piece. cmd.exe /c on Windows carries the same risk. ProcessStartInfo.Arguments, which takes a single string, is not a safe alternative either: even without running a shell, it splits the string by space and quoting rules, so "Ada Lovelace" becomes two arguments.

Environment leaking to the child. A child inherits all of its parent's environment variables, including connection strings and API keys:

ConnectionStrings__BookstoreDb="Server=sql;Database=BookstoreDb;Password=<secret>" dotnet Bookstore.OsLab.dll env-demo

default:  ConnectionStrings__BookstoreDb=Server=sql;Database=BookstoreDb;Password=<secret>
hidden:   ConnectionStrings__BookstoreDb is not set

When you call an external tool, especially a third-party one, remove the variables it does not need with psi.Environment.Remove. Open files are less of a problem: since .NET opens files with O_CLOEXEC, the descriptors we saw in the first post are not passed to the child.

Calling WaitForExit before reading. This is the shortest form of the pipe deadlock: the parent waits for the exit, the child waits to write to the full pipe. Start reading first, then wait. Finally, dispose the Process object with using; there is an OS handle and pipes behind it.

A Small Experiment

★☆☆ Start report-naive at 1000 and increase the row count to find the first value where it hangs. Multiply by the stderr line length and compare with 65,536.

★★☆ Add an external CancellationToken parameter to ProcessRunner and combine it with the timeout token through CancellationTokenSource.CreateLinkedTokenSource. Verify with an xUnit test that the tool is killed on external cancellation too.

★★★ While proc-info runs, group the lines of /proc/<pid>/maps by permission and write a small C# command that sums the sizes of the r-xp, rw-p and ---p regions. Compare the total with VmSize.

In this post /proc/<pid>/status told us there are eight threads, and every strace line started with a different number. In the next post we look at who owns those numbers: what a thread is, what a .NET thread corresponds to in the kernel, how big its stack is, and why [ThreadStatic] and AsyncLocal<T> behave differently?