Deep Analysis with PerfView: GC Stats, JIT Stats, Grouping/Folding and Diffing Two Traces

Version note: The code in this post was built and run with .NET 10 (runtime 10.0.15 in the container), dotnet-trace 10.0.750501, Microsoft.Diagnostics.Tracing.TraceEvent 4.0.0 and k6 2.4.0 for load. PerfView 4.0.0 runs only on Windows, so the PerfView steps are taken from its official user guide and release notes.

Guess First

For Bookstore, the online bookstore used as the shared example across this site's series, someone on the team has a proposal: publish Bookstore.Api with ReadyToRun so the JIT cost at startup goes away. Until the app starts and answers its first request, 980 methods get JIT-compiled. With ReadyToRun on, how far does that number drop: to 500, to 100, to zero? Second question: if you take two 10-second allocation traces of the same code under the same load and compare allocation per request, how far apart will the two measurements be? The answers are in the JITStats and CI sections; both were measured with the library that sits behind PerfView.

Getting PerfView and Collecting on Windows

PerfView is Microsoft's free, open source performance analysis tool. There is no installer: you download a single PerfView.exe from the Releases page of the microsoft/perfview repository. The latest version at the time of writing is 4.0.0 (October 2026). With this release PerfView and TraceEvent require Windows 10 or later and 32-bit support is gone; if you have to work on an older Windows server, stay on the 3.x line. The same release note says the TraceEvent library keeps running on Linux, and the second half of this post relies on exactly that.

On Windows, PerfView collects data with ETW, so it has to run as administrator. Adapting the command-line usage from the user guide to Bookstore:

PerfView.exe /AcceptEula /ThreadTime /MaxCollectSec:30 /LogFile=collect.log collect

/MaxCollectSec bounds the recording; without it PerfView keeps writing until you stop it. /ThreadTime also turns on context switch events, so besides time on the CPU you see where threads wait. The guide highlights two more options: /DotNetAllocSampled for sampled allocation stacks, and /GCCollectOnly for GC events only. The latter produces files small enough to leave running for hours. Because ETW sees every process on the machine and the kernel, PerfView's CPU view also has native frames; this is the Windows counterpart of what we tried and failed to get with collect-linux on Linux in post 03 of the series.

Opening a .nettrace from Linux

Bookstore runs in a Linux container in production and we take traces there with dotnet-trace. The dotnet-trace page on Microsoft Learn says you can move these .nettrace files to Windows and open them in PerfView or Visual Studio. In PerfView you open the file from the explorer on the left; the stack views and the event list appear under it. There are two differences. First, EventPipe sees only managed code, so there are no native frames and no kernel events. Second, the stacks are built not from CPU samples but from the thread time samples described in 03; waiting threads accumulate time too. If you have to open a truncated trace taken while the app crashed with an unhandled exception, the dotnet-trace docs recommend starting PerfView with /ContinueOnError.

CPU Stacks: By Name, Call Tree, GroupPats and Fold%

PerfView's stack window has three main tabs: By Name (a flat list with each method's exclusive and inclusive time), Call Tree (the call tree from the root down) and Caller-Callee (who calls the method you picked and what it calls). Analysis always starts with By Name. The real power is in the text boxes at the top of the window:

• GroupPats: groups frame names by a pattern. PAT->GROUP turns every matching frame into the group, while PAT=>GROUP also keeps, next to the name, the method through which the group was entered (an "entry group" in the guide).
• Fold%: folds nodes smaller than this share of the total into their caller; press F7 to raise it and cut the noise.
• FoldPats: always folds the named frames into their caller; useful for removing a helper you cannot optimize.
• IncPats and ExcPats: keep only the samples that contain (or do not contain) a given frame.

PerfView's default grouping is "Just My App": it collapses everything outside the folder tree of the app's exe into a single OTHER group and shows only your code in detail. According to the guide's release history, this default was removed for apps launched with dotnet.exe; the note gives as the reason that it did more harm than good. The logic is easy to see: your code does not live in dotnet.exe's folder, so everything would fall into OTHER. For a service running as dotnet Bookstore.Api.dll you have to choose the grouping yourself; the starting point the guide suggests is "group module entries", that is {%}!=>module $1.

None of these steps is PerfView-only magic; PerfView itself is built on the TraceEvent library, and these filters live in the library under the name FilterParams. That is why I could run the same analysis on Linux with a small console app. This is the heart of Bookstore.TraceStats:

// the text boxes at the top of a PerfView stack view
public sealed record StackFilter(string GroupPats, string FoldPercent, string FoldPats, string IncPats, string ExcPats);

public static class CpuStacks
{
    // the same "Thread Time" stacks PerfView builds from a .nettrace
    public static StackSource Load(string path, StackFilter filter)
    {
        TraceLog log = new(TraceLog.CreateFromEventPipeDataFile(path));
        MutableTraceEventStackSource stacks = new(log) { OnlyManagedCodeStacks = true };
        SampleProfilerThreadTimeComputer computer = new(log, new SymbolReader(TextWriter.Null));
        computer.GenerateThreadTimeStacks(stacks);

        // PerfView's GroupPats, FoldPats, IncPats and ExcPats boxes map to these FilterParams fields
        FilterParams filterParams = new()
        {
            GroupRegExs = filter.GroupPats,
            FoldRegExs = filter.FoldPats,
            IncludeRegExs = filter.IncPats,
            ExcludeRegExs = filter.ExcPats
        };
        return new FilterStackSource(filterParams, stacks, ScalingPolicyKind.ScaleToData);
    }

    public static Dictionary<string, (float Exclusive, float Inclusive)> ByName(StackSource source, StackFilter filter)
    {
        CallTree tree = new(ScalingPolicyKind.ScaleToData) { StackSource = source };
        // Fold%: nodes below this share of the total are inlined into their caller
        if (float.TryParse(filter.FoldPercent, out float foldPercent) && foldPercent > 0)
            tree.FoldNodesUnder(foldPercent * tree.Root.InclusiveMetric / 100, true);
        return tree.ByIDSortedExclusiveMetric()
            .GroupBy(n => n.Name)
            .ToDictionary(g => g.Key, g => (g.Sum(n => n.ExclusiveMetric), g.Max(n => n.InclusiveMetric)));
    }
}

SampleProfilerThreadTimeComputer is also the class that produces the CPU_TIME and UNMANAGED_CODE_TIME pseudo-frames in dotnet-trace's speedscope output. Dropping the samples spent in native code or waiting with ExcPats and folding CPU_TIME into its caller with FoldPats leaves the time spent in managed code. As a case I built a regression of the search endpoint from 03: search-regex-only is the fixed search-v2 with the projection rolled back, so it once again returns the whole book graph to the client. I gave both endpoints exactly the same work with k6: 300 requests per second, 10 seconds, 3,000 requests. Grouping the test trace by module:

dotnet run -c Release -- cpu search-regex-only.nettrace \
    --inc "module Microsoft.AspNetCore.Server.Kestrel.Core" --exc UNMANAGED_CODE_TIME \
    --foldpats CPU_TIME --group "{%}!->module $1"

 Exc %  Inc %  Name   (total 3,244 ms)
  92.6   99.7  module System.Private.CoreLib
   4.6   10.6  module Bookstore.Api
   1.3   86.5  module System.Text.Json
   1.0   99.7  module Microsoft.AspNetCore.Server.Kestrel.Core
   0.3  100.0  Threads
   0.1    9.3  module System.Linq

One detail: IncPats looks at group names, because, as in PerfView, grouping is applied before filtering. After grouping, the frame is named module Microsoft.AspNetCore.Server.Kestrel.Core. The table says 86.5 percent of request time is spent under System.Text.Json, yet almost all of the exclusive time is in CoreLib. What happens in CoreLib inside JSON is the question for the next step.

Diffing Two Traces

To diff in PerfView you open two stack windows (test and baseline) with the same filters and pick the baseline from the Diff menu of the test window. The algorithm the guide describes is simple: the metric of every baseline sample is negated and merged with the test samples, nodes with the same name cancel out, and what remains is the extra time in the test. There are two conditions: both traces must represent the same work (which is why I fixed the request count), and you must resolve symbols before diffing, because symbol lookup does not work in a diff window. In the By Name view the same computation means test minus baseline per name:

dotnet run -c Release -- diff search-v2.nettrace search-regex-only.nettrace \
    --inc "Microsoft.AspNetCore.Server.Kestrel.Core!" --exc UNMANAGED_CODE_TIME \
    --foldpats CPU_TIME --fold 2

baseline 642 ms, test 3,244 ms, diff +2,602 ms
   Exc ms    Inc ms  Name
   +1,983    +1,983  System.Private.CoreLib!System.Threading.Thread.<PollGC>g__PollGCWorker|67_0()
     +166      +166  System.Private.CoreLib!System.Threading.Monitor.Enter_Slowpath()
     -162      -331  System.Linq!System.Linq.Enumerable+IteratorSelectIterator`2[System.__Canon,System.__Canon].ToListNoPresizing()
     +139      +139  Bookstore.Api!Bookstore.Api.Catalog.SearchEndpoints+<>c__DisplayClass1_3.<MapSearchEndpoints>b__8()
     +119      +119  System.Private.CoreLib!System.Buffer.MemmoveInternal()

2.6 seconds of extra time across 3,000 requests, 0.87 ms per request. Two seconds of that are in PollGCWorker: threads waiting to be suspended for a GC. So the regression is less about serialization itself and more about the GCs triggered by the allocations of 787 KB responses. That is a conclusion you cannot reach by looking at a single trace; the baseline waits for GC too, but a quarter as much.

The negative row is instructive as well. ToListNoPresizing exists in the baseline and not in the test, because the code path changed when the projection went away; that is a real difference. But the +139 ms Bookstore row is partly fake: the search lambda lives under different compiler-generated names in the two endpoints (DisplayClass1_3 and another class) and the two do not cancel out. The guide also warns not to trust positive values while large negative ones are present. The cure is grouping: in the same diff by module, the Bookstore row becomes a single name and the two sides cancel.

GCStats and JITStats

PerfView's two ready-made reports answer a different question from the stack views: what happened over time? GCStats lists every GC per process and summarizes how the heap grew, when and why GCs happened and how long each one took. JITStats shows which methods were JIT-compiled, their IL and native sizes and how long compilation took. Both are fed by TraceEvent's TraceLoadedDotNetRuntime analysis layer, and reaching the same data in code takes a few lines:

using EventPipeEventSource source = new(path);
source.NeedLoadedDotNetRuntimes();
source.Process();
TraceLoadedDotNetRuntime runtime = source.Processes()
    .Select(p => p.LoadedDotNetRuntime())
    .First(r => r is not null);

List<TraceGC> gcs = runtime.GC.GCs;                   // the rows of GCStats
List<TraceJittedMethod> methods = runtime.JIT.Methods; // the rows of JITStats

I recorded the CSV export from 04 at 40 requests per second for 12 seconds with the gc-collect profile and produced the GCStats summary:

GCs 207, total pause 222.8 ms, 1.8% of 12.1 s
 Gen  Count  Mean pause  Max pause  Promoted MB
   0      6        0.85       1.13         12.7
   1     57        0.92       3.67         11.8
   2    144        1.15       3.87        102.3
Reasons:
    118  gen2 AllocLarge (BackgroundGC)
     56  gen1 AllocSmall (NonConcurrentGC)
     26  gen2 AllocSmall (BackgroundGC)
      5  gen0 AllocSmall (NonConcurrentGC)

This table tells you something counters alone cannot: most GCs are gen2 and 118 of them are caused by AllocLarge, that is allocations on the LOH. The conclusion reached in 04 appears here together with its cause. Because the gen2s are background GCs, the mean pause stays at 1.15 ms; under the 92 requests per second load of 04 this table was much heavier.

The classic use of JITStats is startup analysis. I ran dotnet-trace so that it starts the app itself with -- and collected JIT and loader events at verbose level; ASP.NET Core's Hosting events give the time of the first response:

dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:0x2000000018:5,Microsoft.AspNetCore.Hosting \
    --duration 00:00:00:08 -o startup.nettrace -- dotnet Bookstore.Api.dll

I repeated the measurement for an image published with ReadyToRun (dotnet publish -c Release --ucr -p:PublishReadyToRun=true):

The answer to the guess: from 980 to 928. ReadyToRun precompiled only our assembly, and its 51 methods dropped off the list. The rest already comes from the framework: .NET's own libraries ship as ReadyToRun, but generic instantiations over the app's types and lambdas still go to the JIT. In this app only 53 ms of the 356 ms startup is JIT; the bigger item is elsewhere (for example CatalogStore, which builds 20,000 books in memory on the first request; the first request alone took 160 ms). Without JITStats the proposal would most likely have been accepted because startup "felt faster".

Your Own Analysis with TraceEvent: an Allocation Gate in CI

The real payoff of programmatic analysis is repeatability: you can ask the same question on every build. I added a command to Bookstore.TraceStats that compares allocation per request between two traces. It derives estimated bytes per type from the .NET 10 AllocationSampled event described in 04, and the request count from Hosting's Request/Stop event:

public static AllocationProfile Load(string path)
{
    using EventPipeEventSource source = new(path);
    Dictionary<string, double> bytesByType = [];
    int requests = 0;
    source.Clr.AllocationSampling += e =>
    {
        // estimator from the runtime's randomized allocation sampling design: size / (1 - q^size)
        double estimate = e.ObjectSize / (1 - Math.Pow(1 - 1.0 / 102_400, e.ObjectSize));
        bytesByType[e.TypeName] = bytesByType.GetValueOrDefault(e.TypeName) + estimate;
    };
    source.Dynamic.AddCallbackForProviderEvent("Microsoft.AspNetCore.Hosting", "Request/Stop", _ => requests++);
    source.Process();
    return new AllocationProfile(Math.Max(requests, 1), bytesByType);
}

The gate command compares the two and exits with 1 if allocation per request grows by more than the allowed percentage; that is enough to break a CI step. The traces were taken with --clrevents gc+allocationsampling --providers Microsoft.AspNetCore.Hosting. With the fixed orders-v2.csv from 04 as the baseline and the old orders.csv as the test:

dotnet run -c Release -- gate alloc-orders-v2.nettrace alloc-orders.nettrace --max-growth 10

baseline 88.1 KB/request, test 59,376.3 KB/request, +67298.8%
   23,556.0 KB  System.Char[]
   23,467.6 KB  System.String
   11,735.1 KB  System.Byte[]
FAIL: allocation per request grew more than 10%

So far, as expected. The real lesson came from comparing the same code with itself. The answer to the second guess: I took two 10-second traces of orders-v2.csv, 400 requests each, and the gate broke with a 22.1 percent increase (88.1 KB against 107.6 KB). Sampling takes one sample per 100 KB on average; for an endpoint that allocates 90 KB per request, 10 seconds is too little for a statistically sound estimate. Repeating the measurement with two 30-second traces (1,200 requests) brought the difference down to 1.2 percent. Before setting up a gate like this, compare the same build with itself a few times, measure the size of the noise and put the threshold above it; otherwise the team learns not to trust red builds.

When PerfView, When an IDE Profiler

PerfView has a steep learning curve: the UI is dense, the defaults are tuned for experts, and understanding why something looks empty sometimes means searching the guide. What you get in return is flexibility and scale: large traces from production, diffs of two traces, GC and JIT reports, kernel and native frames on Windows, all free and in a single exe. For profiling your own code on a developer machine, jumping to the line, or seeing a database query or an async flow, the Visual Studio and Rider profilers get you results faster; they are the topic of the next post. My rule: if the question is "what did this change slow down" or "what happened in production", PerfView or TraceEvent; if it is "why is this method slow", the IDE.

In Production

A PerfView trace collected with ETW contains every process on the machine; command lines, file paths and method names of other services are inside the file. Keep that in mind before sharing it and put it under the same access rules as dumps. /ThreadTime and the allocation options make files grow fast; although the guide says sampled allocation is cheap enough to leave on in production, do not rely on it without the with-and-without A/B measurement from 01 on your own service.

Programmatic analysis has two points to watch as well. TraceEvent first converts a large .nettrace into an ETLX file; budget disk and time on the CI agent accordingly. The load for gate traces must be fixed: a constant request rate, the same warm-up, the same data. In my example I counted requests from Hosting events, so the per-request value holds even if the trace duration drifts a little.

Common Pitfalls

• Diffing two traces that contain different amounts of work: if the test trace got twice the requests, every row looks like a regression. Fix the request count or normalize per request.
• Ignoring large negative rows in a diff: changing names such as compiler-generated lambda names do not cancel out; group first.
• Thinking of IncPats before grouped names: grouping is applied before filtering, so the pattern must be written against the group name.
• Reading PollGC or a semaphore wait in thread time as "code eating the CPU"; those are waits.
• Expecting a big startup win from ReadyToRun without looking at JITStats; the framework is already precompiled.
• Turning on a sampling-based CI gate without measuring its noise.

A Small Experiment

★☆☆ If you are on Windows, open the search trace from 03 in PerfView; in By Name, first clear GroupPats, then set it to {%}!=>module $1 and raise Fold% with F7 to see how the list gets simpler. If you are on Linux, try the same steps with the cpu command of Bookstore.TraceStats.

★★☆ Add a line to the gc report of Bookstore.TraceStats that prints p50/p95/p99 of the GC pauses and compare the GCStats of orders.csv and orders-v2.csv from 04.

★★★ Extend the gate command to set its threshold from the mean and standard deviation of N traces taken from the same build. Find how many seconds and how many traces it takes to catch a 5 percent regression reliably.

PerfView and TraceEvent give you every detail of a trace, but the price is time. In the next post we move to the side that gets results faster: the Visual Studio 2026 and Rider profilers; with the CPU Usage, .NET Object Allocation, Instrumentation, .NET Async and Database tools we will find and fix an EF Core N+1 query in Bookstore's order list.