Mert Özen Further With Every Line
Testing & Quality Performance and Load Testing — #07

Advanced BenchmarkDotNet: Parameters, Baselines, Diagnosers, Runtime Comparison and Catching Regressions in CI

Mert Özen Nov 14, 2026 17 min 15 views
Advanced BenchmarkDotNet: Parameters, Baselines, Diagnosers, Runtime Comparison and Catching Regressions in CI

Advanced BenchmarkDotNet means defining realistic scenarios with ParamsSource, making the old code the baseline, finding the cause of a difference with the Memory, Exception, Threading and Disassembly diagnosers, and catching regressions by relative, statistical comparison. On Bookstore's pricing code we measure that .NET 11 speeds up the LINQ version by 30-38 percent but not the loop, that 0.15.8 does not know net11.0, and that parsing with try/catch is ten times slower.

Version note: The code in this post was built and run with .NET 10 and .NET 11; the benchmarks were run with BenchmarkDotNet 0.15.8, the .NET 11 comparison with BenchmarkDotNet 0.16.0-preview.2, and the results were compared as described for the ResultsComparer tool from the dotnet/performance repository.

Guess First

The cart price calculation of Bookstore, the online bookstore used as the shared example across this site's series, can be written in two ways: with LINQ, going over the lines twice, or with a single loop. This is code that checkout calls hundreds of times per second on a campaign evening. Four questions. Which one is how many times faster? Does moving to .NET 11 speed both up by the same ratio? If 10 percent of the rows in a price list are malformed, how many times slower is wrapping decimal.Parse in try/catch than TryParse? Finally, how do you say with confidence that a benchmark got 30 percent slower on a shared CI runner? Setting up BenchmarkDotNet and the basics of reading its table are in the C# Advanced series post "Performance in C#: Allocations, Structs and Measuring with BenchmarkDotNet"; here we build on top of that.

Setting Up the Benchmark Project

The benchmarks live in a separate console project in the bench/ folder and reference Bookstore.Api, so the measured code is the production code itself. The price calculation lives in the API's CartPricing class: OrderService computes the order total with it, and unit tests guard its behavior (the institutional discount of 10 percent from 20 copies). The project builds for two frameworks at once, because we will run the same benchmark on .NET 10 and .NET 11:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <!-- One build per runtime, so the same benchmarks can run on .NET 10 and .NET 11. -->
    <TargetFramework></TargetFramework>
    <TargetFrameworks>net10.0;net11.0</TargetFrameworks>
    <RootNamespace>Bookstore.Benchmarks</RootNamespace>
    <Optimize>true</Optimize>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\Bookstore.Api\Bookstore.Api.csproj" />
  </ItemGroup>

</Project>

The global.json at the root of the repository keeps the SDK in the 10.0.100 band; the net11.0 target needs the .NET 11 SDK. That is why I put a separate global.json into the bench/ folder: dotnet commands run from this folder use the 11.0.100 SDK, while the API keeps building as net10.0. For the same reason I did not add the benchmark project to the solution; dotnet build and dotnet test from the root keep working with the .NET 10 SDK. The .NET 11 SDK has been GA since November 10, 2026; the .NET 10 runtime must be installed too. Runs are always in Release: dotnet run -c Release -f net10.0 -- --filter "*PricingBenchmarks*".

[ParamsSource] and Baselines

[Params] is enough for fixed values, but for scenarios that carry meaning together, such as "1 book", "12 books" and "a school ordering 25 copies each of 40 books", [ParamsSource] fits better. It returns a list of objects from a method or property. The results table shows the object's ToString(), so I gave the record a readable name:

using BenchmarkDotNet.Attributes;
using Bookstore.Ordering;

namespace Bookstore.Benchmarks;

// A named cart shape: ParamsSource shows its ToString() in the results table.
public sealed record CartShape(string Name, int Lines, int QuantityPerLine, bool Institutional)
{
    public override string ToString() => Name;
}

[MemoryDiagnoser]
public class PricingBenchmarks
{
    private PriceLine[] lines = [];

    [ParamsSource(nameof(Carts))]
    public CartShape Cart { get; set; } = null!;

    public static IEnumerable<CartShape> Carts() =>
    [
        new("1 book", 1, 1, false),
        new("12 books", 12, 1, false),
        new("school 40x25", 40, 25, true)
    ];

    [GlobalSetup]
    public void Setup()
    {
        Random random = new(42);
        lines = [.. Enumerable.Range(0, Cart.Lines)
            .Select(i => new PriceLine($"979{i:D10}", Math.Round(99m + (decimal)random.NextDouble() * 900m, 2), Cart.QuantityPerLine))];
    }

    // The first version, before CartPricing: LINQ over the lines, two passes.
    [Benchmark(Baseline = true)]
    public PriceQuote Linq()
    {
        decimal subtotal = lines.Sum(l => OrderItem.CalculateLineTotal(l.UnitPrice, l.Quantity));
        int quantity = lines.Sum(l => l.Quantity);
        decimal discount = Cart.Institutional && quantity >= CartPricing.InstitutionalMinimumQuantity
            ? Math.Round(subtotal * CartPricing.InstitutionalDiscountRate, 2, MidpointRounding.AwayFromZero)
            : 0m;
        return new PriceQuote(subtotal, discount, subtotal - discount);
    }

    [Benchmark]
    public PriceQuote Span() => CartPricing.Quote(lines, Cart.Institutional);
}

Baseline = true makes BenchmarkDotNet compute the other methods' ratio to that method (Ratio). Choosing the "old version" as the baseline makes it easy to read: 0.47 means the new code takes less than half the time of the old one. [GlobalSetup] prepares the data once per parameter combination, outside the measurement. A fixed seed such as Random(42) guarantees that two runs work with the same data. It also matters that benchmark methods return a value: BenchmarkDotNet consumes the returned value, so the JIT cannot remove the computation as "unused".

Diagnosers: Memory, Exception and Threading

We know [MemoryDiagnoser] from C# Advanced 12: allocation and GC counts per operation. For the code that imports Bookstore's supplier price list I added [ExceptionDiagnoser]. Some rows in the list say "n/a" instead of a price:

[MemoryDiagnoser]
[ExceptionDiagnoser]
public class ImportBenchmarks
{
    private string[] rows = [];

    [Params(0, 10)]
    public int BadPercent { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        Random random = new(42);
        rows = [.. Enumerable.Range(0, 1_000).Select(i =>
            $"979{i:D10};{(random.Next(100) < BadPercent ? "n/a" : (99 + i % 900).ToString(CultureInfo.InvariantCulture) + ".90")}")];
    }

    [Benchmark(Baseline = true)]
    public decimal ParseWithCatch()
    {
        decimal total = 0m;
        foreach (string row in rows)
        {
            try
            {
                total += decimal.Parse(row.AsSpan(row.IndexOf(';') + 1), CultureInfo.InvariantCulture);
            }
            catch (FormatException)
            {
                // skip the malformed row
            }
        }
        return total;
    }

    [Benchmark]
    public decimal TryParse()
    {
        decimal total = 0m;
        foreach (string row in rows)
        {
            if (decimal.TryParse(row.AsSpan(row.IndexOf(';') + 1), CultureInfo.InvariantCulture, out decimal price))
            {
                total += price;
| Method         | BadPercent | Mean      | Error    | StdDev   | Ratio | RatioSD | Gen0   | Exceptions | Allocated | Alloc Ratio |
|--------------- |----------- |----------:|---------:|---------:|------:|--------:|-------:|-----------:|----------:|------------:|
| ParseWithCatch | 0          |  27.26 us | 0.241 us | 0.202 us |  1.00 |    0.01 |      - |          - |         - |          NA |
| TryParse       | 0          |  27.33 us | 0.423 us | 0.395 us |  1.00 |    0.02 |      - |          - |         - |          NA |
|                |            |           |          |          |       |         |        |            |           |             |
| ParseWithCatch | 10         | 254.33 us | 4.815 us | 4.504 us |  1.00 |    0.02 | 4.8828 |   111.0000 |   41736 B |        1.00 |
| TryParse       | 10         |  25.34 us | 0.405 us | 0.379 us |  0.10 |    0.00 |      - |          - |         - |        0.00 |

With no malformed rows the two approaches are the same. When 10 percent of the rows are malformed, 111 exceptions are thrown per 1,000 rows and the try/catch version becomes ten times slower and allocates 41 KB. Without the exception count column you would have to guess the reason for that difference. That is the answer to the third guess question: ten times.

[ThreadingDiagnoser] counts the work items sent to the thread pool (Completed Work Items) and lock contentions. I compared the nightly job that re-prices every cart after a price change, sequentially and with Parallel.For:

// Nightly re-pricing of many carts after a price change: is Parallel.For worth it?
[ThreadingDiagnoser]
public class BatchPricingBenchmarks
{
    private PriceLine[][] carts = [];
    private PriceQuote[] quotes = [];

    [Params(100, 10_000)]
    public int Carts { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        Random random = new(42);
        carts = [.. Enumerable.Range(0, Carts).Select(_ =>
            Enumerable.Range(0, 1 + random.Next(5)).Select(i => new PriceLine($"979{i:D10}", 249.90m, 1 + random.Next(3))).ToArray())];
        quotes = new PriceQuote[Carts];
    }

    [Benchmark(Baseline = true)]
    public void Sequential()
    {
        for (int i = 0; i < carts.Length; i++)
        {
            quotes[i] = CartPricing.Quote(carts[i], institutional: false);
        }
    }

    [Benchmark]
    public void Parallel() =>
        System.Threading.Tasks.Parallel.For(0, carts.Length, i => quotes[i] = CartPricing.Quote(carts[i], institutional: false));
}
| Method     | Carts | Mean       | Error     | StdDev     | Median     | Ratio | RatioSD | Completed Work Items | Lock Contentions |
|----------- |------ |-----------:|----------:|-----------:|-----------:|------:|--------:|---------------------:|-----------------:|
| Sequential | 100   |   2.389 us | 0.0474 us |  0.1050 us |   2.386 us |  1.00 |    0.06 |                    - |                - |
| Parallel   | 100   |   6.952 us | 0.4572 us |  1.2517 us |   6.419 us |  2.92 |    0.54 |               5.1395 |                - |
|            |       |            |           |            |            |       |         |                      |                  |
| Sequential | 10000 | 251.785 us | 5.0251 us |  9.0613 us | 250.766 us |  1.00 |    0.05 |                    - |                - |
| Parallel   | 10000 | 102.255 us | 3.6551 us | 10.3093 us |  98.081 us |  0.41 |    0.04 |               8.4739 |                - |

With 100 carts the parallel version is three times slower: 2.4 µs of work is smaller than the cost of sending five work items to the thread pool. With 10,000 carts the same code is 2.5 times faster. Whether parallelism pays off depends on the amount of work, and the only way to know is to measure.

DisassemblyDiagnoser and EventPipeProfiler

One way to see where the difference between two implementations comes from is to read the machine code the JIT produces. DisassemblyDiagnoser does this from inside the benchmark: [DisassemblyDiagnoser(maxDepth: 2)], or --disasm on the command line. Machine code is specific to the processor architecture. To show the output you get on a Windows/x64 machine, I ran the benchmark in an x64 Linux container:

## .NET 10.0.15 (10.0.15, 10.0.1526.53008), X64 RyuJIT x86-64-v3 (Job: ShortRun(IterationCount=3, LaunchCount=1, WarmupCount=3))

; Bookstore.Ordering.CartPricing.Quote(System.ReadOnlySpan`1<Bookstore.Ordering.PriceLine>, Boolean)
       push      rbp
       ...
       call      qword ptr [7FFF8151C450]; System.Decimal+DecCalc.DecAddSub(DecCalc ByRef, DecCalc ByRef, Boolean)
       ...
       call      CORINFO_HELP_NEWSFAST
       ...
       call      qword ptr [7FFF8151C228]; System.Decimal.op_Multiply(System.Decimal, System.Decimal)
       call      qword ptr [7FFF8151C468]; System.Decimal.Round(System.Decimal ByRef, Int32, System.MidpointRounding)
; Total bytes of code 1853

There is no call to OrderItem.CalculateLineTotal in the output: the JIT inlined it into Quote. The real work in the loop is decimal addition (DecAddSub); the single allocation (CORINFO_HELP_NEWSFAST) is the returned PriceQuote object, the 64 bytes in the table. Since this container is Linux, registers follow the Linux x64 calling convention; on Windows parameters arrive in different registers, but the instructions follow the same logic. The table also gains a new Code Size column: the size of the benchmark's machine code.

To see where the code spends time at the method level, use EventPipeProfiler: with --profiler EP it collects a CPU sampling trace while the benchmark runs and produces a file you can open in speedscope:

dotnet run -c Release -f net10.0 -- --filter "Bookstore.Benchmarks.PricingBenchmarks.Span" --profiler EP --job short
// * Diagnostic Output - EventPipeProfiler *
Exported 3 trace file(s). Example:
.../Bookstore.Benchmarks.PricingBenchmarks.Span(Cart_ 1 book)-20261114-174358.speedscope.json

The distribution of samples in the "school 40x25" trace, by managed frame:

 71.7%  Bookstore.Api!Bookstore.Ordering.CartPricing.Quote(value class System.ReadOnlySpan`1<class Bookstore.Ordering.PriceLine>,bool)
 22.0%  System.Private.CoreLib!System.Decimal+DecCalc.DecAddSub(value class DecCalc&,value class DecCalc&,bool)
  2.4%  Bookstore.Benchmarks-ShortRun-1!BenchmarkDotNet.Autogenerated.Runnable_2.OverheadActionUnroll(int64)
  1.6%  Bookstore.Benchmarks!Bookstore.Benchmarks.PricingBenchmarks.Span()

The details of reading the trace and interpreting a flame graph are in the "Profiling and Diagnostics in .NET" series post on CPU profiling with dotnet-trace and flame graphs. The difference with producing a trace from a benchmark is that the code you measure is exactly the benchmark method: no noise from the rest of the application.

Comparing .NET 10 and .NET 11

The way to run the same benchmark on two runtimes is to define more than one job. On the command line that is --runtimes. When I tried it with 0.15.8:

dotnet run -c Release -f net10.0 -- --filter "*PricingBenchmarks*" --runtimes net10.0 net11.0
The provided runtime "net11.0" is invalid. Available options are: hostprocess, notrecognized, mono, ...
net70, net80, net90, net10_0, nativeaot60, ...

0.15.8 came out in November 2025, before .NET 11 was GA, and does not know net11.0. I also tried building the job in code with CoreRuntime.CreateForNewVersion("net11.0", ".NET 11.0"); this time the SDK validation threw a NotImplementedException. .NET 11 support arrived with 0.16.0-preview.1. I ran the comparison with 0.16.0-preview.2. I bound the package version to a property in Directory.Packages.props, so the stable release stays the default:

<Project>
  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <!-- 0.15.8 is the latest stable release. .NET 11 jobs need 0.16: set the BenchmarkDotNetVersion environment variable to 0.16.0-preview.2. -->
    <BenchmarkDotNetVersion Condition="'$(BenchmarkDotNetVersion)' == ''">0.15.8</BenchmarkDotNetVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.0.15" />
    <PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.15" />
    <PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.15" />
    <PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.15" />
    <PackageVersion Include="xunit.v3" Version="4.0.3" />
    <PackageVersion Include="WireMock.Net" Version="2.31.0" />
    <PackageVersion Include="BenchmarkDotNet" Version="$(BenchmarkDotNetVersion)" />
  </ItemGroup>
</Project>
# PowerShell: as an environment variable, so that BenchmarkDotNet's own builds see the same version
$env:BenchmarkDotNetVersion = "0.16.0-preview.2"
dotnet run -c Release -f net10.0 -- --filter "Bookstore.Benchmarks.PricingBenchmarks.*" --runtimes net10.0 net11.0
| Method | Job       | Runtime   | Cart         | Mean      | Error    | StdDev    | Median    | Ratio | RatioSD | Gen0   | Allocated | Alloc Ratio |
|------- |---------- |---------- |------------- |----------:|---------:|----------:|----------:|------:|--------:|-------:|----------:|------------:|
| Linq   | .NET 10.0 | .NET 10.0 | 1 book       |  32.32 ns | 0.771 ns |  2.211 ns |  31.57 ns |  1.00 |    0.00 | 0.0153 |     128 B |        1.00 |
| Span   | .NET 10.0 | .NET 10.0 | 1 book       |  15.13 ns | 0.352 ns |  1.011 ns |  14.81 ns |  0.47 |    0.04 | 0.0076 |      64 B |        0.50 |
| Linq   | .NET 11.0 | .NET 11.0 | 1 book       |  20.04 ns | 0.386 ns |  0.940 ns |  19.82 ns |  0.62 |    0.05 | 0.0115 |      96 B |        0.75 |
| Span   | .NET 11.0 | .NET 11.0 | 1 book       |  15.54 ns | 0.393 ns |  1.152 ns |  15.44 ns |  0.48 |    0.05 | 0.0076 |      64 B |        0.50 |
|        |           |           |              |           |          |           |           |       |         |        |           |             |
| Linq   | .NET 10.0 | .NET 10.0 | 12 books     | 139.44 ns | 2.772 ns |  3.300 ns | 139.71 ns |  1.00 |    0.00 | 0.0153 |     128 B |        1.00 |
| Span   | .NET 10.0 | .NET 10.0 | 12 books     |  64.17 ns | 1.003 ns |  0.837 ns |  64.14 ns |  0.46 |    0.01 | 0.0076 |      64 B |        0.50 |
| Linq   | .NET 11.0 | .NET 11.0 | 12 books     |  89.54 ns | 1.748 ns |  2.508 ns |  89.05 ns |  0.64 |    0.02 | 0.0114 |      96 B |        0.75 |
| Span   | .NET 11.0 | .NET 11.0 | 12 books     |  64.03 ns | 1.532 ns |  4.320 ns |  63.64 ns |  0.46 |    0.03 | 0.0076 |      64 B |        0.50 |
|        |           |           |              |           |          |           |           |       |         |        |           |             |
| Linq   | .NET 10.0 | .NET 10.0 | school 40x25 | 422.87 ns | 8.304 ns | 20.834 ns | 418.59 ns |  1.00 |    0.00 | 0.0153 |     128 B |        1.00 |
| Span   | .NET 10.0 | .NET 10.0 | school 40x25 | 212.95 ns | 7.282 ns | 20.658 ns | 206.75 ns |  0.50 |    0.05 | 0.0076 |      64 B |        0.50 |
| Linq   | .NET 11.0 | .NET 11.0 | school 40x25 | 296.01 ns | 9.266 ns | 27.175 ns | 289.69 ns |  0.70 |    0.07 | 0.0114 |      96 B |        0.75 |
| Span   | .NET 11.0 | .NET 11.0 | school 40x25 | 192.08 ns | 3.349 ns |  5.687 ns | 190.70 ns |  0.46 |    0.03 | 0.0076 |      64 B |        0.50 |

The answer to the first two guess questions. On .NET 10 the loop version takes about half the time of LINQ and allocates half as much. Moving to .NET 11 does not speed both up by the same ratio: the LINQ version gets 30-38 percent faster and allocates 96 bytes instead of 128, while the loop version stays almost the same. The runtime's gain is in LINQ's delegate and enumerator cost. There is nothing left to gain in an already simple loop. Even so, LINQ on .NET 11 does not catch up with the loop on .NET 10.

Read the two warnings under the table as well. "MultimodalDistribution" says the measurements cluster in more than one group rather than around a single peak; that is common on a machine with other workload in the background. The "Outliers" lines show that BenchmarkDotNet removes outliers at the upper end by default. The absolute values are specific to the machine and the processor architecture; they will be different on yours. What carries over are the ratios and the allocated bytes.

Catching Regressions in CI

Running a benchmark in CI and alerting on "Mean went up by 10 ns" does not work. The shared-runner noise we discussed in post 04 matters at the nanosecond level here: the same benchmark can move 10-20 percent between two runs. Two things work. The first is relative comparison: measure the baseline and the new code in the same job, on the same machine, back to back. The Ratio column in this post is exactly that. The second is statistical comparison: compare not only the means of two runs but all their measurements, and test whether the difference is larger than the noise. The ResultsComparer in the dotnet/performance repository does this. It reads BenchmarkDotNet's fulljson export and compares two result sets against a threshold.

To try it, I introduced a regression into CartPricing: a change that copies the lines into a list and sums them with LINQ, "to make it more readable":

    public static PriceQuote Quote(ReadOnlySpan<PriceLine> lines, bool institutional)
    {
        // "More readable": LINQ over a list copy of the lines.
        List<PriceLine> items = [.. lines];
        decimal subtotal = items.Sum(l => OrderItem.CalculateLineTotal(l.UnitPrice, l.Quantity));
        int quantity = items.Sum(l => l.Quantity);
# the code on main
dotnet run -c Release -f net10.0 -- --filter "Bookstore.Benchmarks.PricingBenchmarks.Span" --exporters fulljson --artifacts results/base
# after the change
dotnet run -c Release -f net10.0 -- --filter "Bookstore.Benchmarks.PricingBenchmarks.Span" --exporters fulljson --artifacts results/diff

# dotnet/performance/src/tools/ResultsComparer
dotnet run -c Release -p:PERFLAB_TARGET_FRAMEWORKS=net10.0 -f net10.0 -- --base results/base --diff results/diff --threshold 5%
main:        | Method | Cart         | Mean      | Error    | StdDev   | Gen0   | Allocated |
             | Span   | 1 book       |  11.78 ns | 0.225 ns | 0.211 ns | 0.0076 |      64 B |
             | Span   | 12 books     |  55.76 ns | 0.658 ns | 0.549 ns | 0.0076 |      64 B |
             | Span   | school 40x25 | 163.36 ns | 0.958 ns | 0.800 ns | 0.0076 |      64 B |
change:      | Span   | 1 book       |  25.20 ns | 0.510 ns | 1.086 ns | 0.0201 |     168 B |
             | Span   | 12 books     | 105.93 ns | 1.925 ns | 1.608 ns | 0.0305 |     256 B |
             | Span   | school 40x25 | 315.18 ns | 6.255 ns | 11.748 ns | 0.0572 |     480 B |

The change roughly doubles the time for every cart type (2.1, 1.9 and 1.9 times) and raises allocation from 64 bytes to 168-480 bytes. The difference is clearly larger than the noise; the Error columns (half the confidence interval) are 1-4 percent of the mean in both runs. In other runs on the same machine on the same day this benchmark had reached 15 ns, which shows why a single number is not enough. I did not run ResultsComparer in this post. According to its README it matches the *full.json files in two folders by full benchmark name, produces "Slower" and "Faster" tables using the threshold you give and a statistical test, and leaves out differences below the threshold or at noise level.

The answer to the fourth guess question is this method: ResultsComparer evaluates the difference with a statistical test together with the threshold and lists only significant differences as "Slower" or "Faster". The next step in CI is to fail the job, or write a PR comment, when that table has a "Slower" row. Choose the threshold based on noise: on a shared runner 5 percent may be too tight; look at the differences between nightly runs themselves to learn the normal range of variation. Running base and diff back to back on the same runner removes the hardware difference between two runs.

Release Notes: 0.15.7 Analyzers, 0.15.8 and the 0.16 Previews

0.15.7 (November 2025) added Roslyn analyzers to BenchmarkDotNet: they check the structure of benchmark classes, the use of [Arguments] and [Params], the uniqueness of the baseline and BenchmarkRunner.Run calls at compile time. A broken benchmark is now caught when it compiles, not when it runs. For example, when I marked two methods as baselines:

PricingBenchmarks.cs(36,16): error BDN1107: Only one benchmark method can be marked as baseline per class
PricingBenchmarks.cs(47,16): error BDN1107: Only one benchmark method can be marked as baseline per class

The analyzer reports this directly as an error, so the build stops. 0.15.8 (November 2025) fixed the analyzers for projects with more than one target framework, added a new rule for argument and parameter validation and brought an OpenMetrics exporter. The two 0.16.0 previews came out in September 2026: .NET 11 support, .NET 12 runtime and toolchain support, OSR switched off by default and old APIs removed are some of the changes. At the time of writing 0.16 is not stable; stay on 0.15.8 for your production benchmarks and use the preview only for the runs that need it, such as the .NET 11 comparison above.

Production Notes

BenchmarkDotNet measures at the micro level, the behavior of a single method. I/O, databases, networks and the end-to-end behavior of concurrent users are not its job; that is the job of the load test. Halving the price calculation changes checkout's p95 only if the price calculation is on the hot path. In the measurements in post 06 most of checkout's time was in the database and the payment stub. So confirm a benchmark result with a load test. The Profiling series post "Diagnose, Fix, Prove" is an example of using the two measurements together.

Do not run benchmarks on a laptop in power-saving mode with a browser and an IDE open; BenchmarkDotNet warns in some cases but cannot catch everything. When reporting results, state the environment: runtime version, processor architecture, job settings. I deliberately removed the hardware header from the tables in this post. Absolute numbers are machine-specific; comparisons go through ratios within the same run.

Common Pitfalls

  • Running benchmarks in a Debug build or with a debugger attached.
  • Not returning the benchmark method's result and letting the JIT remove the computation.
  • Comparing the absolute values of two results taken on different machines or on different days.
  • Alerting on a fixed nanosecond threshold on a shared CI runner; use relative and statistical comparison.
  • Mistaking a micro-benchmark improvement for an end-to-end improvement.
  • Spreading a preview BenchmarkDotNet version across the whole project; use it only in the run that needs it.
  • Looking at Mean without reading the multimodal distribution and outlier warnings.

A Small Experiment

★☆☆ Add a second parameter called Institutional to PricingBenchmarks with [Params(false, true)] and remove the field from the ParamsSource. How did the number of rows in the table change? Which presentation reads better?

★★☆ Compare ImportBenchmarks on .NET 10 and .NET 11. Does the cost of throwing exceptions change with the runtime?

★★★ Add a job to GitHub Actions that benchmarks main and the PR branch back to back on the same runner, runs ResultsComparer and fails the job when there is a "Slower" row. Measure how many false alarms you get when you compare the same commit twice, and set the threshold accordingly.

The series ends here. We covered the types of load tests and acceptance criteria, a realistic workload model, k6 and JMeter, the gate in CI, reading results and regressions at the micro level. The common thread: every statement about performance needs a load, an environment and a repeatable measurement behind it.

Tags
#performance #load-testing #benchmarkdotnet #dotnet #csharp
← Previous post
Interpreting Results: Percentiles, Coordinated Omission, the Throughput-Latency Curve and Finding the Bottleneck