Performance in C#: Allocations, Structs and Measuring with BenchmarkDotNet

Version note: The code in this post uses C# 10 and the .NET 6 APIs; it was built and run with the .NET 10 SDK.

Guess First

Two methods that sum the same List<int>. Their bodies are identical character for character; the only difference is the parameter type. One takes a List<int>, the other an IEnumerable<int>. On a list of a thousand elements, is there a speed or memory difference between them? If so, how big?

private static int Sum(List<int> items)
{
    int total = 0;
    foreach (int item in items)
        total += item;
    return total;
}

private static int Sum(IEnumerable<int> items)
{
    int total = 0;
    foreach (int item in items)
        total += item;
    return total;
}

We'll answer in the middle of the post, by measuring. First, let's look at what "measuring" means.

Why Stopwatch Isn't Enough

The first idea is to run the code in a loop and time it with Stopwatch:

using System.Diagnostics;
using System.Globalization;

const string line = "978-0441013593;Dune;9.99;12";

for (int round = 1; round <= 3; round++)
{
    var sw = Stopwatch.StartNew();
    decimal total = 0;
    for (int i = 0; i < 1_000; i++)
        total += Parse(line);
    sw.Stop();
    Console.WriteLine($"round {round}: {sw.Elapsed.TotalMilliseconds.ToString("F3", CultureInfo.InvariantCulture)} ms");
}

static decimal Parse(string text)
{
    string[] parts = text.Split(';');
    return decimal.Parse(parts[2], CultureInfo.InvariantCulture) * int.Parse(parts[3]);
}

round 1: 4.255 ms
round 2: 0.111 ms
round 3: 0.112 ms

Same code, same data; the first round is about forty times slower than the others. The reason isn't the code itself, it's the runtime. In the first round the JIT compiles the methods to machine code, the culture data for decimal.Parse is loaded, and tiered compilation first produces fast but lightly optimized code and recompiles hot methods later. Add when the garbage collector kicks in, the processor changing frequency and whatever else is running on the machine at that moment, and a single Stopwatch measurement tells you almost nothing.

Then there's the cleverness of the compiler and the JIT: the JIT can remove a computation whose result you don't use, and you measure "zero nanoseconds". A good measurement tool has to run warm-up rounds, repeat the measurement enough times, compute statistics, make sure the results are used, and report memory allocation as well. In the .NET world that tool is called BenchmarkDotNet.

Setting Up BenchmarkDotNet

Benchmarks live in a separate console project. Adding the package is enough:

<ItemGroup>
  <PackageReference Include="BenchmarkDotNet" Version="0.15.8" />
</ItemGroup>

Program.cs is a single line; BenchmarkSwitcher finds all the benchmark classes in the assembly and lets you choose from the command line which ones to run:

using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(ParseBenchmarks).Assembly).Run(args);

Benchmarks must run from a Release build. Normally dotnet run -c Release is enough; since this machine has no .NET 6 runtime, I ran them like this:

dotnet build -c Release
DOTNET_ROLL_FORWARD=Major dotnet bin/Release/net6.0/Bookstore.Benchmarks.dll --filter "*"

A note: some of BenchmarkDotNet 0.15.8's dependencies no longer officially support net6.0, and the build gives warnings like System.CodeDom 9.0.5 doesn't support net6.0 and has not been tested with it. The measurement still worked, but .NET 6 is out of support anyway; in a real project the right move is to target the benchmark project at a supported runtime (net8.0 or net10.0). All the numbers below ran on the .NET 8.0.25 runtime.

First Measurement: Split and Span

In the Span post we counted the allocation difference; now let's measure the time too. A benchmark class is an ordinary class: every [Benchmark] method is a measurement, the one with Baseline = true is the reference for comparison, and [MemoryDiagnoser] adds the allocation columns:

using System.Globalization;
using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
public class ParseBenchmarks
{
    private const string Line = "978-0441013593;Dune;9.99;12";

    [Benchmark(Baseline = true)]
    public decimal Split()
    {
        string[] parts = Line.Split(';');
        decimal price = decimal.Parse(parts[2], CultureInfo.InvariantCulture);
        int stock = int.Parse(parts[3]);
        return price * stock;
    }

    [Benchmark]
    public decimal Span()
    {
        ReadOnlySpan<char> rest = Line.AsSpan();
        for (int skip = 0; skip < 2; skip++)
            rest = rest[(rest.IndexOf(';') + 1)..];

        int sep = rest.IndexOf(';');
        decimal price = decimal.Parse(rest[..sep], provider: CultureInfo.InvariantCulture);
        int stock = int.Parse(rest[(sep + 1)..]);
        return price * stock;
    }
}

The methods return their result. This matters: BenchmarkDotNet "uses" the returned value, so the JIT can't remove the computation. The machine and runtime the measurement ran on are at the top of the report:

BenchmarkDotNet v0.15.8, macOS 27.0.1 (26A434) [Darwin 27.0.0]
Apple M5, 1 CPU, 10 logical and 10 physical cores
.NET SDK 10.0.402
  [Host]     : .NET 8.0.25 (8.0.25, 8.0.2526.11203), Arm64 RyuJIT armv8.0-a
  DefaultJob : .NET 8.0.25 (8.0.25, 8.0.2526.11203), Arm64 RyuJIT armv8.0-a

| Method | Mean     | Error    | StdDev   | Ratio | RatioSD | Gen0   | Allocated | Alloc Ratio |
|------- |---------:|---------:|---------:|------:|--------:|-------:|----------:|------------:|
| Split  | 52.45 ns | 0.756 ns | 0.670 ns |  1.00 |    0.02 | 0.0249 |     208 B |        1.00 |
| Span   | 27.45 ns | 0.196 ns | 0.174 ns |  0.52 |    0.01 |      - |         - |        0.00 |

Reading the Table

Mean is the average time of one call: Split takes 52 nanoseconds, Span 27 nanoseconds. Error and StdDev show how much the measurement wobbles; as long as they stay small next to the mean, you can trust the result. Ratio is relative to the baseline: Span takes about half as long as Split (0.52). Gen0 is the number of generation 0 collections per thousand calls; Allocated is the memory allocated per call. The 208 bytes we counted with GC.GetAllocatedBytesForCurrentThread() in the Span post are the same here.

Read the numbers as relative, not absolute. These measurements were taken on an Apple M5 while other builds were running on the machine in parallel; run the same benchmark on a quiet machine or another processor and the nanoseconds will change. The allocation column barely changes at all; that's why the column I usually look at first is Allocated. The ratio is also far more stable across machines than the time.

Class or Struct

Let's measure the difference between a class and a struct when modeling order lines. We define the same type with two properties both ways, fill an array with a thousand of them and sum them:

using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
public class ShapeBenchmarks
{
    private const int Count = 1_000;

    [Benchmark(Baseline = true)]
    public decimal Classes()
    {
        var lines = new OrderLineClass[Count];
        for (int i = 0; i < Count; i++)
            lines[i] = new OrderLineClass(9.99m, i % 5 + 1);

        decimal total = 0;
        foreach (OrderLineClass line in lines)
            total += line.Price * line.Quantity;
        return total;
    }

    [Benchmark]
    public decimal Structs()
    {
        var lines = new OrderLineStruct[Count];
        for (int i = 0; i < Count; i++)
            lines[i] = new OrderLineStruct(9.99m, i % 5 + 1);

        decimal total = 0;
        foreach (OrderLineStruct line in lines)
            total += line.Price * line.Quantity;
        return total;
    }
}

public sealed class OrderLineClass
{
    public OrderLineClass(decimal price, int quantity)
    {
        Price = price;
        Quantity = quantity;
    }

    public decimal Price { get; }
    public int Quantity { get; }
}

public readonly struct OrderLineStruct
{
    public OrderLineStruct(decimal price, int quantity)
    {
        Price = price;
        Quantity = quantity;
    }

    public decimal Price { get; }
    public int Quantity { get; }
}

| Method  | Mean     | Error     | StdDev    | Ratio | Gen0   | Gen1   | Allocated | Alloc Ratio |
|-------- |---------:|----------:|----------:|------:|-------:|-------:|----------:|------------:|
| Classes | 7.830 us | 0.0488 us | 0.0433 us |  1.00 | 5.7373 | 0.3510 |   46.9 KB |        1.00 |
| Structs | 5.354 us | 0.0142 us | 0.0119 us |  0.68 | 2.8610 |      - |  23.46 KB |        0.50 |

The class version allocated 46.9 KB: the array itself (a thousand references) plus a thousand separate objects on the heap, each carrying an object header, a decimal and an int. In the struct version there are no objects; the values sit directly inside the array, one after another, and the total memory halved. The time dropped by thirty percent as well. Look at the Gen1 column too: in the class version some objects survived the first collection and were promoted to the next generation; extra work for the GC.

This doesn't mean "make everything a struct". Structs are moved around by copying; passing a large struct to methods or reading it out of collections often can cost you more than you gain. That's why in and readonly struct, which we saw in the ref post, exist. The rule: small, immutable data created in large numbers (an order line, a coordinate, an amount of money) is a good candidate for a struct; things that have identity, change and are shared are classes.

Hidden Allocations

Now back to the opening question. Next to the two sum methods we added a count with LINQ and a hand-written one:

using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
public class HiddenAllocationBenchmarks
{
    private List<int> _stock = new();

    [Params(5)]
    public int Threshold { get; set; }

    [GlobalSetup]
    public void Setup() => _stock = Enumerable.Range(0, 1_000).Select(i => i % 10).ToList();

    [Benchmark(Baseline = true)]
    public int SumList() => Sum(_stock);

    [Benchmark]
    public int SumEnumerable() => Sum((IEnumerable<int>)_stock);

    [Benchmark]
    public int CountWithLambda()
    {
        int threshold = Threshold;
        return _stock.Count(s => s < threshold);
    }

    [Benchmark]
    public int CountWithLoop()
    {
        int count = 0;
        foreach (int s in _stock)
        {
            if (s < Threshold)
                count++;
        }
        return count;
    }

    private static int Sum(List<int> items)
    {
        int total = 0;
        foreach (int item in items)
            total += item;
        return total;
    }

    private static int Sum(IEnumerable<int> items)
    {
        int total = 0;
        foreach (int item in items)
            total += item;
        return total;
    }
}

| Method          | Threshold | Mean       | Error    | StdDev   | Median     | Ratio | RatioSD | Gen0   | Allocated | Alloc Ratio |
|---------------- |---------- |-----------:|---------:|---------:|-----------:|------:|--------:|-------:|----------:|------------:|
| SumList         | 5         |   461.0 ns |  9.16 ns | 20.48 ns |   464.3 ns |  1.00 |    0.07 |      - |         - |          NA |
| SumEnumerable   | 5         | 2,167.8 ns | 43.07 ns | 73.14 ns | 2,184.3 ns |  4.71 |    0.28 | 0.0038 |      40 B |          NA |
| CountWithLambda | 5         | 2,427.4 ns | 47.15 ns | 56.13 ns | 2,433.9 ns |  5.28 |    0.29 | 0.0153 |     128 B |          NA |
| CountWithLoop   | 5         |   431.4 ns |  8.58 ns | 18.28 ns |   441.5 ns |  0.94 |    0.06 |      - |         - |          NA |

Between two methods with the same body there's a 4.7x difference, and the one taking IEnumerable<int> allocates 40 bytes on every call. List<T>'s enumerator is a struct. When the parameter is List<int>, the compiler builds the foreach on that struct; the calls are direct and the JIT can inline them. When the parameter is IEnumerable<int>, GetEnumerator() is called through the interface and returns an IEnumerator<int>: the struct gets boxed onto the heap (boxing, 40 bytes) and every MoveNext and Current becomes an interface call.

CountWithLambda has 128 bytes: LINQ's Count also walks the source as an IEnumerable<T> (the same 40 bytes), and on top of that, because the lambda captures a local variable, a closure object and a delegate are created on every call. The hand-written loop allocates nothing and is five times faster.

The lesson here isn't "don't use LINQ". Two microseconds over a thousand elements is nothing next to the total time of a web request. The lesson is this: on a hot path, in code that runs millions of times a second, these small costs add up, and you only see them by measuring. The MultimodalDistribution warning under the CountWithLoop row is worth noting too: BenchmarkDotNet saw two peaks in the timing distribution. The other jobs running in the background are a likely cause; when you see a warning like that, you repeat the measurement.

Common Pitfalls

Measuring with a Debug build. Optimizations are off in a Debug build and the results are meaningless. BenchmarkDotNet catches this itself and refuses to run:

//    * Assembly Bookstore.Benchmarks which defines benchmarks is non-optimized
Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.

Not using the result. If a benchmark method is void and doesn't put the value it computes anywhere, the JIT can remove the computation. Return the value. LINQ has a special case of this: a benchmark that returns _stock.Where(...) never runs the query; it only creates the object that describes the query (deferred execution). What you measure is a few nanoseconds of object creation. Run the query with ToList(), Count() or an aggregation.

Measuring the setup cost. If you create the test data inside the benchmark method, you measure the data preparation too. Put the preparation in [GlobalSetup]; that's how we created the list above. Use [Params] to try different sizes.

Generalizing a micro result to the application. "Span is twice as fast" is true for this benchmark; it doesn't mean the application will be twice as fast. If 50 milliseconds of a request are spent in the database, the 25 nanoseconds saved on parsing are invisible. First find where the bottleneck is with a profiler or real metrics, then improve that spot at the micro level with BenchmarkDotNet.

Trusting a single run. Repeat the measurements at least once before sharing them, report the machine information, and run on a quiet machine if you can. In the performance post of the LINQ series we apply the same tool to LINQ queries.

A Small Experiment

★☆☆ Add a Count property with [Params(10, 1_000, 100_000)] to ShapeBenchmarks and remove the constant. How do the Ratio and the Gen1/Gen2 columns change as the size grows?

★★☆ Rewrite CountWithLambda with a lambda that uses a const int instead of a local variable. How many bytes does the allocation drop to? Where do the remaining bytes come from?

★★★ Grow OrderLineStruct by adding six more decimal fields, and sum the elements by passing them by value to a method instead of using foreach. Then write the same method with an in parameter. Measure the three versions (class, large struct by value, large struct with in) and interpret the result.

We've learned to talk about performance by measuring. In the next post we move on to how we wire the parts of an application together: the basics of dependency injection with Microsoft.Extensions.DependencyInjection, lifetimes and the captive dependency trap.