AOT Compilation: ReadyToRun, Native AOT and the Startup vs Throughput Trade-off

Version note: The code in this post was built with .NET 10 (SDK 10.0.404, runtime 10.0.15) and ASP.NET Core 10 and verified with xUnit v3 4.0.3 tests; the measurements use BenchmarkDotNet 0.15.8.

Guess First

We write a small API for Bookstore, the online bookstore used as the shared example across this site's series: /books returns the catalog as JSON, and /books?filter=... uses the filter language we have been writing throughout the series. We will publish the same code in three ways: classic JIT, ReadyToRun and Native AOT.

First question: how far ahead of JIT do you expect Native AOT to be in startup time? Where does ReadyToRun land between the two?

Second question: on every request the filter endpoint parses the filter, turns it into an expression tree and calls Compile(). There is no JIT in Native AOT; how many requests per second does this endpoint handle, and is it slower than the JIT version?

The Promise and the Price of AOT

In post 05, machine code was produced while the program ran, method by method and in tiers. AOT moves this work to build time, before the app runs. Its promise comes in three parts: startup gets shorter because ready machine code runs as soon as the process starts; memory goes down because the JIT itself, the IL and its metadata do not sit in memory; and with Native AOT the output is a single executable that does not need the .NET runtime installed on the target machine.

The price comes in three parts too. The first is trimming: Native AOT only puts into the output the code the compiler sees as reachable; a member looked up by name through reflection at run time is not in the output if nothing references it. That is why serializers, mappers and DI libraries that rely on reflection can cause trouble. The second is dynamic code: generating IL at run time with System.Reflection.Emit and handing it to the JIT is impossible because there is no JIT. The third is the platform: the output is for one operating system and architecture. A Windows x64 executable is produced on Windows, a Linux executable on Linux. There is also a less visible price: the Dynamic PGO and tiered compilation from post 05 are gone. Code is optimized once at build time; it is not recompiled from the run-time profile.

ReadyToRun is a hybrid between the two extremes. Precompiled machine code is put into the assembly next to the IL; the runtime and the JIT are still there. According to the ReadyToRun page on Microsoft Learn, this code is not as optimized as the JIT's Tier1 code, so tiered compilation replaces frequently used ReadyToRun methods with code produced by the JIT. That was the "ReadyToRun code → Tier1 instrumented → Tier1" path in the figure of post 05. There are no restrictions: reflection, Emit, everything works.

One API, Three Publishes

The API is written with ASP.NET Core minimal APIs and CreateSlimBuilder. For JSON we use a source generator (ApiJsonContext) instead of reflection; we will come back to why in the pitfalls section:

using System.Runtime.CompilerServices;
using System.Text.Json.Serialization;
using Bookstore.Catalog;
using Bookstore.Filter;

WebApplicationBuilder builder = WebApplication.CreateSlimBuilder(args);
builder.Services.ConfigureHttpJsonOptions(options =>
    options.SerializerOptions.TypeInfoResolverChain.Insert(0, ApiJsonContext.Default));

WebApplication app = builder.Build();

app.MapGet("/books", (string? filter) =>
{
    if (string.IsNullOrWhiteSpace(filter))
        return Results.Ok(SampleBooks.All);

    CompiledFilter compiled = FilterCompiler.Compile(filter);
    if (!compiled.Success)
        return Results.BadRequest(compiled.Diagnostics.Select(d => d.ToString()).ToArray());

    Func<Book, bool> predicate = compiled.Expression!.Compile();
    return Results.Ok(SampleBooks.All.Where(predicate).ToArray());
});

app.MapGet("/runtime", () => new RuntimeInfo(
    RuntimeFeature.IsDynamicCodeSupported,
    RuntimeFeature.IsDynamicCodeCompiled,
    System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription));

app.Run();

public sealed record RuntimeInfo(bool DynamicCodeSupported, bool DynamicCodeCompiled, string Framework);

[JsonSerializable(typeof(IReadOnlyList<Book>))]
[JsonSerializable(typeof(Book[]))]
[JsonSerializable(typeof(string[]))]
[JsonSerializable(typeof(RuntimeInfo))]
internal sealed partial class ApiJsonContext : JsonSerializerContext;

The three publish commands use the same project; only the MSBuild properties differ:

$ dotnet publish samples/Bookstore.Api -c Release -o publish/jit
$ dotnet publish samples/Bookstore.Api -c Release -p:PublishReadyToRun=true -o publish/r2r
$ dotnet publish samples/Bookstore.Api -c Release -p:PublishAot=true -o publish/aot
$ du -sh publish/jit publish/r2r publish/aot/Bookstore.Api
296K	publish/jit
436K	publish/r2r
9.6M	publish/aot/Bookstore.Api

Because we do not pass -r, the SDK produces the ReadyToRun and Native AOT output for the platform of the machine it runs on; on Windows the executable is Bookstore.Api.exe and the sizes come out somewhat different per operating system. The sizes do not measure the same thing: the JIT and ReadyToRun folders are framework-dependent, meaning they use the .NET runtime installed on the machine. The Native AOT executable carries the runtime, the GC and the framework code we use inside itself. Debug symbols live in a separate file (.pdb on Windows); you need to keep them to inspect crash dumps.

To see the effect of ReadyToRun, let's start both versions, send a single request and count how many methods the JIT compiled:

$ DOTNET_JitStdOutFile=jit-jit.txt DOTNET_JitDisasmSummary=1 ASPNETCORE_URLS=http://127.0.0.1:52506 publish/jit/Bookstore.Api > /dev/null &
$ sleep 1; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:52506/books; kill %1
200
$ grep -c 'JIT compiled' jit-jit.txt
1138
$ grep 'JIT compiled' jit-jit.txt | grep -c -E 'Bookstore|Program'
14
$ DOTNET_JitStdOutFile=jit-r2r.txt DOTNET_JitDisasmSummary=1 ASPNETCORE_URLS=http://127.0.0.1:52506 publish/r2r/Bookstore.Api > /dev/null &
$ sleep 1; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:52506/books; kill %1
200
$ grep -c 'JIT compiled' jit-r2r.txt
1090
$ grep 'JIT compiled' jit-r2r.txt | grep -c -E 'Bookstore|Program'
0

ReadyToRun saved Bookstore's 14 methods from the JIT, but the JIT still compiled close to a thousand methods. Most of them are generic instantiations from ASP.NET Core and the BCL and code generated at run time; the framework itself already ships as ReadyToRun. The page on Learn also says that apps with little code should not expect a significant gain from ReadyToRun.

Measurement: Startup, Memory and Throughput

For the measurement I wrote a small tool: StartupProbe starts the given executable and measures the time until the first successful /books response. It repeats this five times and takes the median; on the last run it reads the working set and puts load on two endpoints with 32 concurrent clients for five seconds each:

static async Task<double> WaitForFirstResponse(HttpClient client)
{
    long start = Stopwatch.GetTimestamp();
    while (true)
    {
        try
        {
            using HttpResponseMessage response = await client.GetAsync("/books");
            if (response.IsSuccessStatusCode)
                return Stopwatch.GetElapsedTime(start).TotalMilliseconds;
        }
        catch (HttpRequestException e) when (e.InnerException is SocketException)
        {
            // the server is not listening yet
        }
        await Task.Delay(2);
    }
}

$ dotnet build -c Release samples/StartupProbe
$ dotnet samples/StartupProbe/bin/Release/net10.0/StartupProbe.dll 5 publish/jit/Bookstore.Api
startup median    134 ms | idle   83 MB | /books  121861 rps | filter   8019 rps | after load  259 MB
$ dotnet samples/StartupProbe/bin/Release/net10.0/StartupProbe.dll 5 publish/r2r/Bookstore.Api
startup median    138 ms | idle   83 MB | /books  106552 rps | filter   7768 rps | after load  266 MB
$ dotnet samples/StartupProbe/bin/Release/net10.0/StartupProbe.dll 5 publish/aot/Bookstore.Api
startup median     68 ms | idle   25 MB | /books  113568 rps | filter 118373 rps | after load  124 MB

The load generator and the server ran on the same machine; absolute numbers depend on the machine, the ratios are what to compare. I repeated the measurement three times: Native AOT's startup median stayed between 49 and 68 ms, JIT's between 122 and 142 ms, and the throughput numbers moved by up to 20 percent. So I treat the differences between JIT and ReadyToRun as measurement error.

The answer to the first guess: Native AOT gives its first response in less than half the time JIT needs and sits idle with a third of the memory. ReadyToRun cannot be told apart from JIT in this small app; the JIT compilations we just counted are the reason. On the plain /books endpoint all three are in the same range: the code produced by the JSON source generator does the same work on all three paths.

The answer to the second guess is a surprise: on the filter endpoint Native AOT handles more than ten times as many requests as JIT. We will see why after moving the filter language to Native AOT.

The Filter Language on Native AOT: Warnings First

Before taking this measurement, the filter language had to be prepared for Native AOT. The code generator from post 04 stopped the build on the first publish (paths shortened):

$ dotnet publish samples/Bookstore.Api -c Release -p:PublishAot=true -o publish/aot
...
src/Bookstore.Filter/CodeGen/ExpressionCompiler.cs(71,50): error IL3050: Using member 'System.Type.MakeGenericType(params Type[])' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. The native code for this instantiation might not be available at runtime. [src/Bookstore.Filter/Bookstore.Filter.csproj]
src/Bookstore.Filter/CodeGen/ExpressionCompiler.cs(72,16): error IL2026: Using member 'System.Linq.Expressions.Expression.Field(Expression, String)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. Creating Expressions requires unreferenced code because the members being referenced by the Expression may be trimmed. [src/Bookstore.Filter/Bookstore.Filter.csproj]

Both warnings come from the lines that produce parameterized literals. MakeGenericType built Captured<T> at run time with a type argument. Native AOT must produce the machine code of generic types at build time; there may be no code for a Captured<decimal> that only shows up at run time (IL3050). Expression.Field(..., "value") looked the field up by name through reflection; the trimmer cannot see that the field is used (IL2026). The warnings came as errors because the project has TreatWarningsAsErrors on. The fix is to write every type and member so that it is visible at compile time:

// every holder type is written out, so Native AOT knows each Captured<T> at compile time
return literal.Value switch
{
    int value => Captured<int>.Read(value),
    decimal value => Captured<decimal>.Read(value),
    string value => Captured<string>.Read(value),
    _ => Captured<bool>.Read((bool)literal.Value),
};

// same shape the C# compiler produces for a captured local: a field read on a closure object
private sealed class Captured<T>(T value)
{
    private static readonly FieldInfo ValueField = typeof(Captured<T>).GetField("value")!;

    // the field name becomes the SQL parameter name (@value, @value1, ...)
    public readonly T value = value;

    public static MemberExpression Read(T value) => Expression.Field(Expression.Constant(new Captured<T>(value)), ValueField);
}

typeof(Captured<T>).GetField("value") is a call the trimmer can analyze: the type and the name are constants. To see these warnings on every build, not only on publish, we added one line to the library:

<PropertyGroup>
    <RootNamespace>Bookstore.Filter</RootNamespace>
    <!-- trim and AOT analyzers run on every build, not only when an app publishes with Native AOT -->
    <IsAotCompatible>true</IsAotCompatible>
</PropertyGroup>

No Warnings, No Problem?

The second publish finished without warnings. The first filtered request returned 500:

$ ASPNETCORE_URLS=http://127.0.0.1:52506 publish/aot/Bookstore.Api > aot.log &
$ curl -s -i 'http://127.0.0.1:52506/books?filter=price%20%3C%20300' | head -1
HTTP/1.1 500 Internal Server Error
$ grep -A5 'fail:' aot.log
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNQ2OS56RLSN", Request id "0HNQ2OS56RLSN:00000001": An unhandled exception was thrown by the application.
      System.InvalidOperationException: The binary operator LessThan is not defined for the types 'System.Decimal' and 'System.Decimal'.
         at System.Linq.Expressions.Expression.GetUserDefinedBinaryOperatorOrThrow(ExpressionType, String, Expression, Expression, Boolean) + 0x148
         at Bookstore.Filter.CodeGen.ExpressionCompiler.EmitComparison(BoundComparison, ParameterExpression, LiteralMode) + 0xc4
         at Bookstore.Filter.CodeGen.ExpressionCompiler.Emit(BoundExpr, ParameterExpression, LiteralMode) + 0x190

< for decimal is an operator method, decimal.op_LessThan. Expression.LessThan(left, right) finds it through reflection at run time. In a JIT app there is no problem; in Native AOT the trimmer saw the method as never called, removed it, and the publish did not report it with a warning. The fix is the same: name the method ourselves and pass it to the expression. The C# compiler also puts the operator method into decimal comparisons itself when it turns a lambda into an expression tree:

// decimal and string compare through operator methods; naming them here keeps them in a Native AOT build
MethodInfo? method = left.Type == typeof(decimal) ? OperatorMethod(typeof(decimal), name)
    : left.Type == typeof(string) ? OperatorMethod(typeof(string), name)
    : null;
return Expression.MakeBinary(kind, left, right, liftToNull: false, method);

private static MethodInfo OperatorMethod(
    [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicMethods)] Type type, string name) =>
    type.GetMethod(name, [type, type])!;

[DynamicallyAccessedMembers] gives the trimmer a contract: the public methods of the type that arrives in this parameter must be kept. Because the callers pass typeof(decimal) and typeof(string), the trimmer knows whose methods to keep. New tests check that every comparison carries its operator method; the SQL produced by EF Core did not change either. Now the endpoint works:

$ curl -s http://127.0.0.1:52506/runtime
{"dynamicCodeSupported":false,"dynamicCodeCompiled":false,"framework":".NET 10.0.15"}

The lesson of this section: trimming warnings are necessary but not sufficient. Every code path that uses reflection must be tested by running the Native AOT output itself.

Expression.Compile() Falls Back to an Interpreter on Native AOT

dynamicCodeSupported: false in the /runtime output says that code cannot be generated at run time on Native AOT. The Native AOT page on Learn also says that System.Linq.Expressions always uses its interpreted form in this mode: instead of generating IL, Compile() returns an interpreter that walks the expression tree. On a JIT runtime we can choose the same interpreter with Compile(preferInterpretation: true); let's compare the two paths with BenchmarkDotNet (the run benchmarks work on 1,000 books):

$ dotnet run -c Release --project benchmarks/Bookstore.Filter.Benchmarks -- --filter '*CompileBenchmarks*'
...
| Method                | Mean         | Error       | StdDev      | Gen0    | Gen1   | Allocated |
|---------------------- |-------------:|------------:|------------:|--------:|-------:|----------:|
| CompileToIL           | 136,640.9 ns | 2,448.78 ns | 2,405.03 ns |  0.4883 | 0.2441 |    5777 B |
| CompileForInterpreter |     738.4 ns |    14.09 ns |    26.13 ns |  0.2728 | 0.0010 |    2288 B |
| RunCompiled           |   4,363.1 ns |    85.26 ns |   144.77 ns |       - |      - |         - |
| RunInterpreted        |  73,148.7 ns | 1,452.99 ns | 2,544.79 ns | 31.4941 |      - |  263984 B |

The table explains both guesses together. Compiling to IL is expensive: a DynamicMethod is generated and the JIT is called, about 137 microseconds per filter. For the interpreter, "compiling" is almost free. In exchange, the compiled delegate runs about 17 times faster and allocates nothing; the interpreter allocates about 264,000 bytes for 1,000 books. Our endpoint compiles on every request and runs on six books, so the compile cost dominates; the JIT version pays that cost on every request. Native AOT is faster here, but for the wrong reason. The right fix, as we said in post 04, is to cache the compiled delegate by filter text; then the JIT version compiles once and runs the fastest code.

Other Meanings of the Word "AOT"

AOT is not a technique but a timing: compilation that happens before the app runs. What is produced depends on the context. According to Learn, ReadyToRun is a form of AOT too, but its output is machine code that sits next to the IL. Angular's AOT compiler turns HTML templates and the TypeScript code of components into JavaScript at build time; the output is not machine code but JavaScript that the browser's JITs will compile again. In new Angular projects ng build runs with AOT by default.

TypeScript 7 touches AOT from another side: the compiler itself was rewritten in Go and ships as a native executable. What got faster is the tool that compiles TypeScript; the JavaScript it produces and the V8 that runs it did not change. The details are in the TypeScript Advanced series post "TypeScript 7 and Beyond: The Go-Based Compiler and the New Defaults". Native AOT support on the ASP.NET Core side, and which features do not work, are covered in the ASP.NET Core Web API series post "What's New in ASP.NET Core with .NET 8: Keyed DI, IExceptionHandler, Identity API and Native AOT".

Production Notes

A Native AOT build is platform-specific and needs its own toolchain. On Windows you need the "Desktop development with C++" workload of Visual Studio, on Linux clang. In CI you need a separate job for each target operating system and architecture; for a Linux container you produce the executable on Linux (or in a Linux build container).

The gain is largest where many instances start and stop often: services that scale to zero, short-lived jobs, CLI tools. Learn also says the benefit shows most in cloud services with many instances. In an API that stays up for hours doing heavy work, the startup difference is paid once; there the throughput that Dynamic PGO buys may be worth more. Before deciding, publish your own app three ways and measure it as above.

Dependencies must be AOT-compatible too. Libraries that rely on reflection produce warnings on publish. The VerifyReferenceAotCompatibility setting that came with .NET 10 lists references not marked as AOT-compatible with the IL3058 warning.

Common Pitfalls

Suppressing trimming warnings. #pragma warning disable IL2026 or SuppressTrimAnalysisWarnings turns the build green, but the warning is often the only signal. The price of a suppressed warning is a 500 on the first request in production that goes down that code path. As we saw, sometimes there is not even a warning; so run the Native AOT output with integration tests.

Using reflection-based libraries as they are. The reflection-based serialization of System.Text.Json is disabled by default when trimming or Native AOT is on; that is why ApiJsonContext exists in the API. The source generator produces the serialization code at compile time, so the trimmer sees what is used. Dapper applies the same idea with Dapper.AOT, covered in the Dapper series post "Dapper.AOT: Dapper with Source Generators, Interceptors and Native AOT".

Assuming "AOT is always faster". AOT reliably improves startup and memory; it may not improve peak throughput. There is no Dynamic PGO, expression trees are interpreted, and some generic code is compiled more generally. AOT coming out more than ten times ahead on the filter endpoint was not AOT's merit either, but the result of our mistake of compiling on every request.

A Small Experiment

★☆☆ Publish the Native AOT output again with -p:OptimizationPreference=Size. How did the size of the executable and the numbers from StartupProbe change?

★★☆ Cache the compiled delegates in the API by filter text (with a size-limited ConcurrentDictionary or IMemoryCache). Run StartupProbe again for the three outputs; how did the ranking change on the filter endpoint?

★★★ Remove the [DynamicallyAccessedMembers] attribute in OperatorMethod and look at the output of both dotnet build and the Native AOT publish. Which warning appeared? Then write a CompileFilter helper that, based on RuntimeFeature.IsDynamicCodeSupported, uses the interpreter on Native AOT and IL compilation on a JIT runtime, and test both paths.

In this post we changed when machine code is produced. In the next post we move to the other big part of the runtime: the garbage collector. We will look at mark-sweep-compact, generations, the write barrier, and .NET's Server GC with DATAS through the memory of the API above, which climbs to around 260 MB under load.