Testing with Aspire: End-to-End Tests That Start the Whole System with Aspire.Hosting.Testing

Version note: The code in this post was built and run with .NET 10, Aspire 13.6 (Aspire.Hosting.Testing 13.6.5) and xUnit 2.9; the tests ran against SQL Server 2025, RabbitMQ 4.3 and Keycloak 26.6 containers.

Guess First

We started from Aspire's test template and, as the template's comments suggest, added the standard resilience handler to the test's HttpClients. The first test adds a book and looks for it in the list. On every test run the SQL Server container starts from scratch and the database is empty. On the first run the test failed with this:

Failed Bookstore.AppHost.Tests.CatalogTests.AddedBookAppearsInTheList [13 s]
  Error Message:
   Assert.Equal() Failure: Values differ
Expected: Created
Actual:   Conflict

How does the first book added to an empty database get a 409 Conflict? And why did the test take 13 seconds? The answer is in two separate places: one on the test side, one in the Redis decision we made in the previous post. Let's set up the test infrastructure first and then come back to this failure.

The Test Project and Starting the AppHost in a Test

The Aspire.Hosting.Testing package runs the AppHost inside a test: like dotnet run, it really starts the containers and processes, and it shuts them down when the test ends. In other words, it's Aspire's way of writing end-to-end tests that run all of the system's parts together. The dotnet new aspire-xunit template generates a project with xUnit 2.9; we put it in the tests/Bookstore.AppHost.Tests folder of Bookstore, the online bookstore used as the shared example across this site's series, and referenced the AppHost project:

<ItemGroup>
  <PackageReference Include="Aspire.Hosting.Testing" />
  <PackageReference Include="Microsoft.NET.Test.Sdk" />
  <PackageReference Include="xunit" />
  <PackageReference Include="xunit.runner.visualstudio" />
</ItemGroup>

<ItemGroup>
  <ProjectReference Include="..\..\src\Bookstore.AppHost\Bookstore.AppHost.csproj" />
</ItemGroup>

The reference generates the Projects.Bookstore_AppHost type. Restarting the AppHost for every test is expensive, so, as the documentation also recommends, we start it once per test class: IClassFixture and IAsyncLifetime in xUnit. The core of the fixture:

// One AppHost per test class. Resources the scenario does not need stay stopped.
public abstract class BookstoreAppFixture(params string[] notNeeded) : IAsyncLifetime
{
    private static readonly TimeSpan StartTimeout = TimeSpan.FromMinutes(3);

    // Generated per run: no user secrets needed on a CI machine
    public string BackofficeSecret { get; } = Guid.NewGuid().ToString("N");

    public DistributedApplication App { get; private set; } = null!;

    public async Task InitializeAsync()
    {
        using CancellationTokenSource cts = new(StartTimeout);

        IDistributedApplicationTestingBuilder builder = await DistributedApplicationTestingBuilder.CreateAsync<Projects.Bookstore_AppHost>(
        [
            $"Parameters:sql-password={Guid.NewGuid():N}Aa1!",
            $"Parameters:backoffice-client-secret={BackofficeSecret}"
        ], cts.Token);

        // ... (below)

        App = await builder.BuildAsync(cts.Token);
        await App.StartAsync(cts.Token);
        await WaitUntilReadyAsync(App, cts.Token);
    }

    protected abstract Task WaitUntilReadyAsync(DistributedApplication app, CancellationToken ct);

    public async Task DisposeAsync()
    {
        if (App is not null)
        {
            await App.DisposeAsync();
        }
    }
}

Three things to note. First, parameters: in 02 we read sql-password and in 05 backoffice-client-secret from user secrets. A CI machine has no user secrets; we pass the values to the AppHost as arguments, and the configuration system reads the Parameters:... keys from there. The values are generated anew on every run, and the test can use the same secret when it asks Keycloak for a token. Second, waiting: StartAsync tells you the resources have started, not that they're ready. For that there's App.ResourceNotifications.WaitForResourceHealthyAsync("catalog", ct); for a worker without a health check, WaitForResourceAsync("inventory", KnownResourceStates.Running, ct). Third, the timeout: the whole startup is tied to a single CancellationTokenSource. If a container image can't be pulled, the test doesn't hang forever; it fails after 3 minutes.

In tests we reach the services with App.CreateHttpClient("gateway"): that client knows the resource's real address for that run. The testing builder picks random ports by default, so two test runs on the same machine, or an open development AppHost, don't collide (the DcpPublisher:RandomizePorts=false argument turns this off).

A Test Must Not Touch the Developer's Database

Our AppHost was written for development: the SQL Server container has a fixed name (aspire-sql), a Persistent lifetime and its data in the aspire-sql-data volume. All three are a problem in a test. A persistent container keeps running after the AppHost stops; with the same name and volume, the test writes into the developer's database. On our first attempt we tried to delete the annotations by hand and saw in docker ps that the container was still running after the test. The builder APIs do this safely by replacing the same annotation:

// Tests must not touch the developer's database: own container names, session lifetime, no volumes
foreach (ContainerResource container in builder.Resources.OfType<ContainerResource>().ToList())
{
    builder.CreateResourceBuilder(container)
        .WithContainerName($"aspire-test-{container.Name}")
        .WithLifetime(ContainerLifetime.Session);
    foreach (ContainerMountAnnotation volume in container.Annotations.OfType<ContainerMountAnnotation>()
                 .Where(m => m.Type == ContainerMountType.Volume).ToList())
    {
        container.Annotations.Remove(volume);
    }
}

// A small catalog is enough for tests; seeding 200,000 books is not
builder.CreateResourceBuilder<ProjectResource>("migrations").WithEnvironment("Seed__GeneratedBooks", "0");

CreateResourceBuilder is how a test gets back to a resource the AppHost added; with WithEnvironment you can override a service's variable (the one added later wins). Here we set the 200,000-book seed we gave the migration worker in 04 back to zero; the test data lives in the test itself.

Each Scenario Starts Its Own Slice

Bookstore's AppHost now has four containers: SQL Server, Redis, RabbitMQ and Keycloak. The order scenario doesn't need Keycloak, and the add-book scenario doesn't need RabbitMQ. The documentation offers two ways for this: resources that depend on configuration in the AppHost, or WithExplicitStart. To avoid changing the AppHost for tests, we applied the second one from the test; the names given to the fixture aren't started:

foreach (string name in notNeeded)
{
    builder.CreateResourceBuilder<IResource>(name).WithExplicitStart();
}

public sealed class OrderingFixture() : BookstoreAppFixture("redis", "keycloak", "web", "web-installer")
{
    protected override async Task WaitUntilReadyAsync(DistributedApplication app, CancellationToken ct)
    {
        await app.ResourceNotifications.WaitForResourceHealthyAsync("ordering", ct);
        await app.ResourceNotifications.WaitForResourceAsync("inventory", KnownResourceStates.Running, ct);
        await app.ResourceNotifications.WaitForResourceHealthyAsync("gateway", ct);
    }
}

The Angular app (web) and web-installer, which runs its npm install, don't start in either scenario; these tests verify the APIs, and UI testing is done with Playwright in the Integration ve E2E Test series. During the run we looked at docker ps every five seconds; there were never more than two containers at once:

aspire-test-sql aspire-test-messaging
aspire-test-sql aspire-test-keycloak

One more condition: by default xUnit runs different test classes in parallel. Since each fixture starts its own containers, in parallel two SQL Servers would come up at the same time. We turned that off on purpose:

// Every fixture starts its own containers; running classes in parallel would double them
[assembly: CollectionBehavior(DisableTestParallelization = true)]

"Add a Book, See It in the List" and the Cause of the 409

[Fact]
public async Task AddedBookAppearsInTheList()
{
    using HttpClient gateway = fixture.App.CreateHttpClient("gateway");
    gateway.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", await GetBackofficeTokenAsync());

    using HttpResponseMessage created = await gateway.PostAsJsonAsync("/api/books/",
        new { isbn = "9780140449136", title = "Crime and Punishment", author = "Fyodor Dostoevsky", price = 310.00m });
    Assert.Equal(HttpStatusCode.Created, created.StatusCode);

    List<BookDto>? page = await gateway.GetFromJsonAsync<List<BookDto>>("/api/books?page=1");
    Assert.Contains(page!, b => b.Isbn == "9780140449136");
}

GetBackofficeTokenAsync sends a client_credentials request to Keycloak's token endpoint with App.CreateHttpClient("keycloak", "http"), using the secret the fixture generated. The requests go through the gateway, as in the real app.

Back to the 409 from Guess First. The AddStandardResilienceHandler the template suggests includes a 10-second timeout per attempt and a retry, and it repeats a POST too, without looking at the HTTP method. When the first POST didn't answer within 10 seconds, the handler cut the request and sent the same POST again. The server, however, had kept processing the first request and saved the book; the second attempt got 409. That's also where the 13 seconds come from: a 10-second wait, a short backoff and a fast 409. When we turned retry off for POST (options.Retry.DisableForUnsafeHttpMethods()), the failure became honest:

Polly.Timeout.TimeoutRejectedException : The operation didn't complete within the allowed timeout of '00:00:10'.

The real question is this: why does inserting one row into an empty database take more than 10 seconds? In this scenario we didn't start Redis, but Catalog's Redis reference is still there. Redis's address is the port of the DCP proxy we met in 01; even without a container behind it, the proxy accepts the connection and never answers. When a book is added, Catalog tries to clear the list's cache, and StackExchange.Redis waits to connect. The FailFast from 05 doesn't help here: the connection was never established, and the error in the log shows the client still trying to connect (state: ConnectedEstablishing). We measured the same situation in the development AppHost without starting Redis: the first POST took 17.96 seconds, the next one 4.99 seconds, with Could not evict the book list cache warnings 5 seconds apart in the log.

We could have started Redis for the test too. But this scenario isn't about the cache, and Catalog being able to work without Redis was already our decision in 05. We moved the decision into code: if there's no Redis connection string, Catalog keeps the output cache in memory.

// The cache is optional: without Redis the catalog must still answer, and quickly
if (string.IsNullOrEmpty(builder.Configuration.GetConnectionString("redis")))
{
    // No Redis configured (a test, or the catalog run on its own): keep the cache in memory
    builder.Services.AddOutputCache();
}
else
{
    builder.AddRedisOutputCache("redis",
        settings => settings.DisableHealthChecks = true,
        options => options.BacklogPolicy = BacklogPolicy.FailFast);
}

public sealed class CatalogFixture() : BookstoreAppFixture("redis", "messaging", "ordering", "inventory", "web", "web-installer")
{
    // Redis stays stopped, but its address would still be the DCP proxy, which accepts and never answers.
    // An empty connection string switches the catalog to its in-memory output cache.
    protected override void Configure(IDistributedApplicationTestingBuilder builder) =>
        builder.CreateResourceBuilder<ProjectResource>("catalog").WithEnvironment("ConnectionStrings__redis", "");
    ...
}

With this change eviction really works as well: the books tag is cleared in the in-memory cache, and the list shows the new book right away. We removed the template's resilience handler entirely. The fixture already waits for the resources to be healthy; after that, repeating a request does nothing but hide a failure, and repeating a POST makes the test itself wrong.

"Place an Order, It Becomes Confirmed" and Task.Delay

The order flow is asynchronous: the POST response returns Pending, and the confirmation arrives over RabbitMQ a few milliseconds later. The first version that comes to mind:

await Task.Delay(TimeSpan.FromSeconds(1));
OrderDto? order = await gateway.GetFromJsonAsync<OrderDto>($"/api/orders/{placed.Id}");
Assert.Equal("Confirmed", order!.Status);

This test is bad in two ways. Locally the confirmation arrives within a few milliseconds, so every run wastes a second; on a busy CI machine, if the confirmation comes later than one second, the test fails at random. Increasing the delay reduces the second problem and makes the first one worse. The right way is to ask again at short intervals until the condition holds and to fail with a meaningful message at an upper limit:

// Asks again until the condition holds; fails with the last value instead of hanging
public static async Task<T> UntilAsync<T>(Func<CancellationToken, Task<T>> probe, Func<T, bool> condition, TimeSpan timeout)
{
    using CancellationTokenSource cts = new(timeout);
    T last = await probe(cts.Token);
    while (!condition(last))
    {
        try
        {
            await Task.Delay(TimeSpan.FromMilliseconds(100), cts.Token);
            last = await probe(cts.Token);
        }
        catch (OperationCanceledException) when (cts.IsCancellationRequested)
        {
            Assert.Fail($"Condition not met within {timeout.TotalSeconds} s. Last value: {last}");
        }
    }

    return last;
}

[Fact]
public async Task PlacedOrderIsConfirmedAfterStockIsReserved()
{
    using HttpClient gateway = fixture.App.CreateHttpClient("gateway");

    using HttpResponseMessage response = await gateway.PostAsJsonAsync("/api/orders/", new
    {
        customerEmail = "ada@example.com",
        items = new[] { new { isbn = "9780441172719", quantity = 1 }, new { isbn = "9780547928227", quantity = 2 } }
    });
    Assert.Equal(HttpStatusCode.Created, response.StatusCode);
    OrderDto? placed = await response.Content.ReadFromJsonAsync<OrderDto>();
    Assert.Equal("Pending", placed!.Status);
    Assert.Equal(1100.00m, placed.Total);

    // Confirmation arrives over RabbitMQ: ask again instead of sleeping a fixed time
    OrderDto confirmed = await Polling.UntilAsync(
        async ct => (await gateway.GetFromJsonAsync<OrderDto>($"/api/orders/{placed.Id}", ct))!,
        order => order.Status == "Confirmed",
        TimeSpan.FromSeconds(10));

    Assert.Equal("Confirmed", confirmed.Status);
}

The Pending assertion is deliberate too: it verifies that the response returns before the asynchronous flow. If Ordering ever starts waiting for the confirmation synchronously, this line fails and the design change gets noticed. The output of the last run:

dotnet test tests/Bookstore.AppHost.Tests --logger "console;verbosity=normal"

  Passed Bookstore.AppHost.Tests.OrderingTests.PlacedOrderIsConfirmedAfterStockIsReserved [346 ms]
  Passed Bookstore.AppHost.Tests.CatalogTests.AddedBookAppearsInTheList [433 ms]
  Passed Bookstore.AppHost.Tests.CatalogTests.AddingABookWithoutATokenIsRejected [2 ms]
Total tests: 3
     Passed: 3
 Total time: 2.0546 Minutes

Which Test Level Does It Fit?

The three tests themselves take less than a second; nearly all of the two minutes is starting the containers, SQL Server, the migration and Keycloak. An AppHost test is therefore expensive, and it's most valuable when it verifies the connections between the system's parts: the gateway route, service discovery, a token from Keycloak being validated in Catalog, the confirmation coming back over RabbitMQ. These can't be tested from inside a single service.

The details of WebApplicationFactory and Testcontainers are in the Integration ve E2E Test series. One more limit: an Aspire test runs the AppHost and the services as separate processes. The test can't put a fake object into a service's DI; you can change behavior only through environment variables, configuration and AppHost arguments. That's also why we gave Catalog an empty connection string.

Running in CI

Because the tests start real containers, the CI machine needs a container runtime. GitHub's hosted Linux runners come with Docker installed; the job doesn't need anything extra:

name: AppHost tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  apphost-tests:
    # Hosted Linux runners come with Docker; the tests start real containers
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          global-json-file: global.json
      - name: Test
        run: dotnet test tests/Bookstore.AppHost.Tests --logger "console;verbosity=normal"

We validated the workflow with actionlint but didn't run it on GitHub; the timings come from the run on our machine. Windows hosted runners don't run Linux containers, so this job should stay on Linux. Since the parameter values are generated inside the test, there's no need to define secrets. Images are pulled again on every run; the SQL Server image is large, so it sets the duration of the first step. The details of GitHub Actions (caching, artifacts, choosing runners) are in the GitHub Actions series.

What you miss most in a failed CI run is the dashboard. With 13.6, DistributedApplicationTestingBuilderOptions.EnableDashboard and GetDashboardUrlAsync() open a dashboard in a test and give you a signed-in URL; we're quoting the release notes here and didn't try it in this post. More practical in CI is writing resource logs into the test output: the testing builder forwards resource logs to ILogger, and with --logger "console;verbosity=normal" the warnings from SQL Server and the services show up in the test output.

In Production

These tests protect the road to production, not production itself; the place to be careful is CI. Use one AppHost per class, not per test; as the number of scenarios grows, consider merging fixtures. Every new fixture means a few dozen seconds.

Layer your timeouts: an upper limit for fixture startup (3 minutes for us), a limit for each asynchronous wait (10 seconds for us), and an upper limit for the job (timeout-minutes). A wait without a limit turns into a CI job that hangs for hours.

The passwords and secrets generated in the test disappear with the test process; don't write them to the log. As we saw in the aspire describe output, connection strings carry the password in plain text, and verbose logs can capture these values.

Common Pitfalls

Leaving the template's resilience handler on without thinking. It repeats a POST and produces failures like the 409 that come from the retry, not from the test.

A test AppHost using development data. A persistent container and volume are persistent in the test too; change the container name, lifetime and volumes in the test.

Assuming a resource that isn't started costs nothing. The reference stays, and the address points to a proxy that doesn't answer; you wait as long as the client's timeout.

Waiting for an asynchronous result with Task.Delay. You get a test that is either slow or fails at random; poll for the condition and set an upper limit.

Testing everything with AppHost tests. Starting the whole system for two minutes to check a validation rule turns the test suite into something nobody wants to run.

A Small Experiment

★☆☆ Add the template's AddStandardResilienceHandler back to the fixture and remove the empty connection string line for Catalog. Do you see the 409 on your machine too? How many seconds does the test take?

★★☆ Remove the DisableTestParallelization line and run the tests. How many containers start at the same time, and how does the total duration change?

★★★ Write a "the order stays Pending if no stock is reserved" test with a fixture that doesn't start Inventory. How would you use the polling helper to verify that the condition never happens; notice what determines the duration of that test.

We've seen how to bring the system up in a test; what's left is taking it off our machine. In the next post we use aspire publish, aspire deploy and aspire do to produce a Docker Compose file, a Helm chart for Kubernetes and Azure output from the same AppHost; that's where we'll see the parameter placeholders promised in 02 and what WithExternalHttpEndpoints becomes in the deploy output.