Testing Microservices: Service Tests, End-to-End Tests with Aspire and Contract Tests

Version note: The code in this post was built and run with .NET 10, Aspire 13.6 (Aspire.Hosting.Testing 13.6.5), Testcontainers for .NET 4.16, PactNet 5.0, Microsoft.AspNetCore.Mvc.Testing 10.0 and xUnit v3.

Guess First

In post 07 we made a claim about the saga: if the payment and the 15-minute timeout arrive at the same time, the saga's Version column rejects the second save and Wolverine retries the message. We never actually tried it. If we send PaymentSucceeded and CheckoutTimedOut to the same order at the same time, ten times, how many conflicts happen and how many orders end up inconsistent? Second question: the Catalog team renames the author field in the book response to authorName. Catalog's own tests pass. Which test, in which pipeline, should catch this change? The answers are in the real output of the tests we write in this post.

The Test Pyramid in Microservices

Until now we mostly tried the behaviour of Bookstore, the online bookstore used as the shared example across this site's series, by hand: we placed orders in a browser, stopped services, read logs. We had 33 unit tests; the saga, the domain model, the webhook signature and the metrics were tested in milliseconds. But those tests don't show that a message really goes to RabbitMQ, that two services speak the same JSON, or that the whole system works together. In the "Test Pyramid, Testing Trophy ve Honeycomb" post of the Test Strategy series we explained why the honeycomb shape stands out for microservices: the thickest layer is the tests at the service's own boundary. In Bookstore the layers turned out like this:

The durations were measured on this machine, with the Docker images already pulled.

A Service Test: WebApplicationFactory and Testcontainers

The service test treats Ordering as a real process boundary: its own SQL Server and RabbitMQ in containers, Program.cs as it is, only authentication replaced. The details of WebApplicationFactory and Testcontainers are in the second and third posts of the Integration and E2E Testing series; here we look at the parts specific to microservices:

public async ValueTask InitializeAsync()
{
    await Task.WhenAll(sql.StartAsync(), rabbitmq.StartAsync());
    string orderingDb = new SqlConnectionStringBuilder(sql.GetConnectionString()) { InitialCatalog = "orderingdb" }.ConnectionString;

    // Schema first (the migrator's job in Aspire): Wolverine creates its own tables in this database at startup.
    // Then the data Ordering needs: its copy of two books.
    await using (OrderingDbContext db = new(new DbContextOptionsBuilder<OrderingDbContext>().UseSqlServer(orderingDb).Options))
    {
        await db.Database.MigrateAsync();
        db.CatalogBooks.AddRange(
            new CatalogBook { Isbn = "9780141439587", Title = "Emma", Price = 249.90m, Currency = "TRY", PriceVersion = 1, TitleVersion = 1 },
            new CatalogBook { Isbn = "9780441172719", Title = "Dune", Price = 489m, Currency = "TRY", PriceVersion = 1, TitleVersion = 1 });
        await db.SaveChangesAsync();
    }

    Factory = new WebApplicationFactory<Program>().WithWebHostBuilder(web =>
    {
        // The same names Aspire's WithReference gives the service
        web.UseSetting("ConnectionStrings:orderingdb", orderingDb);
        web.UseSetting("ConnectionStrings:rabbitmq", RabbitMqUri);
        web.ConfigureLogging(logging => logging.AddProvider(Logs));
        web.ConfigureTestServices(services =>
        {
            // No Keycloak in this test: every request is Ada, a customer
            services.AddAuthentication(TestUser.SchemeName)
                .AddScheme<AuthenticationSchemeOptions, TestUser>(TestUser.SchemeName, _ => { });
        });
    });
}

In the first attempt we ran the migration through the factory's services, and every test failed with the same error: Error Number:4060, meaning orderingdb could not be opened. Touching the factory's Services starts the host; Wolverine tries to create its inbox/outbox tables at startup, and the database does not exist yet. In Aspire that order was guaranteed by WaitForCompletion(migrator) (post 02); in the test you set up the same order yourself. We added nothing to Program.cs for testing: in .NET 10 the Program class generated from top-level statements is public, visible to tests.

The test takes the messages from RabbitMQ itself. It binds a queue to Ordering's exchange (playing Notifications) and declares the inventory.commands queue itself (playing Inventory). As we said in post 07, Ordering does not declare that queue because of ExternallyOwned(); if the test did not declare it, ReserveStock would be lost silently. When a message arrives the consumer writes it to a channel and the test waits on it; no polling loop, just a 15-second upper bound:

[Fact]
public async Task An_accepted_order_publishes_OrderPlaced_and_asks_Inventory_for_stock()
{
    HttpClient client = ordering.Factory.CreateClient();
    // The test plays Notifications (a queue on Ordering's exchange) and Inventory (its command queue)
    await using QueueProbe events = await QueueProbe.StartAsync(ordering.RabbitMqUri, "test.ordering-events", "bookstore.ordering");
    await using QueueProbe inventory = await QueueProbe.StartAsync(ordering.RabbitMqUri, "inventory.commands");

    HttpResponseMessage response = await client.PostAsJsonAsync("/api/v1/orders",
        new { items = new[] { new { isbn = "9780141439587", quantity = 2, unitPrice = 249.90m } } }, Ct);

    Assert.Equal(HttpStatusCode.Accepted, response.StatusCode);
    ReceivedMessage placed = await events.NextAsync("Bookstore.Contracts.Ordering.OrderPlaced", Wait);
    Assert.Contains($"\"customerId\":\"{TestUser.Subject}\"", placed.Body);
    Assert.Contains("\"total\":499.80", placed.Body);
    // The trace context travels with the message (post 12)
    Assert.StartsWith("00-", placed.Headers["parent-id"]);
    ReceivedMessage reserve = await inventory.NextAsync("Bookstore.Contracts.Inventory.ReserveStock", Wait);
    Assert.Contains("\"quantity\":2", reserve.Body);
}

This test checks three things together that unit tests cannot show: that the outbox really drains after the commit, that the exchange and queue names are right, and that the JSON leaves in the shape consumers expect (camelCase, customerId from the token). Including container startup the test class took about 15 seconds. We turned off the clean-up container Testcontainers starts by default (Ryuk) with TESTCONTAINERS_RYUK_DISABLED=true and delete the containers ourselves in DisposeAsync; in CI leaving Ryuk on is safer, because it also cleans up the containers of a run that was cut off.

Actually Trying the Saga Race

For the first Guess First question we wrote a second test on the same fixture. An order is placed, the test, playing Inventory, sends StockReserved, and the saga starts waiting for the payment. Then the payment and the timeout are sent at the same time, from separate IMessageBus scopes:

// The payment and the timeout for the same saga, at the same moment, each with its own bus
await Task.WhenAll(
    Bus().InvokeAsync(new PaymentSucceeded(orderId, $"pay_race_{i}"), Ct),
    Bus().InvokeAsync(new CheckoutTimedOut(orderId, TimeSpan.Zero), Ct));

await using AsyncServiceScope scope = ordering.Factory.Services.CreateAsyncScope();
OrderingDbContext db = scope.ServiceProvider.GetRequiredService<OrderingDbContext>();
Order order = await db.Orders.AsNoTracking().SingleAsync(o => o.Id == new OrderId(orderId), Ct);
CheckoutSaga? saga = await db.Set<CheckoutSaga>().AsNoTracking().SingleOrDefaultAsync(s => s.Id == orderId, Ct);

// Only two consistent endings: paid and waiting for the stock commit, or cancelled with a refund on its way
if (order.Status == OrderStatus.Paid)
{
    Assert.Equal(CheckoutStep.AwaitingStockCommit, saga?.Step);
    outcomes.Add("payment first");
}
else
{
    Assert.Equal(OrderStatus.Cancelled, order.Status);
    Assert.Null(saga);
    await payments.NextAsync("Bookstore.Contracts.Payments.RefundPayment", Wait);
    outcomes.Add("timeout first");
}

The test runs this block ten times and counts the conflicts in Ordering's logs. The result:

passed Bookstore.Ordering.IntegrationTests.PlaceOrderTests.A_payment_racing_the_timeout_leaves_the_order_consistent (1s 675ms)
    payment first: 10; saga version conflicts: 9

Error: Invocation of CheckoutTimedOut { DelayTime = 00:00:00, OrderId = 01a1dad9-f5f0-757e-8b76-179cc9d2e0f6, After = 00:00:00 } failed! SagaConcurrencyException

In nine of the ten attempts both messages loaded the saga at the same time and the timeout's save hit the Version check: Wolverine logged SagaConcurrencyException, the retry rule from post 07 ran the message again, and this time the timeout handler found the saga at AwaitingStockCommit and did nothing. Zero orders were left inconsistent. The payment always winning is a property of this measurement, not of the rule: which InvokeAsync saves first depends on scheduling, and the test accepts both endings. The claim from post 07 is now a test; a change that breaks this behaviour turns CI red.

Event Contracts: A Schema Test

The message contracts are the records in Bookstore.Contracts, and the rule from post 06 is clear: adding a field is fine, removing or renaming one means a new message type. To avoid leaving the rule to people's attention, we generate each message's JSON schema with System.Text.Json's JsonSchemaExporter and compare it with the approved schema in the repository. This is Bookstore's counterpart of the contract tests in the testing post of the Working with Message Queues series:

Type type = typeof(Exchanges).Assembly.GetType(typeName)!;
string actual = JsonSchemaExporter.GetJsonSchemaAsNode(Wire, type).ToJsonString(new JsonSerializerOptions { WriteIndented = true });
string approvedPath = Path.Combine(SourceFolder(), "schemas", $"{typeName}.json");
...
JsonNode approved = JsonNode.Parse(File.ReadAllText(approvedPath))!;
JsonNode current = JsonNode.Parse(actual)!;
string[] removed = [.. Properties(approved).Except(Properties(current))];
string[] added = [.. Properties(current).Except(Properties(approved))];

// Removing or renaming a field (or changing its type) breaks consumers that are already deployed
Assert.True(removed.Length == 0, $"{typeName}: breaking change, removed or changed: {string.Join(", ", removed)}");
// Adding a field is allowed by our versioning rule (post 06), but the approved schema has to be updated on purpose
Assert.True(added.Length == 0, $"{typeName}: new fields {string.Join(", ", added)}; update schemas/{typeName}.json");

The test finds the 17 records in the assembly by itself; when a new message has no schema yet, it writes a .received file and fails, and renaming the file is the approval. To try it, we renamed OrderPlaced's CustomerEmail field to Email and added a CouponCode:

failed Bookstore.Contracts.Tests.MessageSchemaTests.Wire_format_matches_the_approved_schema(typeName: "Bookstore.Contracts.Ordering.OrderPlaced") (0ms)
  Bookstore.Contracts.Ordering.OrderPlaced: breaking change, removed or changed: customerEmail:"string"
  total: 17
  failed: 1

The breaking change was caught by name and the other 16 messages were unaffected. Because the schema files show up as a diff in the pull request, approving a new field becomes a code review decision too.

Pact: The Contract Between the BFF and Catalog

On the HTTP side we have been catching breaking changes in Catalog's OpenAPI document with oasdiff since post 03. But oasdiff looks at Catalog's own document; it does not know which parts of it the BFF actually uses. Pact works the other way round: the consumer writes down what it needs as a test, and the provider verifies that expectation with its real code. On the BFF side the test runs the BFF's real CatalogApi against Pact's mock server:

pact.UponReceiving("a request for an existing book")
    .Given("the book 9780141439587 exists")
    .WithRequest(HttpMethod.Get, "/api/v1/books/9780141439587")
    .WillRespond()
    .WithStatus(HttpStatusCode.OK)
    // Only the fields the BFF reads, with types instead of exact values: Catalog may add fields freely
    .WithJsonBody(new
    {
        isbn = "9780141439587",
        title = Match.Type("Emma"),
        author = Match.Type("Jane Austen"),
        category = Match.Type("Classics"),
        price = Match.Decimal(249.90m),
        currency = Match.Regex("TRY", "^[A-Z]{3}$"),
        publishedYear = Match.Integer(1815),
        description = Match.Type("A young woman in a small English village keeps arranging other people's marriages.")
    });

await pact.VerifyAsync(async context =>
{
    CatalogApi catalog = new(new HttpClient { BaseAddress = context.MockServerUri });

    CatalogBook? book = await catalog.GetBookAsync("9780141439587", TestContext.Current.CancellationToken);

    Assert.Equal("Emma", book?.Title);
    Assert.Equal(249.90m, book?.Price);
});

The second interaction expects a 404 for an unknown book. Both tests passed in 0.5 seconds and produced the file tests/pacts/Bookstore.Bff-Bookstore.Catalog.json. On the Catalog side the real Catalog runs with WebApplicationFactory, but on a real port; .NET 10's UseKestrel makes that possible, and since Pact's verifier is a separate native process it cannot use the in-memory TestServer. SQL Server and Redis come from Testcontainers; instead of RabbitMQ there is Wolverine's DisableAllExternalWolverineTransports(), because events are not the subject of this contract. Before each interaction Pact sends the Given text to /provider-states, and the test sets up that state:

Verifying a pact between Bookstore.Bff and Bookstore.Catalog
  a request for a book that does not exist (0s loading, 402ms verification)
     Given the book 9780000000000 does not exist
    returns a response which
      has status code 404 (OK)
      has a matching body (OK)
  a request for an existing book (0s loading, 309ms verification)
     Given the book 9780141439587 exists
    returns a response which
      has status code 200 (OK)
      includes headers
        "Content-Type" with value "application/json" (OK)
      has a matching body (OK)

The second Guess First question: we added [JsonPropertyName("authorName")] to Catalog's Author property. Catalog's own tests don't notice; the provider verification did:

1) Verifying a pact between Bookstore.Bff and Bookstore.Catalog Given the book 9780141439587 exists - a request for an existing book
    1.1) has a matching body
           $ -> Actual map is missing the following keys: author
    {
    -  "author": "Jane Austen",
    +  "authorName": "Jane Austen",
      "category": "Classics",

The version field that Catalog returns but the BFF doesn't read is not an error; that is the tolerant reader, a client that reads only the fields it needs and ignores the rest, expressed in the contract. The right place matters too: this test has to run in Catalog's pipeline, because the change that breaks things is Catalog's. We set that up in GitHub Actions in post 14. Pact's native library sends anonymous usage statistics by default; we turned that off with PACT_DO_NOT_TRACK=true in both test projects.

End to End with Aspire: A Single Checkout

The top layer has a single test, and it builds the whole system from the AppHost: SQL Server, RabbitMQ, Redis, Keycloak, Mailpit and every service. The test does what Angular does: it signs in through the BFF, connects to the hub and places an order. It learns that the saga ended from the push; there is no fixed wait and no loop polling the status:

// The same hub connection Angular opens, with the same session cookie
TaskCompletionSource<OrderStatusChanged> confirmed = new(TaskCreationOptions.RunContinuationsAsynchronously);
Guid orderId = Guid.Empty;
await using HubConnection hub = new HubConnectionBuilder()
    .WithUrl(new Uri(bff, "/hubs/orders"), options => options.Cookies = cookies)
    .Build();
hub.On<OrderStatusChanged>("OrderStatusChanged", change =>
{
    if (change.OrderId == orderId && change.Status is "Confirmed" or "Cancelled")
    {
        confirmed.TrySetResult(change);
    }
});
await hub.StartAsync(Ct);
...
// The end of the saga is a push, so the test waits for the push: no polling loop, no fixed delay
OrderStatusChanged result = await confirmed.Task.WaitAsync(TimeSpan.FromSeconds(60), Ct);
Assert.Equal("Confirmed", result.Status);

Getting this test green took longer than writing it. Four obstacles came up, and anyone testing microservices end to end can run into all four:

• A clash with the local environment. SQL Server, RabbitMQ and Keycloak in the AppHost are persistent and have named volumes. When the test reused them, it ran with the previous run's data. In the test we switched those containers to a session lifetime, removed their volumes and renamed them with msapp-e2e-; every run starts with empty databases and the containers are deleted when the test ends.
• The HTTPS profile. The testing builder picked the AppHost's first launch profile (https), and Catalog, Payments and Notifications shut down at startup with Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found. CI agents usually have no trusted developer certificate either; with DOTNET_LAUNCH_PROFILE=http we chose the profile we use locally.
• Random ports. By default the testing builder gives every endpoint a random port; Keycloak came up on localhost:63682. The realm from post 10, however, only allows redirects to http://localhost:5100/signin-oidc. With DcpPublisher:RandomizePorts=false the BFF and Keycloak kept their fixed ports.
• Secure cookies. Browsers treat http://localhost as a secure context and send cookies marked Secure to it; HttpClient does not. ASP.NET Core's OIDC correlation cookies and Keycloak's session cookies are Secure, so the sign-in ended with 401 on /bff/user. The test uses a small DelegatingHandler that follows redirects itself and clears that flag on every hop.

passed Bookstore.EndToEndTests.CheckoutEndToEndTests.A_placed_order_is_confirmed_and_the_browser_is_told (1m 20s 009ms)
    system healthy after 61.2 s
    order 01a1db0b-b481-7a0b-83c2-8b6348b46e9e Confirmed 4.35 s after POST

The system took about a minute to come up and the order four to five seconds to be confirmed; most of the latter is Wolverine compiling handler code for the first messages (post 07). The value of this test is showing that everything works together, not showing where a bug is. That is why we test a single happy path. Declined payments, missing stock and timeouts are in the saga's unit tests and the service test.

In Production

CI's duration and cost grow with the test layers. All 56 tests of the solution took 82.7 seconds on this machine. Because the test projects run in parallel, the end-to-end test alone almost determines the total; the service test and the Pact provider went up to 26–29 seconds while running at the same time. Instead of running every layer on every pull request, one option is running the unit, service and contract tests on every change, and the end-to-end test after merging to main or at night.

The CI agent needs Docker. Testcontainers and the Aspire tests start containers; GitHub's Linux runners have Docker, while running Linux containers on Windows and macOS runners is either not possible or needs extra setup. Caching the images (or pulling them from your own registry) cuts minutes from the first run.

Use a Pact Broker for the pact files. We keep the pact file in the repository because both sides live in the same repository. Teams in separate repositories publish pacts to a broker and the provider pipeline fetches them from there; the broker also records which consumer version was verified against which provider version, and deployment decisions can be based on that.

What makes end-to-end tests flaky: fixed waits, shared data and random ports. We removed all three: waiting is a signal, the data is an empty database on every run, the ports are fixed. Still, this test will break more often than the others for infrastructure reasons such as a container starting slowly; when it does, look at the resource logs (Aspire testing writes them to the test output under the category Bookstore.AppHost.Resources.<name>).

Common Pitfalls

Testing everything end to end. Trying declined payments, missing stock and the timeout with the whole system means a minute and a flaky test for each; the same behaviour takes milliseconds in the saga's unit tests.

Waiting for an asynchronous result with Task.Delay. It either waits needlessly long or breaks on a slow day. If what you are waiting for is a message or a push, listen for it.

Forgetting the queues the service does not own in the test. If an ExternallyOwned queue does not exist in the test environment, the message is lost without an error and the test wrongly confirms "the message was sent".

Not putting the consumer-driven contract in the provider's pipeline. If the Pact verification runs only in the BFF's pipeline, Catalog's breaking change is noticed after Catalog has been deployed.

Writing the whole response into the pact. Putting fields the consumer doesn't read into the contract turns the provider's harmless changes red too.

A Small Experiment

★☆☆ In the schema test, change OrderPlacedLine's Quantity field from int to long. What does the test say? Is there really a change in the JSON, and would it break a consumer?

★★☆ Add Catalog's BookPriceChanged event to Ordering's service test: have the test, playing Catalog, publish a message to the bookstore.catalog exchange, and check that the copy's price changes and does not change when the same message arrives a second time.

★★★ Add a second scenario to the end-to-end test: stop Payments inside the test (app.ResourceCommands), place an order, see Pending in a push, start Payments and wait for the confirmation. How long does this test take, and which part of it is already covered by unit tests?

The tests are in place; now the system has to go somewhere. In the next post we build the images with the .NET SDK, embed Angular in the BFF's image, compare Aspire's Kubernetes output with hand-written manifests, and set up a GitHub Actions pipeline per service that runs only when that service changes.