Testing and Observing Resilience: WireMock.Net, Polly Chaos Strategies, FakeTimeProvider and Metrics

Version note: The code in this post was built and run with .NET 10, Polly 8.9, Microsoft.Extensions.Http.Resilience 10.11, WireMock.Net 2.31, xUnit v3 4.0, Microsoft.Extensions.TimeProvider.Testing 10.11 and OpenTelemetry .NET 1.21.

Guess First

When the carrier client gets a 429 it honors Retry-After: 2 and retries twice. How long does an integration test of that behavior take, at minimum?

Second question: you want to test how checkout behaves while the payment provider returns 500 for 20% of requests. Do you need Polly's chaos strategies or a fake for that?

Third question: in production the carrier's circuit breaker opened and the fallback kicked in. Users see no error at all. Who notices, and how?

The first answer is not 4 seconds but 264 milliseconds. The second is "either works, but they test different things". The third is the second half of this post. In the previous two posts we watched resilience behavior by hand, starting the fakes in different modes. Now we turn the same scenarios into automated tests and make them visible in production.

Test Setup: WebApplicationFactory and In-Process WireMock

The tests start the real Bookstore.Api with WebApplicationFactory and point the external service addresses at WireMock.Net servers started inside the test. The details of integration test infrastructure are in the Integration and E2E Testing series; what matters here is that nothing in the HTTP pipeline, the resilience handlers or the adapters is mocked. Only the external service is fake:

// The real API, wired to in-process WireMock fakes instead of the external services.
public sealed class BookstoreApiFactory(
    WireMockServer payments,
    WireMockServer carrier,
    TimeProvider? timeProvider = null,
    IDictionary<string, string?>? settings = null) : WebApplicationFactory<CheckoutService> // any type from Bookstore.Api
{
    public FakeEmailSender Emails { get; } = new();

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.UseEnvironment("Development");
        builder.UseSetting("Payments:Provider:BaseUrl", payments.Url + "/");
        builder.UseSetting("Shipping:LegacyCarrier:BaseUrl", carrier.Url + "/");
        foreach (var (key, value) in settings ?? new Dictionary<string, string?>())
        {
            builder.UseSetting(key, value);
        }

        builder.ConfigureTestServices(services =>
        {
            services.RemoveAll<IEmailSender>();
            services.AddSingleton<IEmailSender>(Emails);
            if (timeProvider is not null)
            {
                services.RemoveAll<TimeProvider>();
                services.AddSingleton(timeProvider);
            }
        });
    }
}

Two small details. We couldn't write WebApplicationFactory<Program>, because the test project references both the API and Bookstore.FakeProviders, and both have a Program generated from top-level statements; the compiler asks which one we mean with CS0433. All the factory needs is a type from the entry assembly, and CheckoutService does the job. Second, we replaced e-mail sending with a fake IEmailSender; otherwise every checkout waits for the SMTP retries from the previous post.

The fakes are the FakePaymentProvider and FakeLegacyCarrier classes we've used since the first post; called without a port, Start() picks a free one. WireMock's LogEntries list keeps every request the server receives, and that's the most honest way to verify retry counts: you count what the other side saw, not what the client did.

private int QuoteRequests => _carrier.LogEntries.Count(e => e.RequestMessage?.Path == "/api/v2/quotes");

private int ChargeRequests => _payments.LogEntries.Count(e => e.RequestMessage?.Path == "/v1/charges");

Verifying Retries and the Circuit

Three tests pin down three behaviors we saw by hand in earlier posts:

[Fact]
public async Task Carrier_errors_are_retried_then_checkout_uses_the_flat_tariff()
{
    FakeLegacyCarrier.Configure(_carrier, FakeMode.Error);
    await using var api = new BookstoreApiFactory(_payments, _carrier);

    using var response = await CheckoutAsync(api.CreateClient());

    var result = await response.Content.ReadFromJsonAsync<CheckoutResponse>(TestContext.Current.CancellationToken);
    Assert.Equal(HttpStatusCode.OK, response.StatusCode);
    Assert.True(result!.ShippingIsFlatRate);
    Assert.Equal(3, QuoteRequests); // 1 attempt + 2 retries
}

[Fact]
public async Task Payment_post_is_never_retried()
{
    FakePaymentProvider.Configure(_payments, FakeMode.Error);
    await using var api = new BookstoreApiFactory(_payments, _carrier);

    using var response = await CheckoutAsync(api.CreateClient());

    Assert.Equal(HttpStatusCode.ServiceUnavailable, response.StatusCode);
    Assert.Equal(1, ChargeRequests);
}

[Fact]
public async Task Circuit_opens_and_stops_calling_the_carrier()
{
    FakeLegacyCarrier.Configure(_carrier, FakeMode.Error);
    await using var api = new BookstoreApiFactory(_payments, _carrier);
    var client = api.CreateClient();

    for (var i = 0; i < 4; i++)
    {
        using var _ = await CheckoutAsync(client);
    }

    var before = QuoteRequests;
    using var next = await CheckoutAsync(client);
    var health = await client.GetStringAsync("/health", TestContext.Current.CancellationToken);

    Assert.Equal(before, QuoteRequests); // the open circuit short-circuits the call
    Assert.Equal("Degraded", health);
}

Payment_post_is_never_retried may be the most valuable test in this series. One day someone adds a retry to the payment client "just in case", or deletes the DisableForUnsafeHttpMethods line; this test turns red in that pull request. Resilience settings don't look like code, so they slip through code review easily; a test pins them down.

In the circuit test every factory is a new host and therefore a new breaker. Breaker state doesn't carry over between tests, but within one test a breaker you open affects every later request. That's why the tests use independent factories.

Testing Backoff Without Waiting: FakeTimeProvider

If the carrier returns 429 with Retry-After: 2 twice and answers on the third request, checkout takes at least 4 seconds in real time. Polly reads time through TimeProvider, and Microsoft.Extensions.Resilience uses the TimeProvider from the DI container when it builds pipelines. The test puts a FakeTimeProvider into the container and moves the clock itself:

[Fact]
public async Task Retry_After_backoff_runs_on_fake_time()
{
    // 429 + Retry-After: 2 twice, then the normal quote.
    _carrier.Given(Request.Create().WithPath("/api/v2/quotes").UsingGet())
        .InScenario("throttle").WillSetStateTo("once")
        .AtPriority(-1) // before the fake's own quote mapping
        .RespondWith(Response.Create().WithStatusCode(429).WithHeader("Retry-After", "2"));
    _carrier.Given(Request.Create().WithPath("/api/v2/quotes").UsingGet())
        .InScenario("throttle").WhenStateIs("once").WillSetStateTo("twice")
        .AtPriority(-1) // before the fake's own quote mapping
        .RespondWith(Response.Create().WithStatusCode(429).WithHeader("Retry-After", "2"));

    var time = new FakeTimeProvider(DateTimeOffset.UtcNow);
    await using var api = new BookstoreApiFactory(_payments, _carrier, time);
    var client = api.CreateClient();
    var started = TimeProvider.System.GetTimestamp();

    var checkout = CheckoutAsync(client);
    while (!checkout.IsCompleted)
    {
        await Task.Delay(TimeSpan.FromMilliseconds(20), TestContext.Current.CancellationToken);
        time.Advance(TimeSpan.FromMilliseconds(500));
    }

    using var response = await checkout;
    var result = await response.Content.ReadFromJsonAsync<CheckoutResponse>(TestContext.Current.CancellationToken);
    var realTime = TimeProvider.System.GetElapsedTime(started);
    TestContext.Current.TestOutputHelper?.WriteLine($"quote requests={QuoteRequests}, flat rate={result?.ShippingIsFlatRate}, real time={realTime.TotalMilliseconds:0} ms");

    Assert.False(result!.ShippingIsFlatRate);
    Assert.Equal(3, QuoteRequests);
    Assert.True(realTime < TimeSpan.FromSeconds(2), $"took {realTime}");
}

passed Bookstore.Api.Tests.CheckoutResilienceTests.Retry_After_backoff_runs_on_fake_time (393ms)
    quote requests=3, flat rate=False, real time=264 ms

Two 2-second waits finished in 264 milliseconds. There's a reason the loop moves the clock in small steps: the same TimeProvider also drives the 2-second attempt timeout and the 6-second total timeout. Jump the clock 10 seconds in one go and the request times out before its response arrives, and the test verifies the wrong thing. General techniques for testing time are in the hard cases post of the Unit Test series.

I fell into a WireMock trap here: my first version added the rules with AtPriority(1), and the test got a 200 on the first request. In WireMock.Net a smaller number means higher priority, and the fake's own rule at the default priority matched first. Together with the "request builders are mutable" trap from the first post, these are the two details that cost the most time when working with fakes.

While the Payment Provider Fails 20% of the Time

The question from the plan: what does checkout do while the provider returns 500 for one request in five? We set up the fake to fail at random with a fixed seed and sent 100 checkouts:

var random = new Random(42);
var failures = 0;
_payments.Reset();
_payments.Given(Request.Create().WithPath("/v1/charges").UsingPost())
    .RespondWith(Response.Create().WithCallback(_ =>
    {
        lock (random)
        {
            if (random.NextDouble() < 0.2)
            {
                failures++;
                return new ResponseMessage { StatusCode = 500 };
            }
        }

        var ok = new ResponseMessage { StatusCode = 201 };
        // ... successful charge body
        return ok;
    }));

passed Bookstore.Api.Tests.CheckoutResilienceTests.Checkout_with_twenty_percent_payment_failures (233ms)
    responses: 200 x 69, 503 x 31; provider 500s: 31; charge requests: 100

The test checks three things: no response is a 500 (failures pass through our own error model and become 503), the number of 503s equals the provider's 500s, and the provider saw exactly 100 requests, so the POST was not retried even under failures. With seed 42, 100 attempts produced 31 failures; the rate is around 20% but not exact. A fixed seed makes the test repeatable, and because we verify behavior rather than the number, changing the seed doesn't break it.

Polly's Chaos Strategies

Testing with a fake needs an environment where you control the external service. In staging, or in a real integration environment, you don't have that control; there you have to create the failure on the client side, inside the pipeline. With Polly 8.3 the former Simmy library joined Polly and its chaos strategies became part of Polly.Core: AddChaosFault injects an exception, AddChaosOutcome a fake result, AddChaosLatency a delay and AddChaosBehavior any behavior you like. We added two of them at the innermost end of the carrier pipeline:

// Chaos strategies go last (innermost). Never active in Production, whatever the config says.
var chaos = context.ServiceProvider.GetRequiredService<IOptionsMonitor<ChaosOptions>>();
var environment = context.ServiceProvider.GetRequiredService<IHostEnvironment>();
ValueTask<bool> ChaosEnabled() =>
    ValueTask.FromResult(!environment.IsProduction() && chaos.Get(LegacyCarrierClient.ClientName).Enabled);
pipeline
    .AddChaosLatency(new ChaosLatencyStrategyOptions
    {
        EnabledGenerator = _ => ChaosEnabled(),
        InjectionRateGenerator = _ => ValueTask.FromResult(chaos.Get(LegacyCarrierClient.ClientName).LatencyRate),
        LatencyGenerator = _ => ValueTask.FromResult(chaos.Get(LegacyCarrierClient.ClientName).Latency)
    })
    .AddChaosOutcome(new ChaosOutcomeStrategyOptions<HttpResponseMessage>
    {
        EnabledGenerator = _ => ChaosEnabled(),
        InjectionRateGenerator = _ => ValueTask.FromResult(chaos.Get(LegacyCarrierClient.ClientName).FaultRate),
        OutcomeGenerator = _ => ValueTask.FromResult<Outcome<HttpResponseMessage>?>(
            Outcome.FromResult(new HttpResponseMessage(HttpStatusCode.ServiceUnavailable)))
    });

There are three decisions here. The chaos strategies are innermost, because we want the retry, breaker and timeout outside them to react to an injected failure as if it were real; Polly's documentation recommends this order too. Second, chaos strategies are active by default as soon as they are added, and the default InjectionRate is 0.001, so code where you "added it but didn't set a rate" produces one failure in a thousand. We control it explicitly with EnabledGenerator. Third, the settings are read on every call through IOptionsMonitor, so in staging you can switch chaos on and off with a configuration change, without a restart.

I started the API with --Chaos:LegacyCarrier:Enabled=true --Chaos:LegacyCarrier:FaultRate=0.2 and sent 50 checkouts:

     50 "shippingIsFlatRate":false
     14 EventName: 'Chaos.OnOutcome'
     14 EventName: 'OnRetry'

Fourteen fake 503s were injected, each triggered a retry, and no checkout fell back to the flat tariff: the chance that all three attempts of a request fail at 20% is about 8 in a thousand. That is the measured form of the sentence "our retry settings absorb a 20% failure rate without users noticing". With the same settings and --environment Production the Chaos.OnOutcome count stayed at 0.

We pinned down one more property of chaos outcomes with a test: the fake result never touches the network.

[Fact]
public async Task Chaos_faults_on_the_carrier_never_reach_the_network()
{
    await using var api = new BookstoreApiFactory(_payments, _carrier, settings: new Dictionary<string, string?>
    {
        ["Chaos:LegacyCarrier:Enabled"] = "true",
        ["Chaos:LegacyCarrier:FaultRate"] = "1.0"
    });

    using var response = await CheckoutAsync(api.CreateClient());
    var result = await response.Content.ReadFromJsonAsync<CheckoutResponse>(TestContext.Current.CancellationToken);

    Assert.True(result!.ShippingIsFlatRate);
    Assert.Equal(0, QuoteRequests);
}

That matters when you use chaos against a real provider: you measure your own side's behavior without adding load on theirs. The flip side holds too: chaos doesn't test the network layer, DNS or TLS; for those you need a fake or a real outage scenario.

Observing: Polly's Logs and Metrics

Polly writes every strategy event both to the log and to a meter named Polly. We added OpenTelemetry to the API and connected it to the console exporter in development:

// Polly and HttpClient publish metrics and spans; here they go to the console, in production to an OTLP collector.
if (builder.Configuration.GetValue<bool>("Telemetry:ConsoleExporter"))
{
    builder.Services.AddOpenTelemetry()
        .WithMetrics(metrics => metrics
            .AddMeter("Polly", "System.Net.Http")
            .AddConsoleExporter((_, reader) => reader.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 10_000))
        .WithTracing(tracing => tracing
            .AddSource("Microsoft.AspNetCore", "System.Net.Http")
            .AddConsoleExporter());
}

We chose the console over OTLP only for the example. These are the metrics the exporter wrote during the chaos run:

Metric Name: http.client.active_requests
Metric Name: http.client.open_connections
Metric Name: http.client.request.duration
Metric Name: http.client.request.time_in_queue
Metric Name: resilience.polly.pipeline.duration
Metric Name: resilience.polly.strategy.attempt.duration
Metric Name: resilience.polly.strategy.events

Metric Name: resilience.polly.strategy.events, Description: Tracks the number of resilience events that occurred in resilience strategies., Metric Type: LongSum
Instrumentation scope (Meter):
	Name: Polly
	Version: 8.9.1
(2026-11-14T09:24:16.8300100Z, 2026-11-14T09:24:21.9080350Z] event.name: Chaos.OnOutcome event.severity: Information pipeline.instance:  pipeline.name: legacy-carrier-carrier strategy.name: Chaos.Outcome
Value: 14
(2026-11-14T09:24:16.8300100Z, 2026-11-14T09:24:21.9080350Z] error.type: 503 event.name: OnRetry event.severity: Warning pipeline.instance:  pipeline.name: legacy-carrier-carrier strategy.name: Retry
Value: 14

resilience.polly.strategy.events is the most important metric in this post. Its event.name tag takes values such as OnRetry, OnCircuitOpened, OnTimeout, OnFallback and OnHedging, and pipeline.name tells you which client it was. The System.Net.Http meter, available since .NET 8, gives the duration of every outbound call with server.address and http.response.status_code tags. On the tracing side every attempt gets its own client span; the three attempts of one carrier request show up as three separate spans in the trace, and the query part of the URL is masked in the span as well:

Activity.DisplayName:        GET
Activity.Kind:               Client
Activity.Tags:
    http.request.method: GET
    server.address: localhost
    server.port: 9092
    url.full: http://localhost:9092/api/v2/quotes?*
    http.response.status_code: 200

Metrics can be tested too. MetricCollector from the Microsoft.Extensions.Diagnostics.Testing package listens to the meter you name:

// Polly publishes resilience events on the "Polly" meter.
using var events = new MetricCollector<int>(null, "Polly", "resilience.polly.strategy.events");
await using var api = new BookstoreApiFactory(_payments, _carrier);

using var _ = await api.CreateClient().PostAsJsonAsync("/api/orders/1001/checkout",
    new { cardToken = "tok_visa" }, TestContext.Current.CancellationToken);

var snapshot = events.GetMeasurementSnapshot();
Assert.Equal(2, snapshot.Count(m => m.Tags["event.name"] is "OnRetry"));
Assert.Equal(1, snapshot.Count(m => m.Tags["event.name"] is "OnFallback"));

passed Bookstore.Api.Tests.ResilienceTelemetryTests.Retries_and_fallbacks_show_up_in_polly_metrics (624ms)
    OnRetry: event.severity=Warning, pipeline.name=legacy-carrier-carrier, strategy.name=Retry
    OnRetry: event.severity=Warning, pipeline.name=legacy-carrier-carrier, strategy.name=Retry
    OnFallback: event.severity=Warning, pipeline.name=shipping-quote, strategy.name=Fallback

The meter is shared across the process, so if two test classes run in parallel the counts get mixed up. We added an xunit.runner.json with parallelizeTestCollections: false to the test project. The result for all tests:

dotnet test --project tests/Bookstore.Api.Tests --output detailed

  total: 7
  failed: 0
  succeeded: 7
  duration: 5s 142ms

Dashboards and Alerts for an External Service

The silent failure from the opening question is the price of a fallback that works well: users see nothing, and the flat tariff quietly loses money. A few metrics and two alerts per external service are enough to catch it. On the dashboard, per service and split by pipeline.name and server.address, have these panels: p50 and p99 from http.client.request.duration, request count by status code, OnRetry events as a share of all requests, and the OnFallback count. If the provider's SLA promises "99.9% monthly availability", this panel is the only place you measure that from your side, and the only data you have in a contract discussion.

For alerts I suggest two rules. First, alert when the OnCircuitOpened event count is above 0: by definition an open breaker means the provider is having serious trouble. Second, alert when the OnFallback rate stays above a threshold for a few minutes; that catches the cases where checkout falls back to the flat tariff even without the breaker opening. Retry rate belongs on the dashboard rather than in an alert, because hiding short failures from users is exactly the retries' job. How to ship metrics to a backend and write alert rules is covered in the Observability series.

On the logging side, keeping Polly's Polly category at Warning is enough: retry, timeout and breaker events are written at Warning or Error, while the Execution attempt line for every successful attempt is Information and only adds volume in production.

Production Notes

Don't switch chaos on in production without control. Our pipeline turns chaos off entirely in production with the IsProduction() check. If you do run chaos experiments in production, make them planned, short, low-rate and, as Polly's documentation suggests, limited to specific users or tenants; and tie EnabledGenerator to a time limit so the experiment can't be forgotten.

Fakes in tests are only as good as the real provider they imitate. If the provider has a sandbox, keep the fake's responses in sync with real samples taken from it; differences like the missing Content-Type from the first post show up in production, not in tests.

Watch the cardinality of metric tags. pipeline.name and server.address are fixed sets; put order numbers or customer IDs into tags and the metric backend's cost grows quickly. That kind of data belongs in logs and traces.

Common Pitfalls

Testing backoff in real time. As retry delays grow, so does the test, and at some point someone disables it for being "slow". With FakeTimeProvider tests finish in milliseconds.

Counting retries on the client side. The call count of a mocked handler may not tell you how many requests the pipeline really sent; count the requests the fake server saw.

Not logging retries. The answer to "why did checkout take 4 seconds" is usually two retries and a Retry-After; with Polly's logs off, you won't find it.

Adding a chaos strategy without a rate. The default rate is 0.001 and the strategy is active by default; you end up with a forgotten line of code that produces one fake failure in a thousand in production.

A Small Experiment

★☆☆ Change the seed in Checkout_with_twenty_percent_payment_failures (say, to 7) and look at the number of 503s in the output. Why does the test still pass?

★★☆ Turn on chaos latency in the carrier pipeline with LatencyRate = 0.3 and Latency = 3 s. How does the 2-second attempt timeout handle that delay? Out of 50 checkouts, how many fall back to the flat tariff, and what is the OnTimeout count?

★★★ Write a MetricCollector test that listens for the OnCircuitOpened event: with the carrier fake returning 500, assert at which checkout the breaker opens. Then get the same result with ChaosOutcome, without changing the fake at all.

With this post, Bookstore's outbound calls are time-bounded, retryable, able to degrade during an outage, and observable. One question remains open: when the payment POST times out, was the money taken or not? The next post deals with that "unknown outcome": retrying a POST safely with an Idempotency-Key, querying the payment status from the provider, and reconciliation.