Polly v8 Strategies in Depth: Retry, Timeout, Circuit Breaker, Rate Limiter and Fallback

Version note: The code in this post was built and run with .NET 10, Polly 8.9, Microsoft.Extensions.Http.Resilience 10.11, MailKit 5.0 and WireMock.Net 2.31.

Guess First

You added a 1-second timeout to a pipeline, and inside it is an old SDK call that takes no CancellationToken; the call takes 3 seconds. Does the pipeline throw TimeoutRejectedException at the first second?

Second question: you gave retry MaxDelay = TimeSpan.FromSeconds(3). The carrier is in maintenance and sends Retry-After: 600 with its 503. Does retry wait at most 3 seconds?

Third question: in the same pipeline you put the timeout outside retry. If the first attempt hangs, does retry get a chance at a second attempt?

All three answers are "no", and all three come from real runs. In the previous post we used the standard pipeline as a ready-made package. In this one we build Polly v8's strategies one by one: their order, their settings, protecting a call that isn't HTTP, and falling back to a flat tariff without stopping checkout when the carrier price is unavailable.

This post's experiments also live in Bookstore.ResilienceLab; for output from the Bookstore API we start the fakes and the API as in the first post:

dotnet run --project samples/Bookstore.ResilienceLab -- order
dotnet run --project samples/Bookstore.ResilienceLab -- context
dotnet run --project samples/Bookstore.ResilienceLab -- optimistic-timeout
dotnet run --project samples/Bookstore.ResilienceLab -- retry-after
dotnet run --project samples/Bookstore.ResilienceLab -- circuit-breaker
dotnet run --project samples/Bookstore.ResilienceLab -- bulkhead

ResiliencePipelineBuilder: Order Is Everything

In Polly v8 everything is a ResiliencePipeline. The first strategy you add to the builder is outermost, the last one innermost. I tried the same two strategies in two orders. The function being called hangs for 5 seconds the first time and returns immediately the second:

var retry = new RetryStrategyOptions { MaxRetryAttempts = 2, Delay = TimeSpan.FromMilliseconds(200) };

await RunAsync("timeout outside retry", new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(2))
    .AddRetry(retry)
    .Build());

await RunAsync("timeout inside retry ", new ResiliencePipelineBuilder()
    .AddRetry(retry)
    .AddTimeout(TimeSpan.FromSeconds(2))
    .Build());

await RunAsync("inside, HttpRequestException only", new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 2,
        Delay = TimeSpan.FromMilliseconds(200),
        ShouldHandle = new PredicateBuilder().Handle<HttpRequestException>()
    })
    .AddTimeout(TimeSpan.FromSeconds(2))
    .Build());

[ 2.03s] timeout outside retry: TimeoutRejectedException in 2.0 s
[ 4.25s] timeout inside retry : quote after 2 call(s) in 2.2 s
[ 6.25s] inside, HttpRequestException only: TimeoutRejectedException in 2.0 s

With the timeout outside, there is a single budget covering all retries; once the first attempt uses it up, retry never gets its turn. Inside, each attempt has its own budget and the second attempt succeeds. The third line is the most common mistake: the timeout is inside, but retry's ShouldHandle only knows HttpRequestException. A TimeoutRejectedException isn't an HttpRequestException, so retry ignores it. The default ShouldHandle of RetryStrategyOptions handles every exception except OperationCanceledException, so the first two pipelines don't have this problem; it starts the moment you narrow ShouldHandle.

The right order is the one the standard pipeline uses: total timeout outermost, then retry, then the circuit breaker, attempt timeout innermost. If you deviate from it in your own pipeline, write down why in a comment.

The CancellationToken going into the pipeline and your own data travel in a ResilienceContext. I took a context from the pool, added a property and read it in the retry callback:

private static readonly ResiliencePropertyKey<int> OrderIdKey = new("bookstore.order-id");

var context = ResilienceContextPool.Shared.Get("carrier-quote", cts.Token);
context.Properties.Set(OrderIdKey, 1001);
try
{
    await pipeline.ExecuteAsync(ctx => /* ... */, context);
}
finally
{
    ResilienceContextPool.Shared.Return(context);
}

[ 0.00s] attempt 1: operation key=carrier-quote, same token=True
[ 0.01s] OnRetry for order 1001, attempt 1
[ 0.01s] attempt 2: operation key=carrier-quote, same token=True

The OperationKey shows up in logs and metrics; if you use one pipeline for different jobs, that's how you tell which one retried. Don't forget to return the context to the pool; Polly reuses these objects to cut allocations.

Beyond HTTP: AddResiliencePipeline for SMTP

The order confirmation e-mail goes out over SMTP, so there is no HttpClient and no IHttpClientFactory. The Polly.Extensions package lets you register pipelines in the DI container by name:

// A named, non-generic pipeline in the DI-backed registry (Polly.Extensions).
services.AddResiliencePipeline(SmtpEmailSender.PipelineName, builder => builder
    .AddTimeout(TimeSpan.FromSeconds(15))
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 2,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        Delay = TimeSpan.FromSeconds(1),
        // Retry only what is transient: network errors, 4xx SMTP replies, our own attempt timeout.
        ShouldHandle = new PredicateBuilder()
            .Handle<SocketException>()
            .Handle<IOException>()
            .Handle<ServiceNotConnectedException>()
            .Handle<SmtpProtocolException>()
            .Handle<SmtpCommandException>(ex => (int)ex.StatusCode is >= 400 and < 500)
            .Handle<Polly.Timeout.TimeoutRejectedException>()
    })
    .AddTimeout(TimeSpan.FromSeconds(4)));

In SMTP, 4xx replies mean "not now, try later" and 5xx replies mean "this e-mail will never go"; ShouldHandle makes that distinction. We get the pipeline from ResiliencePipelineProvider<string> and wrap the MailKit call:

var pipeline = pipelines.GetPipeline(PipelineName);
await pipeline.ExecuteAsync(async token =>
{
    using var smtp = new SmtpClient();
    await smtp.ConnectAsync(options.Value.Host, options.Value.Port, SecureSocketOptions.None, token);
    await smtp.SendAsync(message, token);
    await smtp.DisconnectAsync(quit: true, token);
}, cancellationToken);

Bookstore.FakeProviders in the repo now also starts a small fake SMTP server on port 2525. I ran checkout with no server at all (--smtp Error) and with a server that answers every connection with 421 (--smtp Throttle):

Resilience event occurred. EventName: 'OnRetry', Source: 'smtp/(null)/Retry', Operation Key: '', Result: 'Connection refused'
Resilience event occurred. EventName: 'OnRetry', Source: 'smtp/(null)/Retry', Operation Key: '', Result: 'Connection refused'
Order 1001 confirmation e-mail could not be sent

Resilience event occurred. EventName: 'OnRetry', Source: 'smtp/(null)/Retry', Operation Key: '', Result: '4.3.2 Service not available, try again later'
Resilience event occurred. EventName: 'OnRetry', Source: 'smtp/(null)/Retry', Operation Key: '', Result: '4.3.2 Service not available, try again later'
Order 1001 confirmation e-mail could not be sent

In both cases the order came back Paid; CheckoutService catches and logs the e-mail failure, because showing a paid order as an error just because of an e-mail would be wrong. But that's why the checkout response took 2.7 seconds. Sending e-mail inside the request isn't the right design; as the Background Jobs series explains, it belongs in an outbox and is sent in the background. We left it like this here to show how a pipeline is built for a non-HTTP call.

Retry: Retry-After and the Limit of MaxDelay

If you write your own retry, honoring the Retry-After header is done with DelayGenerator. If the generator returns null, Polly uses its own Delay and BackoffType calculation:

DelayGenerator = args =>
{
    // Retry-After: 2 (seconds) or an HTTP date; null means "use Delay/BackoffType".
    var retryAfter = args.Outcome.Result?.Headers.RetryAfter;
    var wait = retryAfter?.Delta ?? (retryAfter?.Date - DateTimeOffset.UtcNow);
    return ValueTask.FromResult(wait);
},

[ 0.00s] retry 1 after 429, waiting 2.00 s
[ 2.01s] retry 2 after 429, waiting 2.00 s
[ 4.01s] final status 200 after 3 attempts

Even though the base delay is 200 ms, both waits are the 2 seconds the provider asked for. HttpRetryStrategyOptions already does this through ShouldRetryAfterHeader; in generic pipelines you write it yourself.

There's a trap here. In the first version of the carrier client I set MaxDelay = TimeSpan.FromSeconds(3), thinking "whatever the provider says, I'll wait at most 3 seconds". During maintenance the carrier sends Retry-After: 600:

warn: Polly[0]
      Resilience event occurred. EventName: 'OnRetry', Source: 'legacy-carrier-carrier//Retry', Operation Key: '', Result: '503'
fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'legacy-carrier-carrier//Timeout', Operation Key: '', Result: ''
info: Polly[2]
      Resilience pipeline executed. Source: 'legacy-carrier-carrier/', Operation Key: '', Result: 'The operation didn't complete within the allowed timeout of '00:00:06'.', Execution Time: 6008.2294ms

Retry started a 600-second wait and the 6-second total timeout cut it off. Polly's documentation says so plainly: MaxDelay doesn't cap a delay returned by DelayGenerator. Http.Resilience's Retry-After support behaved the same way. Without the total timeout, checkout would have waited 10 minutes. The fix is not to retry a long Retry-After at all; that isn't a transient failure any more, it's maintenance:

.AddRetry(new HttpRetryStrategyOptions
{
    MaxRetryAttempts = 2,
    BackoffType = DelayBackoffType.Exponential,
    UseJitter = true,
    Delay = TimeSpan.FromMilliseconds(200),
    // MaxDelay does not cap Retry-After, so a long Retry-After (maintenance) is not retried at all.
    ShouldHandle = args => ValueTask.FromResult(
        HttpClientResiliencePredicates.IsTransient(args.Outcome) &&
        !(args.Outcome.Result?.Headers.RetryAfter?.Delta > TimeSpan.FromSeconds(3)))
})

HttpClientResiliencePredicates.IsTransient is the ready-made check the standard pipeline uses; we only added one condition. In the same maintenance scenario checkout now completes in 0.18 seconds with the flat tariff (we get to the fallback below).

The Timeout Is Optimistic

Polly's timeout is "optimistic": when time is up it cancels the token it gave the callback and expects the callback to honor it. It can't stop code that ignores the token:

await pipeline.ExecuteAsync(async token =>
{
    if (honorsToken)
    {
        await Task.Delay(TimeSpan.FromSeconds(3), token);
    }
    else
    {
        Thread.Sleep(TimeSpan.FromSeconds(3)); // a legacy SDK call without a CancellationToken
    }
});

[ 1.02s] honors token=True: TimeoutRejectedException in 1.0 s
[ 4.03s] honors token=False: completed in 3.0 s

The second line answers the opening question: there isn't even an exception. The callback finished normally after 3 seconds, so the pipeline counts it as a success. That's why every call you hand to a pipeline has to receive the token; HttpClient, MailKit and Entity Framework Core all support it. The only way to bound an SDK that takes no token is to run it in a separate process, or at least a separate Task, and stop waiting for it, which leaves work running in the background. The reasoning behind cooperative cancellation is in the CancellationToken post of the C# Async series.

Circuit Breaker: Settings, Manual Control and Health Checks

A circuit breaker stops sending calls for a while once the failure ratio within a window crosses a threshold. Polly has four states:

To watch the transitions I built a breaker with small values: at least 4 calls and 50% failures in 10 seconds. Instead of a fixed break duration I tied it to the failure count with BreakDurationGenerator:

var manual = new CircuitBreakerManualControl();
var state = new CircuitBreakerStateProvider();
var pipeline = new ResiliencePipelineBuilder()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,
        MinimumThroughput = 4,
        SamplingDuration = TimeSpan.FromSeconds(10),
        // Longer breaks when more calls failed: 4 failures -> 2 s, capped at 30 s.
        BreakDurationGenerator = args => ValueTask.FromResult(TimeSpan.FromSeconds(Math.Min(30, args.FailureCount * 0.5))),
        ManualControl = manual,
        StateProvider = state,
        ShouldHandle = new PredicateBuilder().Handle<HttpRequestException>(),
        OnOpened = args => { Log($"OnOpened   break={args.BreakDuration.TotalSeconds}s manual={args.IsManual}"); return default; },
        OnHalfOpened = _ => { Log("OnHalfOpened"); return default; },
        OnClosed = args => { Log($"OnClosed   manual={args.IsManual}"); return default; }
    })
    .Build();

[ 0.00s] call 1: HttpRequestException (state Closed)
[ 0.00s] call 2: HttpRequestException (state Closed)
[ 0.00s] call 3: HttpRequestException (state Closed)
[ 0.00s] OnOpened   break=2s manual=False
[ 0.00s] call 4: HttpRequestException (state Open)
[ 0.00s] call 5: BrokenCircuitException (state Open)
[ 2.21s] OnHalfOpened
[ 2.21s] OnClosed   manual=False
[ 2.21s] call 6: ok (state Closed)
[ 2.21s] call 7: ok (state Closed)
[ 2.21s] maintenance window starts: IsolateAsync()
[ 2.21s] OnOpened   break=0s manual=True
[ 2.21s] call 8: IsolatedCircuitException (state Isolated)
[ 2.21s] OnClosed   manual=True
[ 2.21s] call 9: ok (state Closed)

The first three failures didn't reach MinimumThroughput, so the breaker stayed closed; the fourth crossed the threshold. The fifth call never reached the provider and got a BrokenCircuitException. After 2 seconds the first call went through as the HalfOpen trial, and when it succeeded the breaker closed. IsolateAsync() holds the breaker open with no time limit, with IsolatedCircuitException in the meantime; CloseAsync() closes it again.

These two objects are why Bookstore's carrier client builds its own pipeline instead of the standard handler. A CarrierCircuit singleton holds the ManualControl and StateProvider, and AddResilienceHandler passes them to the breaker:

carrier.AddResilienceHandler("carrier", (pipeline, context) =>
{
    var circuit = context.ServiceProvider.GetRequiredService<CarrierCircuit>();
    pipeline
        .AddTimeout(TimeSpan.FromSeconds(6))
        .AddRetry(new HttpRetryStrategyOptions { /* the retry above */ })
        .AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            MinimumThroughput = 10,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(15),
            ManualControl = circuit.ManualControl,
            StateProvider = circuit.StateProvider
        })
        .AddTimeout(TimeSpan.FromSeconds(2));
});

We lowered MinimumThroughput from 100 to 10: carrier prices are requested a few dozen times a minute, and 100 requests never pile up in a 30-second window. The state provider feeds a health check, and the manual control two admin endpoints:

public sealed class CarrierCircuitHealthCheck(CarrierCircuit circuit) : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        var state = circuit.StateProvider.CircuitState;
        return Task.FromResult(state == CircuitState.Closed
            ? HealthCheckResult.Healthy("Carrier circuit is closed.")
            : HealthCheckResult.Degraded($"Carrier circuit is {state}; shipping uses the flat tariff."));
    }
}

app.MapPost("/admin/carrier-circuit/isolate", async (CarrierCircuit circuit) =>
{
    await circuit.ManualControl.IsolateAsync();
    return Results.NoContent();
});

With the carrier fake returning 500 constantly, I ran checkouts back to back and checked /health after each:

checkout 1: 200 [0.872034s] Healthy
checkout 2: 200 [0.587929s] Healthy
checkout 3: 200 [0.549593s] Degraded
checkout 4: 200 [0.016205s] Degraded
checkout 5: 200 [0.014696s] Degraded

Each checkout writes three attempts (one request, two retries) into the breaker's window; in my run the threshold was crossed during the third checkout and OnCircuitOpened appeared in the log. The following checkouts finished in 15 ms without touching the carrier. The health check returns Degraded, not Unhealthy, on purpose: carrier prices are unavailable, but sales go on, and we don't want the load balancer to take this instance out of rotation. When a maintenance window is known in advance, POST /admin/carrier-circuit/isolate opens the breaker and we switch to the flat tariff without living through the carrier's first 500s.

Bulkhead: The Concurrency Limiter

The real damage a slow provider does is the resources it holds while you wait. A concurrency limiter lets at most N calls in at once and rejects the rest immediately; it is the bulkhead idea from the System Design series. From the Polly.RateLimiting package:

var pipeline = new ResiliencePipelineBuilder()
    .AddConcurrencyLimiter(permitLimit: 2, queueLimit: 0)
    .Build();

call 1: ok
call 2: ok
call 3: RateLimiterRejectedException
call 4: RateLimiterRejectedException
call 5: RateLimiterRejectedException

Of five calls started together, two got through and three were rejected without waiting. The rate limiter at the outer edge of the standard pipeline is the same thing: 1000 concurrent requests by default, queue 0. If you want at most 50 concurrent requests to the payment provider, changing that number is enough. Translating the rejection into PaymentFailureKind.Transient in the adapter means the user sees a "try again shortly" message.

Fallback: A Flat Tariff When There's No Carrier Price

When the carrier price is unavailable there are two options: stop checkout with a 503, or carry on with a flat tariff and absorb the difference as a company. Bookstore chose the second. The fallback lives outside the HTTP pipeline, in a decorator around IShippingCarrier, because its subject is a ShippingQuote, not an HTTP response:

// Decorator: when the carrier fails, checkout continues with a flat tariff instead of an error.
public sealed class ResilientShippingCarrier(
    [FromKeyedServices(ResilientShippingCarrier.InnerKey)] IShippingCarrier inner,
    ResiliencePipelineProvider<string> pipelines) : IShippingCarrier
{
    public const string InnerKey = "carrier-direct";
    public const string PipelineName = "shipping-quote";

    public static readonly ShippingQuote FlatTariff = new("FLAT", 79.90m, "TRY", 3) { IsFlatRate = true };

    public async Task<ShippingQuote> GetQuoteAsync(ShippingQuoteRequest request, CancellationToken cancellationToken) =>
        await pipelines.GetPipeline<ShippingQuote>(PipelineName)
            .ExecuteAsync(async token => await inner.GetQuoteAsync(request, token), cancellationToken);

    public static void Configure(ResiliencePipelineBuilder<ShippingQuote> builder, ILogger logger) =>
        builder.AddFallback(new FallbackStrategyOptions<ShippingQuote>
        {
            // Only outages fall back; a contract violation must stay an error and raise an alert.
            ShouldHandle = new PredicateBuilder<ShippingQuote>()
                .Handle<ShippingCarrierException>(ex => ex.Kind is ShippingFailureKind.Transient or ShippingFailureKind.Unavailable),
            FallbackAction = _ => Outcome.FromResultAsValueTask(FlatTariff),
            OnFallback = args =>
            {
                logger.LogWarning(args.Outcome.Exception, "Carrier quote failed, using the flat tariff");
                return default;
            }
        });
}

ShouldHandle makes an important decision here: the schema change from the first post (ContractViolation) does not fall back. If the provider sends wrong data, covering it with a flat tariff means nobody notices the problem for days. The IsFlatRate flag travels all the way to the checkout response, so the UI can show a "shipping cost is an estimate" notice:

{"orderId":1001,"status":"Paid","total":538.90,"providerPaymentId":"ch_3MtwBwLkdIwHu7ix","declineReason":null,"shippingIsFlatRate":true}

Production Notes

For every pipeline, write down the total time and the worst-case number of attempts. For the carrier client that's 6 seconds and 3 attempts; the worst-case checkout time is the sum of these and the payment timeout.

Tune the circuit breaker threshold to your traffic and wire the moment it opens to an alert. If an open breaker doesn't show users an error (because there's a fallback), that alert is the only way to notice. We connect the metrics in the next post.

Treat manual control as an operations tool. Don't leave the admin endpoints open without authorization; the example in this post has none, and in a real application only the Admin role should be able to call them.

Agree on the cost of the fallback with the business. The flat tariff is 79.90 TRY, and when the real price is higher the difference comes out of the company's pocket; a developer shouldn't make that call alone.

Common Pitfalls

Narrowing ShouldHandle and forgetting TimeoutRejectedException. As in the third line above, timeouts are then never retried.

Assuming MaxDelay also caps Retry-After. It doesn't; either don't retry a long Retry-After, or rely on the total timeout.

Not passing the token to the callback. Polly's timeout can't stop code it can't cancel; you think you have a timeout while calls wait 30 seconds.

Retries at every layer. With separate retries in the adapter, in the HTTP pipeline and in the client calling checkout, attempt counts multiply. The Reliability Patterns post of the System Design series shows what that does under load.

Hiding contract errors behind a fallback. Fallbacks are for outages; wrong data should raise an alert.

A Small Experiment

★☆☆ In the circuit-breaker demo, remove BreakDurationGenerator and set BreakDuration = TimeSpan.FromSeconds(3). What does call 6 get after the 2.2-second wait?

★★☆ Add a circuit breaker to the SmtpEmailSender pipeline (MinimumThroughput = 5, BreakDuration = 30 s). With the fake SMTP in --smtp Error mode, run six checkouts in a row; how does the duration of the sixth compare with the first?

★★★ Build a pipeline for the payment provider with AddConcurrencyLimiter(permitLimit: 5, queueLimit: 10) and write a test that sends 50 concurrent checkouts. How many requests are rejected, and how long do the queued ones wait? Which PaymentFailureKind should the adapter translate a rejection into?

We saw every behavior in this post by hand, starting the fakes in different modes. In the next post we turn the same scenarios into automated tests: integration tests with WireMock.Net, FakeTimeProvider to test backoff without waiting, Polly's chaos strategies, and watching retries and the breaker through metrics.