The Standard Pipeline with Microsoft.Extensions.Http.Resilience: Defaults, Settings and Pitfalls

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.Http.Resilience 10.11, Polly 8.9 and WireMock.Net 2.31.

Guess First

You added AddStandardResilienceHandler() to the carrier client and wrote client.Timeout = TimeSpan.FromSeconds(3) in the same registration. The carrier API answers every request in 5 seconds. After how many seconds does the request end?

A teammate adds a second AddStandardResilienceHandler() to the same client, just in case. If the provider keeps returning 500, how many requests reach it?

The payment provider's POST /v1/charges returned 503 once. With default settings, does the standard pipeline retry that POST?

The answers: not 3 seconds but 16.8; not 4 requests but 16; yes, it does. We will see all three with real runs in this post. In the previous post we set up our clients with IHttpClientFactory, but every call was a single attempt. Now we add the standard pipeline from the Microsoft.Extensions.Http.Resilience package.

The experiments are again commands in Bookstore.ResilienceLab; each starts its own fake server:

dotnet run --project samples/Bookstore.ResilienceLab -- standard-defaults
dotnet run --project samples/Bookstore.ResilienceLab -- standard-retry
dotnet run --project samples/Bookstore.ResilienceLab -- standard-post
dotnet run --project samples/Bookstore.ResilienceLab -- standard-validation
dotnet run --project samples/Bookstore.ResilienceLab -- client-timeout
dotnet run --project samples/Bookstore.ResilienceLab -- stacked
dotnet run --project samples/Bookstore.ResilienceLab -- hedging

What the Standard Pipeline Brings

The package adds a DelegatingHandler built on Polly v8 to an IHttpClientBuilder. AddStandardResilienceHandler() stacks five strategies, outside in, in a single line. I read the values from HttpStandardResilienceOptions itself rather than from the docs:

var o = new HttpStandardResilienceOptions();
Console.WriteLine($"RateLimiter       : permits={o.RateLimiter.DefaultRateLimiterOptions.PermitLimit}, queue={o.RateLimiter.DefaultRateLimiterOptions.QueueLimit}");
Console.WriteLine($"TotalRequestTimeout: {o.TotalRequestTimeout.Timeout}");
Console.WriteLine($"Retry             : max={o.Retry.MaxRetryAttempts}, backoff={o.Retry.BackoffType}, jitter={o.Retry.UseJitter}, delay={o.Retry.Delay}, retryAfter={o.Retry.ShouldRetryAfterHeader}");
Console.WriteLine($"CircuitBreaker    : ratio={o.CircuitBreaker.FailureRatio}, minThroughput={o.CircuitBreaker.MinimumThroughput}, sampling={o.CircuitBreaker.SamplingDuration}, break={o.CircuitBreaker.BreakDuration}");
Console.WriteLine($"AttemptTimeout    : {o.AttemptTimeout.Timeout}");

RateLimiter       : permits=1000, queue=0
TotalRequestTimeout: 00:00:30
Retry             : max=3, backoff=Exponential, jitter=True, delay=00:00:02, retryAfter=True
CircuitBreaker    : ratio=0.1, minThroughput=100, sampling=00:00:30, break=00:00:05
AttemptTimeout    : 00:00:10

The order matters. The rate limiter is outermost, so at most 1000 requests can be inside the pipeline at once, and with a queue of 0 the 1001st is rejected without waiting. The total timeout sits outside retry: all attempts and the waits between them can't exceed 30 seconds together. The circuit breaker and the attempt timeout sit inside retry: each attempt is limited to 10 seconds on its own, and each attempt counts as a separate result in the breaker's statistics. Because ShouldRetryAfterHeader is true, if the provider sends Retry-After, retry waits for that instead of its own calculated delay.

Retry and the circuit breaker handle the same failures: status codes 500 and above, 408 and 429, plus HttpRequestException and Polly's TimeoutRejectedException. Responses like 400, 401 or 404 are not retried; sending the same request again gives the same answer.

Microsoft.Extensions.Http.Resilience 11.0 shipped together with .NET 11 and also supports net10.0. This series stays on the .NET 10 LTS base, so we use the 10.x line; the APIs in this post are the same in both.

The First Retry: Exponential Backoff and Jitter

A fake standing in for the exchange-rate service answers the first two requests with 503 and the third with 200. I added only the standard handler to the client and turned on Polly's logs:

services.AddHttpClient("rates", c => c.BaseAddress = server.Url)
    .AddStandardResilienceHandler();

12:07:20.448 warn: Polly[0] Resilience event occurred. EventName: 'OnRetry', Source: 'rates-standard//Standard-Retry', Operation Key: '', Result: '503'
12:07:21.610 warn: Polly[0] Resilience event occurred. EventName: 'OnRetry', Source: 'rates-standard//Standard-Retry', Operation Key: '', Result: '503'
12:07:24.610 info: Polly[3] Execution attempt. Source: 'rates-standard//Standard-Retry', Operation Key: '', Result: '200', Handled: 'False', Attempt: '2', Execution Time: 1.5283ms
12:07:24.614 info: Polly[2] Resilience pipeline executed. Source: 'rates-standard/', Operation Key: '', Result: '200', Execution Time: 4435.7822ms
GET -> 200 after 3 attempts in 4.4 s

The first wait is about 1.2 seconds, the second 3 seconds. The base delay is 2 seconds and exponential, but with jitter on the waits are not exactly 2 and 4. For exponential backoff Polly uses a formula called decorrelated jitter; the goal is to keep hundreds of clients that failed at the same moment from coming back in the same second. Why that matters is covered in the Reliability Patterns post of the System Design series.

The Source field in the log has the shape <client-name>-standard//Standard-Retry. That is the name we will see in metrics and alerts in post 05; it tells you which strategy of which client fired.

Retrying a POST: DisableForUnsafeHttpMethods

By default the standard pipeline retries every HTTP method. For payments that's dangerous: the first request may have reached the provider and charged the card, and a proxy may have produced the 503 on the way back. I sent a POST to the same fake, first with the defaults, then with retries turned off for unsafe methods:

.AddStandardResilienceHandler(o =>
{
    o.Retry.Delay = TimeSpan.FromMilliseconds(100);
    if (disable)
    {
        o.Retry.DisableForUnsafeHttpMethods();
    }
})

DisableForUnsafeHttpMethods=False: POST -> 200, provider saw 3 request(s)
DisableForUnsafeHttpMethods=True : POST -> 503, provider saw 1 request(s)

With the defaults the provider saw the same payment request three times. DisableForUnsafeHttpMethods turns retries off for POST, PATCH, PUT, DELETE and CONNECT, the methods RFC 9110 doesn't consider "safe". If you want to pick methods yourself there is DisableFor(HttpMethod.Post). By RFC 9110, PUT and DELETE are idempotent, so resending them is safe by definition; but unless you're sure the other side really implements them that way, turning them all off produces fewer surprises.

The way to retry a POST safely is to make it idempotent: send the provider an Idempotency-Key, and have the provider refuse to process a second request with the same key. That is the topic of post 06 in this series. Until then, retries stay off in Bookstore's payment client.

Bookstore's Pipelines: appsettings and Code

The three Bookstore clients have three different needs. The payment POST must not be retried and has to be quick, because the user is waiting on the checkout screen. The carrier price is a GET; retrying twice is reasonable, but the user is still waiting. Exchange rates are read-heavy and come from a service running in two regions. We put the values in appsettings so they can change per environment, but the "no retries on POST" rule stays in code; it is a decision configuration must not be able to turn off:

"Resilience": {
  "PaymentProvider": {
    "AttemptTimeout": { "Timeout": "00:00:03" },
    "TotalRequestTimeout": { "Timeout": "00:00:04" }
  },
  "LegacyCarrier": {
    "AttemptTimeout": { "Timeout": "00:00:02" },
    "TotalRequestTimeout": { "Timeout": "00:00:06" },
    "Retry": { "MaxRetryAttempts": 2, "Delay": "00:00:00.200" }
  }
}

// Payments: POST is not idempotent yet (post 06), so no retries; short timeouts from appsettings.
var payments = builder.Services.AddHttpClient(ProviderPaymentGateway.ClientName, client =>
    {
        client.BaseAddress = new Uri(builder.Configuration["Payments:Provider:BaseUrl"]!);
        client.DefaultRequestHeaders.Authorization = new("Bearer", builder.Configuration["Payments:Provider:ApiKey"]);
    })
    .AddAsKeyed();
payments.AddStandardResilienceHandler()
    .Configure(builder.Configuration.GetSection("Resilience:PaymentProvider"))
    .Configure(options => options.Retry.DisableForUnsafeHttpMethods());

var carrier = builder.Services.AddHttpClient(LegacyCarrierClient.ClientName, (sp, client) =>
        client.BaseAddress = sp.GetRequiredService<IOptions<LegacyCarrierOptions>>().Value.BaseUrl)
    .AddAsKeyed();
carrier.AddStandardResilienceHandler()
    .Configure(builder.Configuration.GetSection("Resilience:LegacyCarrier"));
// Added after (inside) the resilience handler: every retry asks for the current token.
carrier.AddHttpMessageHandler<CarrierAuthHandler>();

Watch the handler order. CarrierAuthHandler is added after the resilience handler, so it sits inside it in the chain. Every retry passes through the auth handler again, and if the token was refreshed in the meantime the new one is sent. In the opposite order the token is added once and all retries go out with the same token.

The pipeline now throws Polly's own exceptions too, and the adapter has to translate them into our error model. We added two catch blocks to the payment adapter:

catch (BrokenCircuitException ex)
{
    // The circuit is open: the request never left our process.
    throw new PaymentGatewayException(PaymentFailureKind.Unavailable, "Payment provider circuit is open.", ex)
        { RetryAfter = ex.RetryAfter };
}
catch (TimeoutRejectedException ex)
{
    // The resilience pipeline gave up; the request may have reached the provider.
    throw new PaymentGatewayException(PaymentFailureKind.UnknownOutcome, "Payment provider did not answer in time.", ex);
}

The block from the first post that catches TaskCanceledException is no longer enough on its own: when the attempt timeout elapses, the exception is TimeoutRejectedException. Had we missed that, the "unknown outcome" case would come back as a 500. I tried checkout while the payment provider took 5 seconds to answer:

HTTP/1.1 202 Accepted
{"title":"Payment is being verified.","status":202,"traceId":"00-ca7e0baa8b311fc77f09210106d4bb8d-905daef5fce3105b-00"}

The response arrived after 4.05 seconds: about 1 second for the carrier call (the first request, so including the token fetch) and 3 seconds for the payment attempt. The log has the OnTimeout event from Standard-AttemptTimeout and an Execution Time: 3006.5167ms line.

Validation: The Timeouts Have to Fit Together

The options are validated when the first client is created. I made the attempt timeout larger than the total timeout:

.AddStandardResilienceHandler(o =>
{
    o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(20);
    o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(15);
})

Total request timeout resilience strategy must have a greater timeout than the attempt resilience strategy. Total Request Timeout: 15s, Attempt Timeout: 20s; The sampling duration of circuit breaker strategy needs to be at least double of an attempt timeout strategy’s timeout interval, in order to be effective. Sampling Duration: 30s,Attempt Timeout: 20s

Two rules fired at once. The first is obvious: a single attempt can't be longer than the whole budget. The second is less known: the circuit breaker's sampling duration must be at least twice the attempt timeout. A 20-second attempt can time out at most once in a 30-second window, so the breaker can't compute a meaningful ratio. The exception is an OptionsValidationException, and it arrives not at startup but when the client is first created. That's why triggering a first request with a smoke test after a deployment that changes these values is a good habit.

How HttpClient.Timeout Affects the Pipeline

Here is the surprise from the opening question. I gave the carrier client a 3-second HttpClient.Timeout and a 2-second attempt timeout, with the fake holding every request for 5 seconds. First I set the timeout inside AddHttpClient, then after the handler with ConfigureHttpClient:

var http = s.AddHttpClient("carrier", c =>
{
    c.BaseAddress = server.Url;
    c.Timeout = TimeSpan.FromSeconds(3);
});
http.AddStandardResilienceHandler(o => o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2));
if (setAfterHandler)
{
    http.ConfigureHttpClient(c => c.Timeout = TimeSpan.FromSeconds(3));
}

Timeout set before the handler: HttpClient.Timeout=-00:00:00.0010000
  TimeoutRejectedException after 16.8 s, server saw 4 attempt(s)
Timeout set after the handler: HttpClient.Timeout=00:00:03
  TaskCanceledException after 3.0 s, server saw 2 attempt(s)

In the first case HttpClient.Timeout is -1 ms, that is, Timeout.InfiniteTimeSpan. As a comment in its source says, AddStandardResilienceHandler sets HttpClient.Timeout to infinite so the pipeline controls timeouts. It does this with its own ConfigureHttpClient call, and because configure actions run in registration order, any value you set before it is silently overwritten. The result: four attempts, 16.8 seconds. In the second case the value came after the handler and cut everything, retries included, at the 3-second mark.

A practical rule: with the standard handler, set durations through AttemptTimeout and TotalRequestTimeout and leave HttpClient.Timeout alone. That's why we removed the 5- and 10-second values we gave the clients in the previous post. Only when you build your own pipeline with AddResilienceHandler do you need to make sure HttpClient.Timeout is larger than the total; nothing sets it for you there.

One Handler Rule and Hedging

Microsoft's documentation says to add only one resilience handler per client. I tried what happens if you don't, against a fake that always returns 500:

1 handler(s): 500, server saw 4 requests
2 handler(s): 500, server saw 16 requests

Each attempt of the outer handler runs four attempts of the inner one: 4 × 4 = 16. In a three-hop microservice chain where every hop retries like this, the number climbs to 64. That is the retry amplification from the System Design series, in code. If you set an application-wide default with ConfigureHttpClientDefaults(b => b.AddStandardResilienceHandler()) and want a different strategy for one client, you first remove the existing handler with RemoveAllResilienceHandlers(). In 10.11 that method is marked experimental; the build fails with EXTEXP0001, and you have to suppress the diagnostic on purpose to use it.

The exchange-rate service is an example of that need. It runs in two regions and we're only reading; if one region is slow, asking the other one as well beats waiting. Hedging does exactly that: if the first request doesn't return within a set delay, a second one starts in parallel, the first to return wins and the other is cancelled. With the primary region answering 3 seconds late:

s.ConfigureHttpClientDefaults(b => b.AddStandardResilienceHandler());
#pragma warning disable EXTEXP0001 // RemoveAllResilienceHandlers is experimental
s.AddHttpClient("exchange-rates", c => c.BaseAddress = new Uri(primary.Url!))
    .RemoveAllResilienceHandlers()
#pragma warning restore EXTEXP0001
    .AddStandardHedgingHandler(routing => routing.ConfigureOrderedGroups(groups =>
    {
        groups.Groups.Add(new UriEndpointGroup { Endpoints = { new WeightedUriEndpoint { Uri = new Uri(primary.Url!) } } });
        groups.Groups.Add(new UriEndpointGroup { Endpoints = { new WeightedUriEndpoint { Uri = new Uri(secondary.Url!) } } });
    }))
    .Configure(o => o.Hedging.Delay = TimeSpan.FromMilliseconds(500));

12:07:49.997 warn: Polly[0] Resilience event occurred. EventName: 'OnHedging', Source: 'exchange-rates-standard-hedging//StandardHedging-Hedging', Operation Key: '', Result: ''
{"pair":"EUR-TRY","rate":44.1250,"source":"secondary"} in 630 ms

After 500 ms a request went to the second region and the answer came back at 630 ms; the request to the primary region was cancelled. In the standard hedging handler the circuit breaker, rate limiter and attempt timeout are kept per endpoint, so if the slow region's breaker opens, hedging skips that region. The price of hedging is extra load: every slow request means two requests to the provider. That is why it only makes sense for idempotent reads with more than one endpoint. Hedging a payment POST is a fast way to charge a card twice.

Production Notes

Microsoft.Extensions.Http.Resilience 10.11 depends on Polly 8.4.2. The latest on NuGet is 8.9.1, and properties such as BrokenCircuitException.RetryAfter arrived in between. In Bookstore we reference Polly.Extensions and Polly.RateLimiting 8.9.1 explicitly. When you raise transitive dependencies on purpose, read the release notes and run the tests.

Review the defaults against your own situation. A 30-second total is far too long for a screen a user is waiting on, and reasonable for a background sync job. The circuit breaker's minimum throughput of 100 requests means the breaker never opens for a low-traffic provider: at 20 requests a minute you never reach that number in a 30-second window. We look at breaker settings in detail in the next post.

Polly logs every retry and timeout under the Polly category. Keeping those logs at Warning in production answers the question "why did checkout take 4 seconds". We cover them together with metrics in post 05.

Common Pitfalls

Retrying POST. The default pipeline retries every method; use DisableForUnsafeHttpMethods for calls that aren't idempotent.

Adding two resilience handlers. Retry counts multiply; the 16 requests above are the smallest example.

Setting HttpClient.Timeout before the standard handler and assuming it works. The value is silently set to infinite.

Not handling TimeoutRejectedException and BrokenCircuitException in the adapter. They aren't HttpRequestException, and if not caught separately they reach the user as a 500.

Adding the auth handler before the resilience handler. The token is added once; even after it is refreshed, retries go out with the old token.

A Small Experiment

★☆☆ In the standard-retry demo, set o.Retry.UseJitter = false. Exactly how long are the two waits? Then try BackoffType = DelayBackoffType.Linear.

★★☆ Start the fake carrier in --carrier Throttle mode (429 + Retry-After: 2) and call checkout. Look at the timestamps of the OnRetry lines in the API log: are the waits 2 seconds, or the exponential calculation? Then set ShouldRetryAfterHeader to false in appsettings and measure again.

★★★ Write a second exchange-rate client that uses the standard pipeline instead of hedging. With the primary region 3 seconds late, compare p50 and p99 for both clients over 100 requests. Count the total requests hedging sends to the provider as well.

The standard pipeline is a ready-made package and a good start in most cases. But falling back to a flat rate when the carrier price is unavailable, opening the breaker by hand during a maintenance window, or protecting a non-HTTP call such as SMTP all need Polly directly. In the next post we build the strategies one by one with ResiliencePipelineBuilder.