Reliability Patterns: Timeouts, Retries, Backoff with Jitter, Circuit Breakers and Load Shedding

Version note: The code in this post was built and run with C# 14 and .NET 10; the simulations use fixed seeds and a virtual clock, so the output is the same on every run.

Guess First

In Bookstore, a checkout request passes through three layers: the API (Application Programming Interface) calls OrderService, OrderService calls Inventory, and Inventory calls its own database. Each team put three attempts on its own call "to handle transient errors". On campaign night, Inventory's database connection pool runs dry and every query starts failing. Meanwhile, none of the HttpClients has a timeout configured.

Two questions. How many queries does a single checkout request send to a database that is already down? And how long, at most, does an HttpClient without a configured timeout wait for a service that doesn't answer?

Timeouts: Every Call Has a Budget

Let's start with the second answer: HttpClient.Timeout defaults to 100 seconds. The user gives up after 3 seconds and refreshes the page; your thread, socket and database connection wait 97 more seconds for an answer nobody is waiting for. Under load these waits pile up, connection pools fill, and one slow dependency slows down everything that calls it. Michael Nygard's "Release It!" (2007, second edition 2018) describes this chain reaction as a cascading failure and puts timeouts first among its stability patterns: every outgoing call needs an upper bound.

How do you pick the timeout? The AWS (Amazon Web Services) Builders' Library article "Timeouts, retries, and backoff with jitter" (Marc Brooker, 2019) suggests a practical approach: choose the false timeout rate you can accept, say 0.1%, and use the dependency's latency at that percentile, p99.9 (the time 99.9% of requests stay under), as the timeout. A timeout based on average latency cuts off a sizeable share of normal but slow requests; in this series' "Back-of-the-Envelope Estimation: QPS, Storage, Bandwidth and Availability Math" post (QPS being queries per second) we saw that mixing up the average and p99 is a trap, and here the price of that trap is a direct error rate. The same article adds a warning: the timeout includes connection setup such as the TLS (Transport Layer Security) handshake; after a deploy, when every connection is being re-established, a very tight timeout can turn into mass failure.

Second rule: timeouts should shrink layer by layer. If checkout's total budget is 2 seconds, OrderService waiting 5 seconds for Inventory makes no sense. Better still is passing the budget down (deadline propagation): in .NET that means passing the request's CancellationToken to every call and narrowing it with CancelAfter where needed. The ASP.NET Core Web API series post "Async/Await and CancellationToken in ASP.NET Core: Carrying Cancellation All the Way to the Database" showed the mechanics.

Retries: Medicine or Poison

A retry is the right answer for transient errors: a packet was lost, an instance was restarting at that very moment, a connection pool was momentarily full. If the dependency has really gone down, retrying doesn't heal it; it multiplies the load. I simulated the first Guess First question: each layer tries the one below at most three times, and the database always fails.

public static int DatabaseCalls(int layers, int attemptsPerLayer, bool retryOnlyAtTop)
{
    int calls = 0;
    bool Call(int layer)
    {
        if (layer == layers) { calls++; return false; }   // the database always fails
        int attempts = retryOnlyAtTop && layer > 0 ? 1 : attemptsPerLayer;
        for (int i = 0; i < attempts; i++)
            if (Call(layer + 1)) return true;
        return false;
    }
    Call(0);
    return calls;
}

3 attempts at every layer : 27 database calls
3 attempts at top only    : 3 database calls
5 layers, 3 attempts each : 243 database calls

A single checkout sends 27 queries to a database that is already down. The example in AWS's article is five layers and 3^5 = 243 times the load, which all but eliminates the dependency's chance of recovering. Hence two rules. First, retry at a single layer; usually the one that sees the error first, directly above the dependency. The "Handling Overload" chapter of Google's SRE (Site Reliability Engineering) book says the same: only the layer immediately above the one rejecting the request should retry. Second, retry only idempotent operations. A GET, or a POST carrying an Idempotency-Key, is safe to repeat; repeating a keyless "create the order" call when the first attempt actually succeeded but its response was lost on the way means two orders. The Solution Patterns series post "Idempotency Keys: Not Creating the Same Order Twice on Double Clicks and Retries" covered that side end to end.

What gets retried should be explicit too: 503, 429, 408 and connection errors yes; 400, 401, 404 and business rule errors no. And if a 429 comes with a Retry-After header, the server is telling you how long to wait.

Exponential Backoff and Jitter

Waiting between retries is a must, and the shape of the wait matters. Exponential backoff doubles the wait on every attempt: 100, 200, 400, 800 ms. But if a thousand clients failed at the same moment, they all wait 100 ms and all come back at the same moment; the load doesn't spread out, it repeats as a wave. The cure is jitter: adding randomness to the wait. Marc Brooker's 2015 AWS Architecture Blog post "Exponential Backoff And Jitter" compared jitter variants by simulation and found backoff without jitter to be by far the worst option; the simplest good option is full jitter, random(0, cap). I made a thousand clients fail at t=0, retried four times, and measured the highest number of calls landing in any 10 ms slot:

for (int attempt = 0; attempt < 4; attempt++)
{
    double cap = 100 * Math.Pow(2, attempt);              // 100, 200, 400, 800 ms
    t += useJitter ? random.NextDouble() * cap : cap;     // full jitter: random(0, cap)
    long slot = (long)(t / 10);
    slots[slot] = slots.GetValueOrDefault(slot) + 1;
}

backoff, no jitter        : peak 1000 calls/10 ms
backoff, full jitter      : peak 156 calls/10 ms

Backoff without jitter piles all thousand calls into the same 10 ms; with full jitter the peak drops to 156. The same idea applies beyond retries: cache TTLs (Time To Live), scheduled jobs, health check intervals. If they all align on the same second, the system creates its own thundering herd.

Retry Budgets

Backoff and jitter spread retries over time but don't reduce their number. If the dependency stays down for a long time, every three-attempt client still triples the load. The Google SRE book's answer is a two-level budget: at most three attempts per request, and per client, retries kept below 10% of all requests. The book's numbers say that with only the per-request limit the worst case approaches three times the load, while with the 10% budget it stays around 1.1 times. The simulation agrees:

public static int TotalAttempts(int requests, bool useBudget)
{
    int attempts = 0, retries = 0;
    for (int r = 1; r <= requests; r++)
    {
        attempts++;                                   // first try fails
        for (int retry = 0; retry < 2; retry++)       // up to 3 attempts in total
        {
            if (useBudget && retries + 1 > r * 0.10) break;   // retries must stay under 10% of requests
            retries++; attempts++;
        }
    }
    return attempts;
}

3 attempts, no budget     : 30000 attempts
3 attempts, 10% budget    : 11000 attempts

In healthy times the budget almost never kicks in; transient error rates are far below 10%. The moment it does kick in is a sign the problem isn't transient. AWS SDKs (Software Development Kit) have limited retries with a local token bucket since 2016; same logic.

Circuit Breakers and Bulkheads

A retry budget says "try less"; a circuit breaker says "don't try at all for a while". Defined by Nygard in "Release It!" and popularized by Martin Fowler's 2014 CircuitBreaker article, the pattern is a three-state state machine: in Closed, calls go through and their results are counted; when the failure rate crosses a threshold it moves to Open, and calls fail immediately without reaching the dependency (fail fast); after a set time it moves to Half-Open and lets a single trial call through, returning to Closed on success and to Open otherwise. A small implementation to see the logic:

public enum BreakerState { Closed, Open, HalfOpen }

public sealed class CircuitBreaker(double failureRatio, int minThroughput, TimeSpan samplingWindow, TimeSpan breakDuration)
{
    private readonly Queue<(TimeSpan At, bool Failed)> _samples = new();
    private TimeSpan _openedAt;
    public BreakerState State { get; private set; } = BreakerState.Closed;

    public bool AllowCall(TimeSpan now)
    {
        if (State == BreakerState.Open && now - _openedAt >= breakDuration)
            State = BreakerState.HalfOpen;               // let one trial call through
        return State != BreakerState.Open;
    }

    public void Record(TimeSpan now, bool failed)
    {
        if (State == BreakerState.HalfOpen)
        {
            State = failed ? BreakerState.Open : BreakerState.Closed;
            if (failed) _openedAt = now;
            _samples.Clear();
            return;
        }
        _samples.Enqueue((now, failed));
        while (now - _samples.Peek().At > samplingWindow) _samples.Dequeue();
        int failures = _samples.Count(s => s.Failed);
        if (_samples.Count >= minThroughput && failures >= failureRatio * _samples.Count)
        {
            State = BreakerState.Open;
            _openedAt = now;
        }
    }
}

With a virtual clock I made one call every 50 ms for 12 seconds; the dependency is down for the first 6 seconds. The breaker's threshold is at least 10 calls and 50% failures in a 2-second window, with 2 seconds in Open:

  t= 0.45s Closed -> Open
  t= 2.45s Open -> HalfOpen
  t= 2.45s HalfOpen -> Open
  t= 4.45s Open -> HalfOpen
  t= 4.45s HalfOpen -> Open
  t= 6.45s Open -> HalfOpen
  t= 6.45s HalfOpen -> Closed
circuit breaker           : 123 calls sent, 117 failed fast without a call

117 of the 240 calls returned without ever reaching the dependency; the failed service got the breathing room it needed to recover, and the caller didn't burn its threads waiting for timeouts. The output shows the cost too: the dependency recovered at second 6.0, and the breaker kept failing fast until 6.45. That's why AWS's article notes that circuit breakers add modal behavior to a system and can slow recovery. Set up a breaker per dependency (a single breaker for all outgoing calls shuts off a healthy dependency too) and design in advance what the user sees while it's open.

The bulkhead is named after a ship's watertight compartments: you partition resources per dependency so that one being exhausted doesn't affect the others. If calls to a slowing payment service can use at most 20 concurrent slots, the catalog page keeps working with its own slots. In .NET the simplest form is a ConcurrencyLimiter per dependency, or a separate HttpClient with a connection limit.

Load Shedding and Graceful Degradation

The patterns so far were the caller protecting itself. Load shedding is the callee protecting itself: when a server sees it has reached the end of its capacity (CPU (Central Processing Unit) usage, queue depth, concurrent request count), instead of trying to process new requests and slowing all of them down, it rejects some of them immediately and cheaply. The AWS Builders' Library article "Using load shedding to avoid overload" explains this with the idea of goodput: an overloaded server working on requests that the client is going to time out anyway looks busy in throughput terms, while the useful work done (goodput) drops to zero.

The subtle part of load shedding is deciding what to drop. In the Stripe example we mentioned in the previous post, part of the fleet is reserved for critical requests, and under overload test mode traffic is dropped first. For Bookstore the priority order is clear: checkout and payment are dropped last, the recommendation list and the "customers who bought this also bought" box first. That connects to graceful degradation: if the recommendation service doesn't answer, the page doesn't fail; the box comes back empty or with yesterday's list from cache; if search is slow, autocomplete turns off. Which feature can be switched off is a product decision, and it should be made in advance, together with feature flags, not in the middle of a crisis.

Redundancy and Failover

The last line of defense is redundancy. In an active-passive setup one copy carries the traffic and the other waits; when the primary goes down, health checks notice and traffic switches to the standby. It's simple, but you only learn whether the standby really works at failover time, and the switch takes seconds to minutes. In an active-active setup every copy carries traffic; when one goes down the rest take over its load. Capacity has to be planned for that: in a two-region active-active system each region must be able to carry the whole load on its own, so it should normally be sized for at most 50% utilization. The availability math from the series' "Back-of-the-Envelope Estimation" post comes back here: the chance that at least one of two independent copies is up is much higher than for a single copy, but only if the failover mechanism itself is reliable. In the data layer, failover brings problems like replication lag and both sides believing they're the primary (split brain); the theory behind them is the subject of the upcoming Distributed Systems Fundamentals series.

All Together in .NET

You don't need to write these patterns by hand for every HttpClient. The Microsoft.Extensions.Http.Resilience package's AddStandardResilienceHandler chains five strategies on top of Polly; the defaults on Microsoft Learn, from outermost to innermost:

The ordering matches this post's logic exactly: the total timeout sits outside the retries, the per-attempt timeout inside; the breaker sits inside the retry, so every attempt counts toward it. Two settings need attention: by default the retry runs for all HTTP (Hypertext Transfer Protocol) methods, so turn it off for non-idempotent calls with DisableForUnsafeHttpMethods(); and a 30-second total timeout is usually far too long for a path the user is waiting on, like checkout. Detailed use of the package is the subject of the upcoming External Service Integration and Resilience series in the Backend category, and the production incident side belongs to the SRE and Production Incidents series in the DevOps category.

Common Pitfalls

Retrying at every layer

Three layers, three attempts each: one request becomes 27 calls. Put retries at a single layer, and tell upper layers "don't retry" through the error code.

Trusting the default timeout

A 100-second HttpClient.Timeout fits no user flow. Choose the timeout from the dependency's p99.9 and pass the total budget down with CancellationToken.

Backoff without jitter

Everyone who failed comes back at the same moment. Backoff should always have jitter; the same goes for TTLs and scheduled jobs.

Retrying a non-idempotent operation

Repeating a "take payment" call whose response got lost means charging twice. Either use an Idempotency-Key or turn retries off.

Shutting off all dependencies with a single breaker

Breakers and bulkheads are set up per dependency; otherwise a slow recommendation service stops payment calls too.

A Small Experiment

★☆☆ Change DatabaseCalls so retries happen at two layers (the top and the bottom). How many calls does the database see in the three-layer chain?

★★☆ Add "equal jitter" (cap / 2 + random(0, cap / 2)) to the Backoff simulation. How does the peak compare with full jitter? Compare with the result in Brooker's post.

★★★ Bookstore's product detail page calls five dependencies: catalog, price, stock, reviews, recommendations. Prepare a table with the timeout, retry, breaker and degradation decision for each. Which one going down should make the whole page fail?

We now protect the system both from clients asking too much and from dependencies failing. One building block remains before the case studies: how do we generate unique, sortable IDs across this many instances, shards and services? In the next post we look at the options, from auto-increment to UUIDv7 and Snowflake.