Cache Stampede, Penetration and Avalanche: Three Problems That Break Caches in Production

Version note: The code in this post was built and run with .NET 10; the examples ran on Entity Framework Core (EF Core) 10.0.14 (SQLite), Microsoft.Extensions.Caching.Hybrid 10.11.0, StackExchange.Redis 3.4.31, Redis 8.2 and k6 2.3.

Guess First

Bookstore's API (Application Programming Interface) runs on three instances, all using the same Redis as L2 (level 2, the shared out-of-process cache). The category sales report is cached with HybridCache. After a deploy the cache is empty, and k6 fires 2,000 requests at the report endpoint from 500 concurrent VUs (virtual users). How many times does the report query run against the database?

Second question: book details are read by ISBN (International Standard Book Number), and HybridCache.GetOrCreateAsync returns null for a missing book. A bot sends 50,000 requests with ISBNs that are well-formed but not in the database. If the bot cycles through the same 5,000 ISBNs, how many queries reach the database? And if it generates a new ISBN for every request?

Three Problems, Three Different Mechanisms

All three end the same way, with the load the cache was absorbing landing on the database at once, but the causes differ. A stampede is every request waiting on one popular key running the factory at the same moment the key expires. Penetration is keys that never existed punching through the cache: since the cache only holds what was found, every request goes to the database. An avalanche is many keys dying at once, either because they were written in the same second with the same TTL (Time to Live) or because the cache server went down.

The theory of stampedes, meaning request coalescing, adding jitter to TTLs and probabilistic early refresh with XFetch, was covered with simulation numbers in the System Design series post "Caching Strategies: Cache-Aside, Write-Through, Write-Behind and Cache Stampede"; we won't repeat it here. This post measures the same problems in .NET with a real Redis and k6. Queries are counted by the EF Core interceptor from the first post of the series, and each instance reports its own count from /stats. k6 itself and scenario design belong to the upcoming Performance and Load Testing series.

Stampede: Coalescing Inside an Instance

Start with the classic cache-aside from the ASP.NET Core Web API series post "Caching in ASP.NET Core Web API: IMemoryCache, Redis and Response Caching", on Redis. The report query is a GROUP BY that aggregates 50,000 books by category and takes about 15 ms on its own:

app.MapGet("/api/reports/category-sales/naive", async (CatalogDb db, IDistributedCache cache, CancellationToken ct) =>
{
    string? json = await cache.GetStringAsync("v1:report:category-sales:naive", ct);
    if (json is not null) return JsonSerializer.Deserialize<List<CategorySales>>(json)!;
    List<CategorySales> report = await Reports.CategorySalesAsync(db, ct);
    await cache.SetStringAsync("v1:report:category-sales:naive", JsonSerializer.Serialize(report),
        new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5) }, ct);
    return report;
});

app.MapGet("/api/reports/category-sales", (CatalogDb db, HybridCache cache, CancellationToken ct) =>
    cache.GetOrCreateAsync("v1:report:category-sales",
        async token => await Reports.CategorySalesAsync(db, token), reportOptions, cancellationToken: ct));

The k6 scenario imitates a cold cache: 500 VUs start at once and spread their requests across the three instances in turn:

import http from 'k6/http';
import { summaryLine } from './summary.js';

// cold cache: 500 VUs ask for the same report at once, requests spread over 3 instances
export const options = {
  scenarios: { cold: { executor: 'shared-iterations', vus: 500, iterations: 2000, maxDuration: '120s' } },
};
const hosts = __ENV.HOSTS.split(',');

export default function () {
  http.get(`http://${hosts[__VU % hosts.length]}${__ENV.ENDPOINT}`);
}

export function handleSummary(data) { return { stdout: summaryLine(data) }; }

Before every run, run_stampede.sh empties Redis with FLUSHALL, restarts the three instances (so L1 (level 1, the in-process cache) is empty too), runs k6 in a grafana/k6 container and finally adds up each instance's query count:

$ ./run_stampede.sh /api/reports/category-sales/naive
reqs=2000 rps=869 p50=89.6ms p95=1584.4ms failed=0.00%
instance 17540 queries=89
instance 17541 queries=79
instance 17542 queries=55
total queries=223
$ ./run_stampede.sh /api/reports/category-sales
reqs=2000 rps=6923 p50=30.5ms p95=194.1ms failed=0.00%
instance 17540 queries=1
instance 17541 queries=1
instance 17542 queries=1
total queries=3

With classic cache-aside, 223 of the 2,000 requests found the cache empty and computed the report themselves; a second run of the same command gave 117, because how many requests land in the same window depends on how simultaneously the VUs start. With HybridCache the number is 3: as Microsoft Learn states, only one caller per key runs the factory per instance and the others wait for its result. That's why the answer to the first guess is 3, not 1; a ten-instance deployment computes ten reports at the same time.

In output cache the same protection is called resource locking and it's on by default (SetLocking(false) turns it off); the details are in the sixth post of the series. It, too, protects only within its own instance.

One Computation Across Instances: A Redis Lock

If the report is expensive (an aggregation that takes seconds, a paid call to an external service), even one computation per instance is too many. The fix is to put a cross-instance lock in front of the factory: whoever takes the lock computes and writes the result to Redis, and the others wait for it. StackExchange.Redis's LockTakeAsync and LockReleaseAsync methods already do this with SET NX and a token-checked delete:

// One recomputation per key across all instances: the winner computes, the others wait for the result
public sealed class RedisSingleFlight(IConnectionMultiplexer redis, ILogger<RedisSingleFlight> logger)
{
    private static readonly TimeSpan LockTtl = TimeSpan.FromSeconds(10);
    private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(50);
    private static readonly TimeSpan MaxWait = TimeSpan.FromSeconds(3);

    public async Task<T> GetOrComputeAsync<T>(
        string key, Func<CancellationToken, Task<T>> factory, TimeSpan ttl, CancellationToken ct)
    {
        IDatabase db = redis.GetDatabase();
        if (await TryReadAsync<T>(db, key) is { } hit) return hit.Value;

        string lockKey = $"{key}:lock";
        string token = Guid.NewGuid().ToString("N");
        if (await db.LockTakeAsync(lockKey, token, LockTtl))
        {
            try
            {
                // someone may have written it while we were taking the lock
                if (await TryReadAsync<T>(db, key) is { } late) return late.Value;
                T value = await factory(ct);
                await db.StringSetAsync(key, JsonSerializer.Serialize(value), ttl);
                return value;
            }
            finally
            {
                await db.LockReleaseAsync(lockKey, token);
            }
        }

        // someone else holds the lock: wait for the result, but not forever
        long deadline = Environment.TickCount64 + (long)MaxWait.TotalMilliseconds;
        while (Environment.TickCount64 < deadline)
        {
            await Task.Delay(PollInterval, ct);
            if (await TryReadAsync<T>(db, key) is { } awaited) return awaited.Value;
        }

        logger.LogWarning("Lock holder for {Key} did not finish in {MaxWait}; computing locally", key, MaxWait);
        return await factory(ct);
    }

    private static async Task<Hit<T>?> TryReadAsync<T>(IDatabase db, string key)
    {
        RedisValue raw = await db.StringGetAsync(key);
        return raw.HasValue ? new Hit<T>(JsonSerializer.Deserialize<T>(raw.ToString())!) : null;
    }

    private sealed record Hit<T>(T Value);
}

The Hit<T> wrapper is deliberate: when T is a value type or a type that can legitimately be null, "not in the cache" must be distinguishable from "the cached value is null". The endpoint stacks two layers: HybridCache collapses the 500 requests inside an instance into one call, and that one call races the other instances through the lock. We turn L2 off for this key, because the shared copy now lives under RedisSingleFlight's key:

// L1 coalescing from HybridCache, single computation across instances from the Redis lock
app.MapGet("/api/reports/category-sales/locked", (CatalogDb db, HybridCache cache, RedisSingleFlight singleFlight, CancellationToken ct) =>
    cache.GetOrCreateAsync("v1:report:category-sales:l1",
        async token => await singleFlight.GetOrComputeAsync("v1:report:category-sales:shared",
            t => Reports.CategorySalesAsync(db, t), TimeSpan.FromMinutes(5), token),
        new HybridCacheEntryOptions
        {
            LocalCacheExpiration = TimeSpan.FromMinutes(1),
            Flags = HybridCacheEntryFlags.DisableDistributedCache
        },
        cancellationToken: ct));

$ ./run_stampede.sh /api/reports/category-sales/locked
reqs=2000 rps=7064 p50=34.2ms p95=142.7ms failed=0.00%
instance 17540 queries=0
instance 17541 queries=0
instance 17542 queries=1
total queries=1

Three instances, 2,000 requests, one query. This is an efficiency lock: even if the lock TTL runs out and two instances compute at the same time, the result is still correct; the work is just done twice. The Distributed Systems Fundamentals series post "Distributed Locks, Leases and Fencing Tokens: Why a Lock Is Not Enough" explains that distinction and why a single Redis node is enough for efficiency but not for correctness; the rule that the lease must outlive the factory comes from there as well. If you'd rather not write this code yourself, FusionCache 2.9's WithDistributedLocker with RedisDistributedLocker from the ZiggyCreatures.FusionCache.Locking.Distributed.Redis package does the same job. For refreshing in the background before expiry, FusionCache also has EagerRefreshThreshold; HybridCache has no XFetch-style early refresh.

Penetration: The Attack of Missing Keys

Classic cache-aside doesn't write a missing record to the cache; under normal traffic that's the right behavior. Under attack it leaves the door open. The k6 scenario generates ISBNs that are well-formed but not in the database; with POOL set it cycles through the same 5,000 ISBNs, without it every request gets a new one:

import http from 'k6/http';
import { sleep } from 'k6';
import { summaryLine } from './summary.js';

// well-formed ISBNs that don't exist; with POOL=0 every request gets a new ISBN
export const options = {
  scenarios: { attack: { executor: 'shared-iterations', vus: 500, iterations: 50000, maxDuration: '300s' } },
};
http.setResponseCallback(http.expectedStatuses(200, 404));
const pool = parseInt(__ENV.POOL || '0');

function isbn(n) {
  const first12 = '9782' + String(n).padStart(8, '0');
  let sum = 0;
  for (let i = 0; i < 12; i++) sum += Number(first12[i]) * (i % 2 === 0 ? 1 : 3);
  return first12 + ((10 - (sum % 10)) % 10);
}

export default function () {
  // spread connection setup over the first 2 seconds (so the macOS listen backlog doesn't overflow)
  if (__ITER === 0) sleep(Math.random() * 2);
  const n = pool > 0 ? Math.floor(Math.random() * pool) : Math.floor(Math.random() * 1e8);
  http.get(`http://${__ENV.HOST}${__ENV.ENDPOINT}/${isbn(n)}`, { tags: { name: 'book-by-isbn' } });
}

export function handleSummary(data) { return { stdout: summaryLine(data) }; }

The first attempt had no sleep line: with 500 connections opening at once, 0.63 percent of requests got "connection refused" (most likely the macOS listen backlog limit). Every new ISBN in the URL (Uniform Resource Locator) also opened a separate metric series in k6; the name tag folds them into one. Both lines fix the measurement, not the application. The first two runs compare IDistributedCache code that doesn't cache null against HybridCache:

app.MapGet("/negative/books/{isbn}", async (string isbn, CatalogDb db, HybridCache cache, CancellationToken ct) =>
{
    BookDto? book = await cache.GetOrCreateAsync($"v1:book:{isbn}",
        async token => await Books.LoadAsync(db, isbn, token),
        Ttl.WithJitter(bookTtl, ttlJitter), cancellationToken: ct);
    return book is null ? Results.NotFound() : Results.Ok(book);
});

$ ./run_pen.sh /naive/books 5000
== /naive/books POOL=5000
reqs=50000 rps=15317 p50=15.8ms p95=67.5ms failed=0.00%
db queries: {"queries":50000}
redis keys: 0
$ ./run_pen.sh /negative/books 5000
== /negative/books POOL=5000
reqs=50000 rps=25016 p50=8.9ms p95=18.4ms failed=0.00%
db queries: {"queries":5000}
redis keys: 5000
$ ./run_pen.sh /negative/books 0
== /negative/books POOL=0
reqs=50000 rps=12462 p50=17.4ms p95=102.1ms failed=0.00%
db queries: {"queries":49980}
redis keys: 49980

The first observation may come as a surprise: HybridCache also caches the null the factory returns. The payload of the v1:book:9782123456786 key in Redis ended with null, and the key took 144 bytes. So negative caching, caching the fact that "this record doesn't exist", comes for free, and for the bot cycling through the same 5,000 ISBNs 50,000 queries dropped to 5,000. The second half of the second guess has a bitter answer, though: against a bot that makes up a new ISBN per request, 49,980 queries went through (20 ISBNs repeated by chance) and 49,980 useless keys were written to Redis. The attacker now fills both the database and the cache's memory.

Validation, a Bloom Filter and a Short Negative TTL

The defense has three layers: let a malformed key go nowhere, don't even ask the cache about a key that certainly doesn't exist, and cache the small remainder as negative for a short time:

app.MapGet("/api/books/{isbn}", async (string isbn, CatalogDb db, HybridCache cache, BloomFilter filter, CancellationToken ct) =>
{
    // 1. format: a malformed key reaches neither the cache nor the database
    if (!Isbn13.IsValid(isbn)) return Results.BadRequest(new { error = "Invalid ISBN-13" });

    // 2. existence filter: if it says "definitely not", there's no need to even check the cache
    if (!filter.MightContain(isbn)) return Results.NotFound();

    // 3. negative caching: a key that slips through as a false positive stays cached as "missing" briefly
    bool loaded = false;
    BookDto? book = await cache.GetOrCreateAsync($"v1:book:{isbn}", async token =>
    {
        loaded = true;
        return await Books.LoadAsync(db, isbn, token);
    }, Ttl.WithJitter(bookTtl, ttlJitter), cancellationToken: ct);

    if (book is null)
    {
        // in .NET 10 the TTL can't be changed from inside the factory; rewrite the "missing" result with a short TTL
        if (loaded) await cache.SetAsync<BookDto?>($"v1:book:{isbn}", null, negativeOptions, cancellationToken: ct);
        return Results.NotFound();
    }
    return Results.Ok(book);
});

The first layer is Isbn13.IsValid: 13 digits, a 978 or 979 prefix and the check digit. Even for a random 13-digit number, nine out of ten fail on the check digit. Turning unvalidated input into a key is also the door to the memory DoS (Denial of Service) from the second post of the series. The loaded flag in the third layer works around a .NET 10 limitation: entry options are passed before the factory runs, so you can't say "if not found, make the TTL 60 seconds". The call that actually ran the factory rewrites the result with a short TTL; for the other callers waiting through coalescing the flag stays false, so there's no redundant write to Redis. .NET 11 adds overloads that allow setting options from inside the factory, covered in the tenth post of the series.

The second layer is a Bloom filter: a bit array that answers "definitely not" or "maybe" for a key, never gives a false negative and has a tunable false positive rate:

// Says "definitely not" or "maybe"; no false negatives, tunable false positive rate
public sealed class BloomFilter
{
    private readonly long[] _bits;
    private readonly int _bitCount;
    private readonly int _hashCount;

    public BloomFilter(int expectedItems, double falsePositiveRate)
    {
        double ln2 = Math.Log(2);
        _bitCount = (int)Math.Ceiling(-expectedItems * Math.Log(falsePositiveRate) / (ln2 * ln2));
        _hashCount = Math.Max(1, (int)Math.Round(_bitCount / (double)expectedItems * ln2));
        _bits = new long[(_bitCount + 63) / 64];
    }

    public int SizeInBytes => _bits.Length * sizeof(long);
    public int HashCount => _hashCount;

    public void Add(string key)
    {
        (uint h1, uint h2) = Hash(key);
        for (int i = 0; i < _hashCount; i++)
        {
            int bit = (int)((h1 + (uint)i * h2) % (uint)_bitCount);
            Interlocked.Or(ref _bits[bit >> 6], 1L << (bit & 63));
        }
    }

    public bool MightContain(string key)
    {
        (uint h1, uint h2) = Hash(key);
        for (int i = 0; i < _hashCount; i++)
        {
            int bit = (int)((h1 + (uint)i * h2) % (uint)_bitCount);
            if ((Volatile.Read(ref _bits[bit >> 6]) & (1L << (bit & 63))) == 0)
            {
                return false;
            }
        }
        return true;
    }

    // derive two hashes from a single 64-bit hash (Kirsch–Mitzenmacher)
    private static (uint, uint) Hash(string key)
    {
        ulong hash = XxHash3.HashToUInt64(Encoding.UTF8.GetBytes(key));
        return ((uint)hash, (uint)(hash >> 32) | 1);
    }
}

At startup the application adds every ISBN in the database to the filter; XxHash3 needs the System.IO.Hashing package. Sized for 100,000 items at a 1 percent target, the filter takes 117 KB and uses 7 hashes (startup log: Bloom filter: 117 KB, 7 hashes). The same attack, this time against the guarded endpoint:

$ ./run_pen.sh /api/books 0
== /api/books POOL=0
reqs=50000 rps=24994 p50=6.5ms p95=13.5ms failed=0.00%
db queries: {"queries":9}
redis keys: 9

Of 50,000 unique ISBNs, 9 passed the filter as false positives, went to the database and were cached as 60-second negative entries. The rate is far below 1 percent because the filter was sized for 100,000 but holds 50,000 ISBNs; the formula predicts 0.025 percent, about 12.5 in 50,000 requests. When a new book is added, it has to go into the filter as well:

$ curl -s localhost:17540/api/books/12345
{"error":"Invalid ISBN-13"} 400
$ curl -s localhost:17540/api/books/9790000000018
404
$ curl -s -X POST localhost:17540/api/books -H "Content-Type: application/json" -d '{"isbn":"9790000000018","title":"Caching in Practice","price":42.5,"categoryId":7}'
{"id":50001,"isbn":"9790000000018","title":"Caching in Practice","price":42.5} 201
$ curl -s localhost:17540/api/books/9790000000018
{"id":50001,"isbn":"9790000000018","title":"Caching in Practice","price":42.5} 200
$ curl -s localhost:17540/stats
{"queries":2}

The first 404 never touched the database; the two queries are the INSERT and the first read. After writing the record, the POST endpoint calls filter.Add and, for any earlier negative entry, cache.RemoveAsync. The multi-instance trap: the filter lives in-process; only the instance that added the book is updated, the others keep saying "definitely not". You need to rebuild the filter from the database periodically or keep a shared filter with Redis 8's built-in Bloom filter commands; the latter belongs to the upcoming Database › Redis series. To slow the attack itself down there's rate limiting, covered in the Web API series post "What's New in ASP.NET Core with .NET 7: MapGroup, TypedResults, Rate Limiting and Output Caching".

Avalanche: TTLs That Die Together

The most common cause of an avalanche is an innocent warm-up: at startup the application loads the 5,000 best-selling books into the cache, all written in the same second, all with the same TTL. Jitter adds a little randomness to each entry's TTL; the warm-up itself limits parallelism so it doesn't choke Redis:

public static class Ttl
{
    // a random ±spread on every entry: entries written together don't die together
    public static HybridCacheEntryOptions WithJitter(TimeSpan baseTtl, double spread)
    {
        TimeSpan ttl = baseTtl * (1 + (Random.Shared.NextDouble() * 2 - 1) * spread);
        return new HybridCacheEntryOptions { Expiration = ttl, LocalCacheExpiration = ttl };
    }
}

public static class CacheWarmer
{
    // load the best sellers at startup; at most 16 parallel writes so Redis isn't flooded
    public static async Task WarmAsync(CatalogDb db, HybridCache cache, int count, TimeSpan ttl, double jitter)
    {
        long start = Environment.TickCount64;
        List<BookDto> top = await db.Books.AsNoTracking()
            .OrderByDescending(b => b.Sold).ThenBy(b => b.Id).Take(count)
            .Select(b => new BookDto(b.Id, b.Isbn, b.Title, b.Price))
            .ToListAsync();

        await Parallel.ForEachAsync(top, new ParallelOptions { MaxDegreeOfParallelism = 16 }, async (book, ct) =>
            await cache.SetAsync<BookDto?>($"v1:book:{book.Isbn}", book, Ttl.WithJitter(ttl, jitter), cancellationToken: ct));

        Console.WriteLine($"Warm-up: {top.Count} books in {Environment.TickCount64 - start} ms (ttl {ttl.TotalSeconds}s, jitter {jitter:P0})");
    }
}

Expiration and LocalCacheExpiration get the same duration, otherwise L1 and L2 die separately. In the experiment the TTL is 60 seconds; k6 sends 2,000 requests per second to these 5,000 ISBNs for 100 seconds (constant-arrival-rate, a pool of 500 VUs), and a small BackgroundService writes the number of SQL (Structured Query Language) commands per second to a file. Two runs, one without jitter and one with 20 percent jitter (the %0 format comes from the machine's Turkish culture):

Warm-up: 5000 books in 116 ms (ttl 60s, jitter %0)
reqs=200003 rps=2000 p50=0.4ms p95=0.7ms failed=0.00%
Warm-up: 5000 books in 124 ms (ttl 60s, jitter %20)
reqs=200005 rps=2000 p50=0.4ms p95=0.8ms failed=0.00%
$ ./qps_compare.sh
second no-jitter jitter-20%
56 0 191
57 0 213
58 0 199
59 0 193
60 87 199
61 1627 192
62 1054 247
63 771 216
64 461 197
65 325 227
66 235 207
67 136 219
68 103 215
69 77 204
70 49 198
71 23 196
72 14 213
qps_jitter_0.log: peak 1627/s at 61s, total 5000
qps_jitter_0.2.log: peak 247/s at 62s, total 5018

In the run without jitter the database didn't see a single query for 59 seconds, then 1,627 queries arrived in second 61: almost every request in that second found the cache empty. With 20 percent jitter the TTLs spread between 48 and 72 seconds and the peak dropped to 247 queries per second. The total is nearly the same (5,000 and 5,018; the extra are keys reloaded with a short TTL that died once more within the window): jitter doesn't reduce the work, it spreads it over time. Don't be fooled by p95 staying under 1 ms; a local SQLite handled this peak easily, while on a real database the same peak fills the connection pool. Measure an avalanche by the database's per-second load.

Staying Up When Redis Is Unreachable

The second form of avalanche is the cache server itself going away. While a 100-VU scenario was reading across all books (50,000 keys, so L1 is mostly cold), we stopped Redis with docker stop. The Redis connection comes from a single multiplexer whose settings are read from configuration:

// one multiplexer: IDistributedCache, HybridCache's L2 and the lock share the same connection
ConfigurationOptions redisOptions = ConfigurationOptions.Parse(builder.Configuration["Redis"]!);
if (builder.Configuration.GetValue("RedisFailFast", false))
{
    // don't hold commands in the backlog while disconnected, fail immediately
    redisOptions.BacklogPolicy = BacklogPolicy.FailFast;
}
IConnectionMultiplexer redis = await ConnectionMultiplexer.ConnectAsync(redisOptions);
builder.Services.AddSingleton(redis);
builder.Services.AddStackExchangeRedisCache(o => o.ConnectionMultiplexerFactory = () => Task.FromResult(redis));
builder.Services.AddHybridCache();

Three runs: default settings (localhost:17550,abortConnect=false), short timeouts (connectTimeout=1000,syncTimeout=250,asyncTimeout=250) and short timeouts plus BacklogPolicy.FailFast. At the start of each run we send a single curl to the classic cache-aside endpoint:

== Redis stopped, config: localhost:17550,abortConnect=false
500 5.268785s
reqs=401 rps=17 p50=5961.0ms p95=6024.3ms failed=0.00%
db queries: {"queries":398}
== Redis stopped, config: localhost:17550,abortConnect=false,connectTimeout=1000,syncTimeout=250,asyncTimeout=250
500 0.335138s
reqs=2141 rps=103 p50=998.6ms p95=1009.1ms failed=0.00%
db queries: {"queries":2097}
== Redis stopped, config: localhost:17550,abortConnect=false,connectTimeout=1000,syncTimeout=250,asyncTimeout=250 --RedisFailFast=true
500 0.023238s
reqs=662423 rps=33120 p50=2.2ms p95=7.3ms failed=0.00%
db queries: {"queries":50000}

Classic cache-aside returned 500 in every configuration: the IDistributedCache exception bubbles straight up to the endpoint. HybridCache didn't fail in any run, because it treats an L2 read failure as a miss and goes to the database, and it logs a write failure and moves on. The log had these two entries for every request:

fail: Microsoft.Extensions.Caching.Hybrid.HybridCache[6]
      Cache backend read failure.
      StackExchange.Redis.RedisConnectionException: The message timed out in the backlog attempting to send because no connection became available (5000ms) - Last Connection Exception: SocketClosed on localhost:17550/Interactive, ...
fail: Microsoft.Extensions.Caching.Hybrid.HybridCache[7]
      Cache backend write failure.

No errors, but with default settings the application effectively stopped: p95 of 6 seconds, 17 requests per second. StackExchange.Redis's default BacklogPolicy keeps commands queued while disconnected and holds them until asyncTimeout expires (it defaults to syncTimeout, 5,000 ms). Lowering the timeout to 250 ms brought p50 to exactly 1 second, not 250 ms: according to the docs, async timeouts are checked during the heartbeat, once per second by default. With FailFast no command is queued at all; p95 dropped to 7.3 ms and the application, on L1 plus the database, went up to 33,120 requests per second. In a docker pause run, where the connection looks open but no reply comes back, the connection was marked unhealthy after the first timeouts and p95 was 6.7 ms.

FusionCache goes further here: SetFailSafe(true, ...) keeps serving an expired value for a while when the factory or L2 fails, and DistributedCacheSoftTimeout caps how long it waits for L2. The comparison is in the fourth post of the series.

In Production

Putting every number from this post in one table:

To see these in production, put the database query count next to the cache metrics. A stampede shows up as a short spike in query count that lines up with a deploy or a TTL boundary; penetration shows up as a rising miss rate while the cache entry count keeps growing too. How to collect hit ratio, miss and eviction metrics is the topic of the next post.

Two Redis settings deserve a conscious choice in every environment: abortConnect=false (so the application can start without Redis; the default is true outside Azure) and an asyncTimeout short enough for the workload. A short timeout also turns a few slow replies under normal load into errors; measure, then choose. HybridCache writes two fail entries per L2 failure: during an outage, 50,000 requests mean 100,000 log entries. Put a log filter on that category; log management is for the upcoming Logging series.

None of these fixes is free. Every request that misses the lock pays a 50 ms polling delay. The Bloom filter takes memory on every instance and has to be rebuilt. With jitter, compute acceptable staleness at the upper bound: with a 10-minute TTL and 20 percent jitter, a value can be 12 minutes old.

Common Pitfalls

Assuming HybridCache's stampede protection works across instances. It's per instance; a ten-pod deployment runs ten factories at once.

Doing negative caching with a long TTL. HybridCache keeps null with the same TTL as a positive value; a newly added book keeps returning 404 for 10 minutes if the write path doesn't invalidate. Keep the negative TTL short and delete the key on the write path.

Treating negative caching as the only defense against penetration. An attacker using unique keys turns it into a memory-exhaustion tool; format validation and an existence filter come first.

Setting the lock TTL shorter than the factory. If the factory takes 12 seconds and the lock drops after 10, a second instance starts computing and the lock loses its point.

Leaving the Redis connection on default timeouts. A cache that never errors but makes every request wait 5 seconds is harder to notice than one that fails.

A Small Experiment

★☆☆ Run run_stampede.sh three times for the classic cache-aside endpoint. How much does the query count vary between runs? What happens when you change the k6 scenario so every VU waits a random 0–2 seconds before its first request?

★★☆ In RedisSingleFlight, lower LockTtl to 1 second and add a 2-second Task.Delay to the report factory. How high does the total query count go, and which warning shows up in the log?

★★★ Run the Bloom filter on a second instance too, add a new book through one and read it from the other. See the 404, then write a BackgroundService that rebuilds the filter from the database every 5 minutes. How do you make sure requests arriving while the filter is being swapped don't get a wrong "missing"?

In this post we solved the three problems that break caches in production by measuring them, and saw the cost of each fix in numbers. The next post makes those numbers permanent: hit ratio, miss and eviction metrics, Redis's own statistics, proving that a cache actually pays off, and tests that cover expiration and invalidation.