IMemoryCache in Depth: SizeLimit, Eviction, Change Tokens and the Hidden Stampede in GetOrCreate

Version note: The code in this post was built and run with .NET 10; the examples ran with Microsoft.Extensions.Caching.Memory 10.0.14, Entity Framework Core (EF Core) 10.0.14 (SQLite) and Microsoft.Extensions.Caching.Hybrid 10.11.0.

Guess First

You've created a separate MemoryCache for Bookstore's catalog: SizeLimit = 100, CompactionPercentage = 0.15. You put 100 books in it, each with Size = 1; the first 10 are CacheItemPriority.Low, the last 5 are NeverRemove, the rest are Normal. Then you read books 1 to 20, so those are the most recently used. Now you add book 101 with High priority and immediately call TryGetValue("book:101").

Two questions. First: what does TryGetValue return, and how many entries are left in the cache 200 ms later? Which books are gone? Second: while the cache is empty, 100 concurrent requests call IMemoryCache.GetOrCreateAsync for the same key. How many times does the factory run?

Where We Pick Up

In the ASP.NET Core Web API (Application Programming Interface) series, "Caching in ASP.NET Core Web API: IMemoryCache, Redis and Response Caching" laid the groundwork for IMemoryCache: Set, TryGetValue, absolute and sliding expiration, removing the key when data changes. In the first post of this series we decided which data goes into which layer. This post goes into the internals of the lowest layer, the in-process L1 (level 1) cache: how to put a memory limit on it, who gets evicted when the limit is hit, when expiration actually runs, how to remove a group of entries in one go, and why GetOrCreateAsync is less safe than you think.

All examples live in a small console app running against Bookstore's SQLite database (200 authors, 20,000 books, 40 categories). The source code we read is MemoryCache.cs on the release/10.0 branch of .NET 10.

Your Own Cache Instance: Why SizeLimit Isn't Shared

SizeLimit has no unit. The cache can't measure how many bytes an object holds; you give each entry a Size, and the cache compares the sum with SizeLimit. The unit is your choice: number of entries, number of books, estimated bytes. There is only one rule: if there's a limit, every entry must specify a Size.

using MemoryCache cache = new(new MemoryCacheOptions { SizeLimit = 1_000 });

try
{
    cache.Set("book:1", "Dune");
}
catch (InvalidOperationException ex)
{
    Console.WriteLine($"{ex.GetType().Name}: {ex.Message}");
}

cache.Set("book:1", "Dune", new MemoryCacheEntryOptions { Size = 1 });
Console.WriteLine($"with Size=1 -> cached: {cache.TryGetValue("book:1", out _)}");

InvalidOperationException: Cache entry must specify a value for Size when SizeLimit is set.
with Size=1 -> cached: True

That rule makes it dangerous to put a limit on the shared IMemoryCache you get from AddMemoryCache(). The framework itself and the libraries you use share that instance, and none of them know your Size contract. The day you set the limit with AddMemoryCache(o => o.SizeLimit = ...), the first library that calls Set without a Size throws. Microsoft Learn says the same: if you want a limit, create your own cache singleton. Since .NET 8 the cleanest way to do that is a keyed service:

builder.Services.AddMemoryCache(); // leave the shared, unlimited cache used by the framework and libraries alone
builder.Services.AddKeyedSingleton<IMemoryCache>("catalog", (sp, _) => new MemoryCache(
    new MemoryCacheOptions
    {
        SizeLimit = 50_000,           // the unit is ours: number of book summaries in the cache
        CompactionPercentage = 0.10,  // free 10% when the limit is exceeded
        TrackStatistics = true
    },
    sp.GetRequiredService<ILoggerFactory>())); // so eviction callback errors reach the logs

app.MapGet("/api/categories/{id:int}/books", async (int id, int page, CatalogDb db,
    [FromKeyedServices("catalog")] IMemoryCache cache, CancellationToken ct) =>
{
    string key = $"v1:books:cat:{id}:page:{page}";
    if (cache.TryGetValue(key, out List<BookSummary>? books)) return books;

    books = await db.Books.AsNoTracking()
        .Where(b => b.CategoryId == id)
        .OrderBy(b => b.Id).Skip((page - 1) * 20).Take(20)
        .Select(b => new BookSummary(b.Id, b.Title, b.Price, b.Author!.Name))
        .ToListAsync(ct);

    cache.Set(key, books, new MemoryCacheEntryOptions
    {
        Size = books.Count,  // takes as much room as the number of books on the page
        AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10)
    });
    return books;
});

app.MapGet("/stats/catalog-cache", ([FromKeyedServices("catalog")] IMemoryCache cache) =>
    cache.GetCurrentStatistics());

After requesting two pages from three categories and repeating one of them:

$ curl -s localhost:17510/stats/catalog-cache
{"currentEntryCount":6,"currentEstimatedSize":120,"totalMisses":6,"totalHits":1}

Six pages of 20 books each, total size 120. Because we made Size the number of books on the page, SizeLimit = 50_000 means "at most 50,000 book summaries"; multiplying that by the average object size turns it into a memory budget. GetCurrentStatistics has been around since .NET 7 and only returns data when TrackStatistics = true. We'll see why the constructor gets an ILoggerFactory shortly, in the callbacks section.

When the Limit Is Hit: Rejected Entries, Priority and Compaction

Now the scenario from the guessing question. We attach a PostEvictionCallback to each entry so we can see who leaves and why:

MemoryCacheEntryOptions Options(CacheItemPriority priority) => new MemoryCacheEntryOptions
{
    Size = 1,
    Priority = priority
}.RegisterPostEvictionCallback((key, _, reason, _) =>
{
    lock (evicted) evicted.Add($"{key}({reason})");
});

for (int i = 1; i <= 100; i++)
{
    CacheItemPriority priority = i switch
    {
        <= 10 => CacheItemPriority.Low,
        > 95 => CacheItemPriority.NeverRemove,
        _ => CacheItemPriority.Normal
    };
    cache.Set($"book:{i}", $"Book {i}", Options(priority));
    await Task.Delay(1); // in real life entries are created at different moments
}

await Task.Delay(20);
// most recently read: Low priority 1-10 and Normal 11-20
for (int i = 1; i <= 20; i++) cache.TryGetValue($"book:{i}", out _);

cache.Set("book:101", "Book 101", Options(CacheItemPriority.High));
Console.WriteLine($"book:101 cached right after Set: {cache.TryGetValue("book:101", out _)}");

book:101 cached right after Set: False
after compaction: entries=85 size=85
evicted (16): book:1(Capacity), book:2(Capacity), book:3(Capacity), book:4(Capacity), book:5(Capacity), book:6(Capacity), book:7(Capacity), book:8(Capacity), book:9(Capacity), book:10(Capacity), book:21(Capacity), book:22(Capacity), book:23(Capacity), book:24(Capacity), book:25(Capacity), book:101(Capacity)
book:102 cached: True

The answer to the guess: False. An entry that would exceed the limit never gets in; High priority doesn't save it, and its own callback fires with reason Capacity. In the source, when UpdateCacheSizeExceedsCapacity returns true the entry is marked with EvictionReason.Capacity and an "overcapacity compaction" is queued on the thread pool. That compaction brings the cache down to SizeLimit × (1 − CompactionPercentage), which is 85 here. The Low bucket goes first: 1–10 leave first even though they were read most recently, because priority comes before LRU (Least Recently Used). Then the entries with the oldest access in the Normal bucket: 11–20 stayed because we had just read them, 21–25 went. Nobody touched the NeverRemove entries 96–100. Book 102 fits now, so it's in the cache.

Two practical consequences follow. First, when the limit is full, the newest data doesn't get cached on its first request; the default CompactionPercentage is 0.05, so in a large cache each overflow frees only 5 percent, and rejections are common in a full cache. Second, Microsoft Learn lists the compaction order as "expired, priority, LRU, earliest absolute expiration, earliest sliding expiration", but in the .NET 10 source the sort inside a bucket is by LastAccessed only; the last two steps exist in the code only as a to-do note (TODO). If you want to free space by hand, MemoryCache.Compact(0.25) runs the same algorithm based on entry count.

Expiration Is Not a Timer

The second surprise is when an expired entry is actually removed. We put two prices in with a 1-second absolute expiration and don't touch the cache for 3 seconds:

using MemoryCache cache = new(new MemoryCacheOptions
{
    ExpirationScanFrequency = TimeSpan.FromSeconds(1)
});
Stopwatch clock = Stopwatch.StartNew();
void Log(string message) => Console.WriteLine($"[{clock.Elapsed.TotalSeconds:0.0}s] {message}");

MemoryCacheEntryOptions options = new MemoryCacheEntryOptions()
    .SetAbsoluteExpiration(TimeSpan.FromSeconds(1))
    .RegisterPostEvictionCallback((key, _, reason, _) => Log($"evicted {key}: {reason}"));

cache.Set("price:42", 19.90m, options);
cache.Set("price:43", 24.50m, options);
Log("set price:42 and price:43 with 1 s absolute expiration");

await Task.Delay(3000);
Log("3 s passed without touching the cache");

cache.TryGetValue("menu", out _);
Log("read an unrelated key");

[0.0s] set price:42 and price:43 with 1 s absolute expiration
[3.0s] 3 s passed without touching the cache
[3.0s] read an unrelated key
[3.0s] evicted price:43: Expired
[3.0s] evicted price:42: Expired

Nothing happened at the second mark. MemoryCache has no background timer; every operation such as Set, TryGetValue or Remove checks "has ExpirationScanFrequency passed since the last scan?" and, if so, hands the scan to a task. The default scan interval is 1 minute; we lowered it to 1 second. If you read an expired key directly you get a miss anyway, so correctness isn't the issue. The issue shows up in two places: on a low-traffic instance, expired entries keep sitting in memory and in the Size total, and code that uses the callback to "refresh exactly when it expires" doesn't fire when it expects to. The callback itself runs on the thread pool; the cache catches any exception thrown inside it and writes "EvictionCallback invoked failed" to its own logger. An instance created with new MemoryCache(options) gets its logger from NullLoggerFactory, so that error silently disappears. That's why the keyed registration passes an ILoggerFactory.

Group Invalidation with Change Tokens

When a new book is added to category 3, every page of that category should become invalid. You can't remove the keys one by one without knowing the page count, and IMemoryCache doesn't offer removal by prefix. The solution is to give each group a CancellationTokenSource and tie the entries to that token:

// one CancellationTokenSource per group to invalidate all of the group's entries in one go
public sealed class CacheGroups
{
    private readonly ConcurrentDictionary<string, CancellationTokenSource> _sources = new();

    public IChangeToken GetToken(string group) =>
        new CancellationChangeToken(_sources.GetOrAdd(group, _ => new CancellationTokenSource()).Token);

    public void Invalidate(string group)
    {
        // remove from the dictionary, then cancel: the next GetToken gets a fresh source
        if (_sources.TryRemove(group, out CancellationTokenSource? source))
        {
            source.Cancel();
        }
    }
}

public async Task<List<BookSummary>> GetPageAsync(int categoryId, int page, CancellationToken ct = default)
{
    string key = $"v1:books:cat:{categoryId}:page:{page}";
    if (cache.TryGetValue(key, out List<BookSummary>? cached)) return cached!;

    // take the token BEFORE the query: if invalidation happens during the load, this entry never gets cached
    IChangeToken groupToken = groups.GetToken($"category:{categoryId}");

    await using CatalogDb db = CatalogDb.Create();
    List<BookSummary> books = await db.Books.AsNoTracking()
        .Where(b => b.CategoryId == categoryId)
        .OrderBy(b => b.Id).Skip((page - 1) * 20).Take(20)
        .Select(b => new BookSummary(b.Id, b.Title, b.Price, b.Author!.Name))
        .ToListAsync(ct);

    cache.Set(key, books, new MemoryCacheEntryOptions()
        .SetAbsoluteExpiration(TimeSpan.FromMinutes(10))
        .AddExpirationToken(groupToken)
        .RegisterPostEvictionCallback((k, _, reason, _) => Console.WriteLine($"  evicted {k}: {reason}")));
    return books;
}

cached entries: 6, queries: 6
page 1 of category 3 still cached: False
page 1 of category 4 still cached: True
  evicted v1:books:cat:3:page:3: TokenExpired
  evicted v1:books:cat:3:page:1: TokenExpired
  evicted v1:books:cat:3:page:2: TokenExpired
cached entries: 3

The read right after Cancel is a miss, because TryGetValue checks whether the entry's token HasChanged. The physical removal and the callbacks happen a few milliseconds later, on a task started by the ExpirationTokensExpired method in the source.

The most important line in the code is the commented one. If you took the token after the query, this sequence would be possible: the request reads old data, right then an admin adds a book and invalidates the group, the request picks up the newly created token and attaches the old data to it, and the old page stays in the cache for 10 minutes. Taking the token first means that if invalidation happens during the query, the token is already canceled, and Set treats the entry as expired and doesn't cache it. This is the in-process version of the "stale value written back" race we'll look at in detail in the seventh post of the series.

One more note: before .NET 7, nested entries created inside a factory automatically inherited the outer entry's tokens and expiration. Now MemoryCacheOptions.TrackLinkedCacheEntries defaults to false; set up the dependency explicitly, as above.

The Hidden Stampede in GetOrCreateAsync

The System Design series post "Caching Strategies: Cache-Aside, Write-Through, Write-Behind and Cache Stampede" covered the theory of stampedes. The question here is narrower: what does .NET's most used cache method do about it? The source is five lines: if TryGetValue fails, create an entry, await the factory, write the value. No lock anywhere in between. For the factory we use a query that ranks authors by sales, and we release 100 requests at once while the cache is empty:

private static async Task<List<AuthorSales>> LoadTopAuthorsAsync(CancellationToken ct)
{
    Interlocked.Increment(ref _factoryRuns);
    await Task.Delay(20, ct); // network latency: a SQL Server/PostgreSQL driver releases the thread meanwhile
    await using CatalogDb db = CatalogDb.Create();
    return await db.Books.AsNoTracking()
        .GroupBy(b => b.AuthorId)
        .Select(g => new { AuthorId = g.Key, Books = g.Count(), Sold = g.Sum(b => b.Sold) })
        .OrderByDescending(x => x.Sold).Take(10)
        .Select(x => new AuthorSales(x.AuthorId, x.Books, x.Sold))
        .ToListAsync(ct);
}

await MeasureAsync("GetOrCreateAsync", () => cache.GetOrCreateAsync(Key, entry =>
{
    entry.SetOptions(Options);
    return LoadTopAuthorsAsync(CancellationToken.None);
}));

Task.Delay(20) imitates a real network driver. Microsoft.Data.Sqlite's async methods actually run synchronously; without the delay, the same measurement ran the factory 10 times, which is the number of cores on the machine. In a database you talk to over the network, such as PostgreSQL, the thread is released while the query waits on the network, and the number approaches the number of concurrent requests. We try two fixes. The first is a lock, but a dictionary holding one SemaphoreSlim per key grows with every new key. A fixed number of "stripes" keeps memory bounded:

// a fixed number of "stripes" instead of one semaphore per key: memory doesn't grow without bound
public sealed class StripedLocks(int stripes = 64)
{
    private readonly SemaphoreSlim[] _locks = [.. Enumerable.Range(0, stripes).Select(_ => new SemaphoreSlim(1, 1))];

    public SemaphoreSlim For(string key) => _locks[(key.GetHashCode() & int.MaxValue) % _locks.Length];
}

public static class MemoryCacheSingleFlightExtensions
{
    public static async Task<T> GetOrCreateOnceAsync<T>(
        this IMemoryCache cache, string key, Func<CancellationToken, Task<T>> factory,
        MemoryCacheEntryOptions options, StripedLocks locks, CancellationToken ct = default)
    {
        if (cache.TryGetValue(key, out T? value)) return value!;

        SemaphoreSlim gate = locks.For(key);
        await gate.WaitAsync(ct);
        try
        {
            // one of the waiters may already have written the value: check again
            if (cache.TryGetValue(key, out value)) return value!;

            value = await factory(ct);
            cache.Set(key, value, options);
            return value;
        }
        finally
        {
            gate.Release();
        }
    }
}

The second is the often-repeated advice to "put a Lazy<Task<T>> in the cache": the entry's value is not the result but the task that produces it. All four side by side:

GetOrCreateAsync       requests=100 factory runs=100 db queries=100 elapsed= 252.7 ms
GetOrCreateOnceAsync   requests=100 factory runs=  1 db queries=  1 elapsed=  26.2 ms
Lazy<Task> in cache    requests=100 factory runs= 11 db queries= 11 elapsed=  45.1 ms
HybridCache            requests=100 factory runs=  1 db queries=  1 elapsed=  37.5 ms

The answer to the second guess: 100 requests, 100 factory runs, 100 queries. Microsoft Learn states it plainly: GetOrCreate and GetOrCreateAsync aren't atomic, every request that misses the same key runs the factory, and the last value written is the one that stays. The striped lock brings it down to 1. Lazy<Task> narrows the window but doesn't close it: because GetOrCreate itself isn't atomic, every thread that misses at the same moment creates its own Lazy and starts its own factory. On top of that, if the factory fails, the faulted task stays in the cache until you Remove it and hands the same exception to everyone.

The striped lock has costs too. Different keys that land on the same stripe wait for each other, and the lock only applies within this process: on three instances the factory runs three times. A single computation across instances needs a distributed lock; we discussed why that alone isn't enough in the Distributed Systems Fundamentals series post "Distributed Locks, Leases and Fencing Tokens: Why a Lock Is Not Enough". HybridCache gives you the same protection out of the box and also handles timeouts, cancellation and L2 (level 2, the shared out-of-process cache), so in new code switching to it is better than writing your own lock. The reason HybridCache looks 11 ms slower is interesting too: for a mutable type it gives each caller a copy. More on that in the fourth post of the series.

Memory Pressure: Entity Graphs and Shared Objects

IMemoryCache stores a reference to the object, not the object itself. That causes two separate problems. The first is what comes along with what you cache. Suppose the author's book list was loaded earlier in the same scope, and then the book detail is fetched with Include and put in the cache:

await using CatalogDb db = CatalogDb.Create();
// the author's book list was loaded earlier in the same scope (e.g. an "other books by this author" section)
List<Book> authorBooks = await db.Books.Where(b => b.AuthorId == 7).ToListAsync();
Book book = await db.Books.Include(b => b.Author).FirstAsync(b => b.Id == 650);
cache.Set("book:650", book, TimeSpan.FromMinutes(10));

We measured the GC.GetTotalMemory(forceFullCollection: true) difference before and after removing the entry; the second line is a BookSummary record for the same book, fetched with a projection:

entity : book.Author.Books.Count = 100, freed on Remove: 92 KB
summary: BookSummary { Id = 650, Title = Book 650, Price = 10.0, AuthorName = Author 7 }, freed on Remove: 1 KB

You cached one book, and 100 books stayed in memory. The change tracker's navigation fixup filled the book.Author.Books collection with every book loaded in the same context, and all of them, descriptions included, live on through the cache's reference. The mechanics of fixup are in the EF Core series post "Change Tracking in EF Core: Entity States, AsNoTracking and Identity Resolution". If you use lazy-loading proxies it gets worse: the entity also holds a reference to the DbContext that loaded it. The rule is simple: put a DTO (Data Transfer Object) or record carrying only the fields you need in the cache, not an entity.

The second problem is everyone sharing the same instance. An endpoint filters the list it got from the cache in place, for its own purposes:

List<BookSummary> cheap = await GetCategoryAsync();
cheap.RemoveAll(b => b.Price >= 30);

request 1 sees 500 books
request 2 filtered in place to 200 cheap books
request 3 sees 200 books, same instance: True

The third request lost 300 books, and the bug will be served to everyone until the TTL (Time to Live) runs out. The records can't be changed individually, but the List<T> holding them can. Put an ImmutableArray<BookSummary> produced by ToImmutableArray() in the cache, or at least an array behind IReadOnlyList<T>; code that needs to change it should copy it first.

Production Notes

Don't run a cache without measuring it. TrackStatistics = true and GetCurrentStatistics() give you hits, misses, entry count and estimated size; exporting them as metrics is the topic of the ninth post of the series, and the built-in metrics coming with .NET 11 are the topic of the tenth. Don't read the Count property on the request path: Microsoft Learn notes that it's expensive under high concurrency because it aggregates internal partitions.

Derive SizeLimit from a memory budget. Measure the average entry size once (the GC.GetTotalMemory difference above is a good enough approximation) and divide the share of the pod's memory limit you allocate to the cache by it. Give Size in the same unit everywhere; if you use entry count in one place and bytes in another, the limit becomes meaningless.

Without a limit, the runtime won't shrink the cache for you. Microsoft Learn is explicit: ASP.NET Core doesn't trim the cache under memory pressure, and old options such as CompactOnMemoryPressure now produce a compile error. For an API running in a container, an unlimited cache is the first line of a story that ends with OOMKilled (Out Of Memory Killed).

Use NeverRemove sparingly. Those entries are never considered during compaction; if they take up most of SizeLimit, every other new entry is rejected because of the limit and the cache effectively becomes read-only.

Common Pitfalls

Building keys from user input. Picture an endpoint that caches search results under a v1:search:{q} key, and an attacker who sends a random q with every request. After 500,000 requests, with and without a limit:

// the attacker sends a different search term with every request: /api/search?q=<random>
for (int i = 0; i < 500_000; i++)
{
    string query = Guid.NewGuid().ToString("N");
    cache.Set($"v1:search:{query}", new int[64], new MemoryCacheEntryOptions
    {
        SlidingExpiration = TimeSpan.FromMinutes(5),
        Size = 1
    });
}

SizeLimit=none: entries=500,000 retained=268 MB
SizeLimit=10000: entries=9,500 retained=10 MB

The unlimited cache grew to 268 MB, and because of the sliding expiration no entry dies as long as the attack goes on; this is a memory DoS (Denial of Service). The limit saves memory but isn't enough on its own: the attacker's entries push your hot keys out through compaction and the hit ratio collapses. Normalize and constrain user input before it goes into a key (length, allowed values), reduce parameters like page and sort order to a fixed set, and add rate limiting to the endpoint if needed (the Web API series post "What's New in ASP.NET Core with .NET 7: MapGroup, TypedResults, Rate Limiting and Output Caching").

Putting SizeLimit on the shared IMemoryCache. We saw it above: the first library that doesn't pass a Size throws. A limited cache should always be your own instance.

Using GetOrCreateAsync with an expensive factory and thinking "it's cached anyway". On a cold start, after a deploy and whenever a popular key expires, the factory runs as many times as there are concurrent requests.

Taking the change token after reading the data. If invalidation and loading overlap, the old data gets attached to the new token and stays for the whole TTL.

Mutating an object that came from the cache. The change affects every request immediately, and the cause never shows up in the logs.

A Small Experiment

★☆☆ In the capacity example, set CompactionPercentage back to the default 0.05 and add 20 books in a row, from 101 to 120. How many of them got into the cache, and how many compactions were triggered? Work it out from the callback list.

★★☆ In the expiration example, leave ExpirationScanFrequency at its default (1 minute) and, after 3 seconds, read price:42 instead of an unrelated key. Which callbacks run, and when does the callback for price:43 arrive?

★★★ Add error handling to GetOrCreateOnceAsync: when the factory throws, what should the 99 requests waiting on the lock do? Should each of them retry the factory, or should they all get the same error? Implement both behaviors and compare the query counts with 100 concurrent requests.

In this post we saw the limits of the in-process cache: the limit is your responsibility, expiration isn't a timer, group invalidation is built with tokens, and GetOrCreateAsync has no protection against stampedes. But this cache lives in each instance's own memory and resets when the app restarts. In the next post we move up to the shared layer: running Redis as a cache with IDistributedCache, serialization, key design, TTL and maxmemory-policy.