HybridCache in Depth: L1/L2, Tags, Logical Invalidation and L1 Staleness Across Servers

Version note: The code in this post was built and run with .NET 10; the examples ran on Microsoft.Extensions.Caching.Hybrid 10.11.0, Microsoft.Extensions.Caching.StackExchangeRedis 10.0.14, ZiggyCreatures.FusionCache 2.9.0 and Redis 8.2.

Guess First

Bookstore's storefront API (Application Programming Interface) runs on two instances: A on port 17521 and B on 17522. Both use the same Redis as their L2 (level 2, a shared out-of-process cache), and each has its own L1 (level 1, an in-process cache) in memory. The category menu sits in HybridCache with the category-menu tag; its lifetime in Redis is 30 minutes. An admin renames the "Fiction" category to "Novels" through A, and the endpoint calls RemoveByTagAsync("category-menu").

Two questions. First: with the L1 lifetime also at 30 minutes, which name does a request to B return right after the change? Second: we lowered the L1 lifetime (LocalCacheExpiration) to 10 seconds, waited 22 seconds after the change, and A received no requests in the meantime. Which name does B return now?

Expiration, LocalCacheExpiration and the Defaults

The ASP.NET Core Web API series post "What's New in .NET 9: Built-in OpenAPI, Scalar and HybridCache in ASP.NET Core" introduced GetOrCreateAsync, stampede protection and the two layers, and the first post of this series cached and measured the menu with this API. Instead of repeating the basics, here we look at how the settings are actually resolved. The registration is short:

builder.Services.AddStackExchangeRedisCache(o =>
{
    o.Configuration = redis;
    o.InstanceName = "bookstore:";
});
builder.Services.AddHybridCache();

HybridCacheEntryOptions menuOptions = new()
{
    Expiration = expiration,      // lifetime in L2; 30 minutes in the example
    LocalCacheExpiration = local  // lifetime in L1; comes from the command line
};

app.MapGet("/api/categories/menu", async (CatalogDb db, HybridCache cache, FactoryCounter counter, CancellationToken ct) =>
{
    CategoryMenu menu = await cache.GetOrCreateAsync(
        "v1:category-menu",
        async token =>
        {
            counter.Increment();
            return await MenuQuery.LoadAsync(db, instance, token);
        },
        menuOptions,
        tags: ["category-menu"],
        cancellationToken: ct);
    return new { servedBy = instance, menu.BuiltBy, first = menu.Items[0].Name, factoryCalls = counter.Calls };
});

The response carries two traces: servedBy shows which instance handled the request, BuiltBy shows whose factory produced the menu. There is also a per-instance factory counter. With these three values we can tell from the outside which layer the cache answered from.

Three rules about lifetimes are spelled out in the source code of DefaultHybridCache. If you configure nothing, Expiration is 5 minutes. A LocalCacheExpiration longer than Expiration is cut down to Expiration; L1 never outlives L2. The third is the most surprising: when you pass per-entry options, that object is not merged property by property with DefaultEntryOptions. If you set only Expiration, the L1 lifetime becomes the same value. We tried it in a small console app with the defaults set to "1 hour in Redis, 1 second in memory":

services.AddHybridCache(o =>
{
    o.DefaultEntryOptions = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromHours(1),
        LocalCacheExpiration = TimeSpan.FromSeconds(1)
    };
});

// per-entry options replace the defaults instead of merging with them
int defaults = 0, custom = 0;
HybridCacheEntryOptions onlyExpiration = new() { Expiration = TimeSpan.FromHours(1) };
for (int i = 0; i < 2; i++)
{
    await cache.GetOrCreateAsync("with-defaults", _ => { defaults++; return ValueTask.FromResult(1); });
    await cache.GetOrCreateAsync("with-custom", _ => { custom++; return ValueTask.FromResult(1); }, onlyExpiration);
    await Task.Delay(1500);
}
Console.WriteLine($"factory calls after 3 s: defaults={defaults}, custom={custom}");

factory calls after 3 s: defaults=2, custom=1

The key that used the defaults dropped out of L1 after 1.5 seconds and the factory ran again. The key with only Expiration got a one-hour L1 lifetime. When a teammate says "I set L1 to 1 second globally", remember that every call passing its own options is outside that rule.

HybridCacheEntryFlags lets you switch individual behaviours off. The most useful ones:

Two limits also fail silently. MaximumPayloadBytes defaults to 1 MB (megabyte) and MaximumKeyLength to 1024 characters; a payload or key over the limit is logged but not cached. We imitated an endpoint that puts the user's search text into the key:

int longKeyCalls = 0;
string longKey = "v1:search:" + new string('x', 1100);
for (int i = 0; i < 2; i++)
    await cache.GetOrCreateAsync(longKey, _ => { longKeyCalls++; return ValueTask.FromResult(1); });
Console.WriteLine($"long key factory calls: {longKeyCalls}");

long key factory calls: 2
fail: Microsoft.Extensions.Caching.Hybrid.HybridCache[5] Cache key maximum length exceeded (maximum: 1024, actual: 1110).
fail: Microsoft.Extensions.Caching.Hybrid.HybridCache[5] Cache key maximum length exceeded (maximum: 1024, actual: 1110).

No exception, the request succeeds; the cache simply never works. Because the log level is Error, wiring this line to an alert is easy, as long as someone looks. Don't put long, user-supplied input into a key; hash it or don't cache it at all.

Instance Reuse: sealed and [ImmutableObject(true)]

L1 is not a plain IMemoryCache. For a mutable type, HybridCache produces a fresh copy from the serialized bytes on every call, because if one caller changes the object the others must not see it. If the type is sealed and marked [ImmutableObject(true)], it can hand the same instance to everyone:

// sealed + [ImmutableObject(true)]: HybridCache can hand the same instance to every caller
[ImmutableObject(true)]
public sealed record CategoryMenu(IReadOnlyList<MenuItem> Items, string BuiltBy, DateTimeOffset BuiltAt);

FrozenMenu f1 = await cache.GetOrCreateAsync("frozen", _ => ValueTask.FromResult(new FrozenMenu("Fiction")));
FrozenMenu f2 = await cache.GetOrCreateAsync("frozen", _ => ValueTask.FromResult(new FrozenMenu("Fiction")));
PlainMenu p1 = await cache.GetOrCreateAsync("plain", _ => ValueTask.FromResult(new PlainMenu("Fiction")));
PlainMenu p2 = await cache.GetOrCreateAsync("plain", _ => ValueTask.FromResult(new PlainMenu("Fiction")));
Console.WriteLine($"FrozenMenu same instance: {ReferenceEquals(f1, f2)}");
Console.WriteLine($"PlainMenu same instance:  {ReferenceEquals(p1, p2)}");

FrozenMenu same instance: True
PlainMenu same instance:  False

PlainMenu is also a record, so in practice it is immutable; but HybridCache can't know that and deserializes it on every L1 hit. For a large, frequently read menu that means CPU (Central Processing Unit) time and allocations wasted on every request. There is a price too: the attribute is a promise. If you later add an item to a List inside a marked type, every request holding that same instance sees it.

Tags: A Timestamp, Not a Delete

RemoveByTagAsync does not physically delete anything. Microsoft Learn describes it as "ignore anything created before this point"; in the source code the mechanism is simple: the current timestamp for the tag is written both to an in-process dictionary and to L2. Every read compares the entry's creation time with the timestamps of the entry's tags and treats older entries as invalid. Our admin endpoint:

// The only way to change the menu: database first, then invalidation (?by=key or the default tag)
app.MapPut("/api/admin/categories/{id:int}", async (int id, string? by, RenameCategory body, CatalogDb db, HybridCache cache, CancellationToken ct) =>
{
    int updated = await db.Categories.Where(c => c.Id == id)
        .ExecuteUpdateAsync(s => s.SetProperty(c => c.Name, body.Name), ct);
    if (updated == 0) return Results.NotFound();
    if (by == "key")
        await cache.RemoveAsync("v1:category-menu", ct);
    else
        await cache.RemoveByTagAsync("category-menu", ct);
    return Results.NoContent();
});

We looked at Redis before and after the invalidation:

$ docker exec chwork-redis-b redis-cli --scan
bookstore:v1:category-menu
$ curl -s -X PUT localhost:17521/api/admin/categories/1 -H 'Content-Type: application/json' -d '{"name":"Novels"}' -w '%{http_code}'
204
$ docker exec chwork-redis-b redis-cli --scan
bookstore:__MSFT_HCT__category-menu
bookstore:v1:category-menu
$ docker exec chwork-redis-b redis-cli TTL bookstore:v1:category-menu
1800
$ docker exec chwork-redis-b redis-cli TTL bookstore:__MSFT_HCT__category-menu
86400000

The menu key is still there, at the start of its 1800-second life. A tag record was added next to it, and its TTL (Time to Live) is 86,400,000 seconds, the 1000 days hard-coded in the source. The nice part of this design is that a single small write invalidates ten thousand entries attached to a tag. It has two costs: invalid values keep taking space in Redis and in L1 until they expire, and every instance has to learn about the timestamp somehow. RemoveByTagAsync("*") works the same way; the only difference is that the timestamp applies to every tag.

Two Instances, One Redis: Watching L1 Go Stale

We started two instances from the same build with different ports and names:

$ dotnet Ch04.Api.dll --urls http://localhost:17521 --Instance A --Cache:LocalSeconds 1800
$ dotnet Ch04.Api.dll --urls http://localhost:17522 --Instance B --Cache:LocalSeconds 1800

$ curl -s localhost:17521/api/categories/menu
{"servedBy":"A","builtBy":"A","first":"Fiction","factoryCalls":1}
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Fiction","factoryCalls":0}
$ curl -s -X PUT localhost:17521/api/admin/categories/1 -H 'Content-Type: application/json' -d '{"name":"Novels"}' -w '%{http_code}'
204
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Fiction","factoryCalls":0}
$ curl -s localhost:17521/api/categories/menu
{"servedBy":"A","builtBy":"A","first":"Novels","factoryCalls":2}
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Fiction","factoryCalls":0}

The first two lines show L2 at work: B never computed the menu, it took the copy A had written to Redis. After the invalidation A ran its factory again and returned "Novels"; B will keep returning "Fiction" for 30 minutes. That is the answer to the first question, and Microsoft Learn says it plainly: invalidation by key or tag affects the current server and L2, not the in-memory cache on other servers.

Is a Short LocalCacheExpiration Enough?

The first remedy that comes to mind is a shorter L1: let B's copy die after 10 seconds and fetch a fresh one from Redis. We ran the same scenario twice with --Cache:LocalSeconds 10. First, removal by key (?by=key):

$ curl -s -X PUT 'localhost:17521/api/admin/categories/1?by=key' -H 'Content-Type: application/json' -d '{"name":"Novels"}' -w '%{http_code}'
204
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Fiction","factoryCalls":0}
$ sleep 11
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"B","first":"Novels","factoryCalls":1}

As expected: B's L1 expired, the key was gone from Redis, B ran its own factory and returned "Novels". The inconsistency window is at most 10 seconds. Now the same thing with a tag:

$ curl -s -X PUT localhost:17521/api/admin/categories/1 -H 'Content-Type: application/json' -d '{"name":"Novels"}' -w '%{http_code}'
204
$ sleep 11
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Fiction","factoryCalls":0}
$ sleep 11
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Fiction","factoryCalls":0}
$ curl -s localhost:17521/api/categories/menu
{"servedBy":"A","builtBy":"A","first":"Novels","factoryCalls":2}
$ sleep 11
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"A","first":"Novels","factoryCalls":0}

The answer to the second question: "Fiction", even after 22 seconds. B's L1 expired twice, yet each time B accepted the old payload in Redis as valid. B only recovered once a request reached A and A rebuilt the menu, overwriting the copy in Redis.

The reason is in the source, in DefaultHybridCache.TagInvalidation.cs. Each instance keeps tag timestamps in a ConcurrentDictionary<string, Task<long>> and reads a tag from Redis only the first time it sees it; the record has no lifetime and is never read again. When B first got the menu, it read "never invalidated" (0) for category-menu and kept it. It never saw the timestamp A wrote to Redis later, and compared the creation time of the old payload in Redis with 0. For "*" it is even stricter: the global timestamp is read only in the constructor. We tried it: after calling RemoveByTagAsync("*") on A and waiting 11 seconds, B still returned "Fiction"; when we restarted B, it ran the factory on the first request and returned "Novels". There is an open issue in the dotnet/extensions repository proposing a lifetime for these records; until it lands, design around this behaviour.

The practical outcome with plain HybridCache: if you want to shorten the inconsistency window, also remove the entries whose keys you know with RemoveAsync alongside the tag, and bring LocalCacheExpiration down to the staleness you can accept. Use tags for broad groups whose keys you don't know, and accept that the upper bound of staleness for those groups is the Expiration in L2.

A Backplane: Same Code with FusionCache

The lasting fix is a channel that tells the other instances about a change on one instance right away: a backplane. Microsoft's implementation doesn't have one; FusionCache ships one on top of Redis pub/sub. Better still, FusionCache can register itself as HybridCache, so we change only the registration and leave the endpoints alone:

builder.Services.AddFusionCache()
    .WithDefaultEntryOptions(new FusionCacheEntryOptions
    {
        IsFailSafeEnabled = true,
        FailSafeMaxDuration = TimeSpan.FromHours(2),
        FailSafeThrottleDuration = TimeSpan.FromSeconds(30),
        FactorySoftTimeout = TimeSpan.FromMilliseconds(500)
    })
    .WithSerializer(new FusionCacheSystemTextJsonSerializer(new JsonSerializerOptions { IncludeFields = true }))
    .WithDistributedCache(new RedisCache(new RedisCacheOptions { Configuration = redis }))
    .WithBackplane(new RedisBackplane(new RedisBackplaneOptions { Configuration = redis }))
    .AsHybridCache();

The same scenario, with the L1 lifetime back at 30 minutes:

$ docker exec chwork-redis-b redis-cli PUBSUB CHANNELS
__Booksleeve_MasterChanged
FusionCache.Backplane:v2
$ curl -s -X PUT localhost:17521/api/admin/categories/1 -H 'Content-Type: application/json' -d '{"name":"Novels"}' -w '%{http_code}'
204
$ curl -s localhost:17522/api/categories/menu
{"servedBy":"B","builtBy":"B","first":"Novels","factoryCalls":1}
$ curl -s localhost:17521/api/categories/menu
{"servedBy":"A","builtBy":"B","first":"Novels","factoryCalls":1}
$ docker exec chwork-redis-b redis-cli --scan
v2:v1:category-menu
v2:__fc:t:category-menu

The first channel belongs to StackExchange.Redis itself, the second is the backplane. B treated its L1 copy as invalid because of the tag message A published, ran its factory and returned "Novels". The "builtBy":"B" in A's next response is interesting too: when B wrote the new value, the backplane notified A this time, and A picked up B's copy from L2. In FusionCache the tag record starts with __fc:t:; the mechanism is the same, the difference is that a message goes out to the other instances.

Two details in the registration also come from FusionCache. On our first attempt, without IncludeFields, we got this warning: "it looks like you are using a serializer based on System.Text.Json not correctly configured to handle value tuples". It comes from value tuple fields not being serialized with the default System.Text.Json settings; IncludeFields = true made it go away. The second is fail-safe. We cut the menu's Redis lifetime to 5 seconds, broke the table and sent the same request to both implementations:

$ sqlite3 bookstore.db 'ALTER TABLE Categories RENAME TO Categories_broken'
$ sleep 6
$ curl -s localhost:17521/api/categories/menu -w ' %{http_code}'
 500

$ sqlite3 bookstore.db 'ALTER TABLE Categories RENAME TO Categories_broken'
$ sleep 6
$ curl -s localhost:17521/api/categories/menu -w ' %{http_code}'
{"servedBy":"A","builtBy":"A","first":"Fiction","factoryCalls":2} 200

warn: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNPT7MU313PR] (O=0HNPT7MU313Q2 K=v1:category-menu): FAIL-SAFE activated (from memory)

The first is Microsoft's HybridCache: the expired value was gone, the factory threw, HTTP (Hypertext Transfer Protocol) 500. The second is FusionCache: the factory still ran and failed (the counter is 2), but the expired value was kept for FailSafeMaxDuration, so that was returned. FactorySoftTimeout does something similar: according to the docs, if the factory takes longer than 500 milliseconds and an old value is at hand, the old value is returned immediately and the factory finishes in the background. We'll measure both in detail in the eighth post of this series, in the context of keeping the app up while Redis and the database go down.

In Production

Write the L1 lifetime down as a consistency promise. "After the menu changes, an instance may show the old menu for at most 10 seconds" is what LocalCacheExpiration really means in a setup without a backplane; if you use tags, that sentence becomes "until the L2 lifetime". The price of a shorter value is more trips to Redis: one network round trip and one deserialization per L1 refill. For a small payload like the menu that is negligible; for large payloads, don't decide without measuring.

Reading tags has its own failure behaviour. In the source, the tag timestamp is read from Redis with a 4-second timeout, and if the read fails the tag is treated as "invalidated just now". So when Redis slows down, tagged entries stay on the safe side and fall through to the factory; expect database load to rise at the same moment.

A backplane has limits as well. Redis pub/sub messages are not durable; an instance whose connection drops misses whatever arrives in the meantime. The FusionCache docs describe an auto-recovery mechanism for this, but don't trust it unmeasured: a test where you cut one instance's Redis connection and bring it back is the first thing to run before going to production. Since every instance holds the same data in its own memory, the memory cost multiplies by the number of instances; use L1 only for genuinely hot keys.

Common Pitfalls

Believing RemoveByTagAsync frees space in Redis and in memory. Values live until they expire; invalidating millions of entries through a tag does not relieve memory pressure right away.

Passing per-entry options and assuming the LocalCacheExpiration in DefaultEntryOptions still applies. If you set only Expiration, L1 lives just as long; set both.

Marking a type [ImmutableObject(true)] and then mutating it. The attribute is not a check but a promise; the instance now belongs to every request.

Taking AddDistributedMemoryCache for an L2. The HybridCache constructor deliberately disables L2 when it sees this registration; on a single instance it makes no difference anyway, on several instances it means you have no shared L2.

Producing long or unbounded keys. A key over 1024 characters throws nothing, it is silently not cached; the only trace is an Error-level log line.

Treating RemoveByTagAsync("*") as a "clear everywhere" button in a multi-instance setup. In our test the other instance did not see the timestamp until it was restarted.

A Small Experiment

★☆☆ Start both instances with --Cache:LocalSeconds 5 and rename a category with ?by=key. Measure the inconsistency window with a loop that hits B every second; does it ever exceed 5 seconds?

★★☆ Change the admin endpoint to rebuild the menu with GetOrCreateAsync right after RemoveByTagAsync. In the tag scenario, when does B start returning "Novels"? Explain why this "warm up after invalidation" approach doesn't help for entries whose keys you don't know.

★★★ In FusionCache mode, add a 2-second Task.Delay to the factory, cut the menu's Redis lifetime to 5 seconds and measure the first request after expiry with curl -w '%{time_total}'. How does the result change with and without FactorySoftTimeout?

In this post we measured the consistency cost of the two in-app layers: tags are cheap but logical, and without a backplane L1 is each instance's own truth. In the next post we move the cache out of the server, to the client and the proxies in between: using Cache-Control, ETag, 304 Not Modified and Vary so that a request never reaches the server, or comes back without a body.