Controlling LLM Cost: DistributedCachingChatClient, Rate Limiting and Token Usage

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, OllamaSharp 5.5.2, Microsoft.Extensions.Caching.StackExchangeRedis 10.0.14, Redis 8.2 (Docker) and a local Ollama 0.34.4 (qwen3:4b).

Guess First

Bookstore's product page has the assistant write a one-sentence blurb for every book. You added UseDistributedCache() to the pipeline and put Redis behind it. Ada opens the "Refactoring" page, and a minute later grace opens the same page. How many requests reach the model? When is the entry written to Redis deleted?

And one more: you wrote a middleware that counts token usage and put it outside the cache. When Ada opens the page a second time, the answer comes from the cache in 3 milliseconds. How many tokens get written to the ledger?

In the cost post of LLM (Large Language Model) Fundamentals we did the math from tokens to the bill. In this post we turn that math into code: tokens never spent thanks to the cache, a per-user limit with rate limiting, and a daily budget per user and feature with UsageDetails.

UseDistributedCache and Redis

I ran Redis in its own container and pointed ASP.NET Core's IDistributedCache registration at it. When UseDistributedCache() gets no parameter, it takes the cache from DI (Dependency Injection):

docker run -d --name aiwork1-redis -p 19601:6379 redis:8.2

builder.Services.AddStackExchangeRedisCache(o =>
{
    o.Configuration = "localhost:19601";
    o.InstanceName = "bookstore-ai:";
});

IChatClient ollama = new OllamaApiClient(
    new HttpClient { BaseAddress = new Uri("http://localhost:11434"), Timeout = TimeSpan.FromMinutes(30) }, "qwen3:4b");

// Order: cache outside, accounting inside. An answer served from the cache spends no tokens and isn't charged to the budget.
builder.Services.AddChatClient(ollama)
    .UseDistributedCache()
    .Use((inner, services) => new UsageTrackingChatClient(
        inner,
        services.GetRequiredService<TokenLedger>(),
        services.GetRequiredService<IHttpContextAccessor>(),
        dailyLimit: 600));

The endpoint is simple; along with the answer it returns the duration and the token counts:

app.MapPost("/books/blurb", async (BlurbRequest request, IChatClient chat, CancellationToken ct) =>
{
    Stopwatch sw = Stopwatch.StartNew();
    try
    {
        ChatResponse response = await chat.GetResponseAsync(
            $"Write a one-sentence shop blurb for the book '{request.Title}'. Reply with the sentence only.",
            new ChatOptions { Temperature = 0, Seed = 42 }, ct);
        return Results.Ok(new
        {
            text = response.Text.Trim(),
            elapsedMs = sw.ElapsedMilliseconds,
            inputTokens = response.Usage?.InputTokenCount,
            outputTokens = response.Usage?.OutputTokenCount,
        });
    }
    catch (TokenBudgetExceededException ex)
    {
        return Results.Json(new { error = ex.Message }, statusCode: StatusCodes.Status429TooManyRequests);
    }
}).RequireRateLimiting("per-user");

Ada asks for the same book twice, then for two different books:

$ ada: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":93139,"inputTokens":31,"outputTokens":581}

$ ada: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":3,"inputTokens":31,"outputTokens":581}

$ ada: Clean Code
HTTP/1.1 429 Too Many Requests
{"error":"Daily token budget exceeded for 'ada': 612/600."}

$ ada: Domain-Driven Design
HTTP/1.1 429 Too Many Requests
{"error":"Daily token budget exceeded for 'ada': 612/600."}

$ GET /usage
[{"user":"ada","feature":"/books/blurb","tokens":612}]

The first request took 93 seconds: waiting in line on the shared Ollama plus qwen3:4b's thinking time. The second took 3 milliseconds; it never reached the model. Same question, same answer, thirty thousand times faster and zero tokens. But look at the two numbers in the output: the answer from the cache also says inputTokens: 31, outputTokens: 581. The cache stores the response together with its Usage; these numbers don't mean "this request spent this much" but "this much was spent when this answer was produced". That's the answer to the second opening question: had you put the accounting outside the cache, you'd have written another 612 tokens to the ledger for this answer and your bill would look twice as large as it is.

Look at the 581 as well. A one-sentence blurb is about 25 tokens; the rest are qwen3:4b's thinking (reasoning) tokens. With reasoning models these tokens are invisible to the user but are counted and billed as output tokens. This line shows most clearly what choosing a reasoning model for a simple task costs.

What Goes into the Cache Key?

Let's look at the entry in Redis:

$ redis-cli KEYS / TTL
bookstore-ai:3AF0B8848DAD0E650FEAF7FFA0AF4C18C0A613FB8DC8F63134B66D9FEFBFDA81695DE896D9ECCE976EEF751226DF8DA0 TTL=-1

$ grace: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":2,"inputTokens":31,"outputTokens":581}

After the InstanceName prefix, the key is a 96-hex-character hash, that is, 384 bits. According to Microsoft.Extensions.AI's API (Application Programming Interface) documentation, DistributedCachingChatClient computes this key from the JSON (JavaScript Object Notation) serialization of the messages, the ChatOptions and the values you add to CacheKeyAdditionalValues. In practice this means:

• A single character of difference in the messages (including changing the system prompt) means a new key; when you change the prompt, the old cache drops out by itself.
• Options fields such as Temperature, MaxOutputTokens and ModelId go into the key; two endpoints asking the same question with different settings don't share an entry.
• The HTTP (Hypertext Transfer Protocol) request itself, meaning the user's identity, headers or route, doesn't go into the key. Two users sending the same text read the same entry.
• The documentation says explicitly that the key isn't guaranteed to be stable across library versions; expect the cache to empty when you upgrade the package.
• If ChatOptions.ConversationId is set, the cache doesn't kick in at all; since the answer in a stateful conversation depends on its history, the default EnableCaching lets these requests pass straight through.

The third point calls for a decision. For a book blurb a shared cache is exactly what we want: when grace opens the same page she gets the answer Ada had generated, as the last request in the output shows. If the same pipeline also served "where are my orders?", one user could see another's orders whenever their texts matched. Either don't cache answers containing personal data at all (a separate keyed client without a cache) or separate them per user by adding the user's identity to CacheKeyAdditionalValues.

Entry Lifetime: TTL

The output above says TTL=-1: in Redis a TTL (Time To Live) of -1 means "no expiry". DistributedCachingChatClient writes entries without an expiry, and the package has no property to set one. The result: every distinct prompt stays in Redis forever; even if the blurb is updated, the old answer keeps coming back. I solved it at the IDistributedCache level, with a small decorator:

using Microsoft.Extensions.Caching.Distributed;

namespace Bookstore.Assistant.Api;

// DistributedCachingChatClient gives entries no expiry; add a default TTL to every write that has none
public sealed class ExpiringDistributedCache(IDistributedCache inner, TimeSpan ttl) : IDistributedCache
{
    public byte[]? Get(string key) => inner.Get(key);

    public Task<byte[]?> GetAsync(string key, CancellationToken token = default) => inner.GetAsync(key, token);

    public void Refresh(string key) => inner.Refresh(key);

    public Task RefreshAsync(string key, CancellationToken token = default) => inner.RefreshAsync(key, token);

    public void Remove(string key) => inner.Remove(key);

    public Task RemoveAsync(string key, CancellationToken token = default) => inner.RemoveAsync(key, token);

    public void Set(string key, byte[] value, DistributedCacheEntryOptions options) =>
        inner.Set(key, value, WithTtl(options));

    public Task SetAsync(string key, byte[] value, DistributedCacheEntryOptions options, CancellationToken token = default) =>
        inner.SetAsync(key, value, WithTtl(options), token);

    private DistributedCacheEntryOptions WithTtl(DistributedCacheEntryOptions options) =>
        options.AbsoluteExpiration is null && options.AbsoluteExpirationRelativeToNow is null && options.SlidingExpiration is null
            ? new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = ttl }
            : options;
}

builder.Services.AddChatClient(ollama)
    .Use((inner, services) => new DistributedCachingChatClient(
        inner, new ExpiringDistributedCache(services.GetRequiredService<IDistributedCache>(), TimeSpan.FromHours(1))))
    .Use((inner, services) => new UsageTrackingChatClient(
        inner,
        services.GetRequiredService<TokenLedger>(),
        services.GetRequiredService<IHttpContextAccessor>(),
        dailyLimit: 600));

$ grace: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":92127,"inputTokens":31,"outputTokens":581}

$ redis-cli TTL
bookstore-ai:3AF0B8848DAD0E650FEAF7FFA0AF4C18C0A613FB8DC8F63134B66D9FEFBFDA81695DE896D9ECCE976EEF751226DF8DA0 TTL=3600

grace's first request went to the model (92 seconds), and this time the entry was written with a TTL of 3600 seconds, that is, one hour. How you pick the duration depends entirely on the content: for a book blurb even a day is reasonable, for an answer containing stock or prices, minutes. The decorator only touches writes without an expiry; if other code that sets its own expiry uses the same IDistributedCache, its settings are kept.

Per-User Rate Limiting

A cache makes repeated questions cheap, but it doesn't stop a malicious or buggy client: a script asking a different question every time never uses the cache. The first line of defense is ASP.NET Core's rate limiting middleware. It's the same structure we met in the ASP.NET Core Web API series post ".NET 7 ile ASP.NET Core Yenilikleri" (What's New in ASP.NET Core with .NET 7), this time partitioned per user:

// Per-user request limit: 3 requests per minute
builder.Services.AddRateLimiter(o =>
{
    o.AddPolicy("per-user", http => RateLimitPartition.GetFixedWindowLimiter(
        http.Request.Headers["X-Demo-User"].ToString(),
        _ => new FixedWindowRateLimiterOptions { PermitLimit = 3, Window = TimeSpan.FromMinutes(1), QueueLimit = 0 }));

    o.OnRejected = async (context, ct) =>
    {
        context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
        if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out TimeSpan retryAfter))
        {
            context.HttpContext.Response.Headers.RetryAfter = ((int)retryAfter.TotalSeconds).ToString();
        }
        await context.HttpContext.Response.WriteAsJsonAsync(
            new { error = "Too many requests. Please wait before asking again." }, ct);
    };
});

On the first try this limit never kicked in, and the reason is instructive. Ada's first request took 93 seconds; the first one-minute fixed window ran out while that request was still running. The next three requests fell into a new window and didn't exceed the limit. A limit based on request count doesn't behave the way you expect when requests take this long. To see the limit, I had grace ask for the same cached book four times in a row:

$ grace: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":2,"inputTokens":31,"outputTokens":581}

$ grace: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":0,"inputTokens":31,"outputTokens":581}

$ grace: Refactoring
HTTP/1.1 200 OK
{"text":"Refactoring: Write better code by making small, incremental improvements that reduce technical debt and boost maintainability.","elapsedMs":0,"inputTokens":31,"outputTokens":581}

$ grace: Refactoring
HTTP/1.1 429 Too Many Requests
Retry-After: 60
{"error":"Too many requests. Please wait before asking again."}

Three requests came back from the cache within milliseconds, and the fourth was rejected with a 429 and a Retry-After: 60 header. Note: the rate limiter sits at the HTTP layer and counts a request that would have been answered from the cache too. Rejecting a request for which the model would never run may sometimes look unnecessary, but this layer limits abuse, not cost; what limits cost is the token budget in the next section.

Token Accounting: UsageDetails and a Daily Budget

Request count is a poor measure of cost: "Hello" and "summarize this 40-page order" count the same. What's really spent is tokens, and every ChatResponse reports them under Usage regardless of the provider. I keep a daily ledger per user and feature (here, the endpoint path):

using System.Collections.Concurrent;
using Microsoft.Extensions.AI;

namespace Bookstore.Assistant.Api;

// Daily token accounting per user and feature. In memory for the demo; Redis or a database in production.
public sealed class TokenLedger(TimeProvider clock)
{
    private readonly ConcurrentDictionary<(string User, string Feature, DateOnly Day), long> _totals = new();

    public long TodayFor(string user) =>
        _totals.Where(e => e.Key.User == user && e.Key.Day == Today).Sum(e => e.Value);

    public void Add(string user, string feature, UsageDetails? usage)
    {
        long tokens = (usage?.InputTokenCount ?? 0) + (usage?.OutputTokenCount ?? 0);
        _totals.AddOrUpdate((user, feature, Today), tokens, (_, current) => current + tokens);
    }

    public IEnumerable<object> Snapshot() =>
        _totals.Where(e => e.Key.Day == Today)
            .OrderBy(e => e.Key.User).ThenBy(e => e.Key.Feature)
            .Select(e => new { e.Key.User, e.Key.Feature, Tokens = e.Value });

    private DateOnly Today => DateOnly.FromDateTime(clock.GetUtcNow().UtcDateTime);
}

public sealed class TokenBudgetExceededException(string user, long used, long limit)
    : Exception($"Daily token budget exceeded for '{user}': {used}/{limit}.")
{
    public long Used { get; } = used;
    public long Limit { get; } = limit;
}

The middleware that enforces the budget and fills the ledger overrides both methods, following the rule from post 2; in streaming, usage is collected from the UsageContent in the last updates:

using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;

namespace Bookstore.Assistant.Api;

// Sits INSIDE the cache: counts only requests that really reach the model and enforces the budget there
public sealed class UsageTrackingChatClient(
    IChatClient innerClient, TokenLedger ledger, IHttpContextAccessor http, long dailyLimit)
    : DelegatingChatClient(innerClient)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        (string user, string feature) = EnsureBudget();
        ChatResponse response = await base.GetResponseAsync(messages, options, cancellationToken);
        ledger.Add(user, feature, response.Usage);
        return response;
    }

    public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        (string user, string feature) = EnsureBudget();
        UsageDetails usage = new();
        await foreach (ChatResponseUpdate update in base.GetStreamingResponseAsync(messages, options, cancellationToken))
        {
            foreach (UsageContent content in update.Contents.OfType<UsageContent>())
            {
                usage.Add(content.Details);
            }
            yield return update;
        }
        ledger.Add(user, feature, usage);
    }

    private (string User, string Feature) EnsureBudget()
    {
        HttpContext context = http.HttpContext ?? throw new InvalidOperationException("No HTTP context.");
        string user = context.Request.Headers["X-Demo-User"].ToString();
        string feature = context.Request.Path.Value ?? "unknown";

        long used = ledger.TodayFor(user);
        if (used >= dailyLimit)
        {
            throw new TokenBudgetExceededException(user, used, dailyLimit);
        }
        return (user, feature);
    }
}

That's why Ada's third request in the first output got a 429: the first answer had spent 612 tokens and the 600-token daily budget was used up. The budget check happens at the start of a request and the spending is recorded at the end, so the last request can overshoot the budget a little. If you want a hard ceiling, set MaxOutputTokens on the request and compute the remaining budget from it. One more detail: when the budget is exhausted, answers already in the cache keep coming back, because the accounting sits inside the cache and an answer served from the cache never reaches this middleware. There's no reason to reject an answer that spends no tokens because of the budget.

The reason I use IHttpContextAccessor is the lifetime question from post 3: the pipeline is a singleton, while the user belongs to the request. You can't give the middleware's constructor a scoped "current user" service; it has to read the request context on every call.

Streaming Responses and the Cache

DistributedCachingChatClient caches streaming calls too: it collects the ChatResponseUpdate items of the first stream and stores them, then streams them again when the same request comes back. According to the API documentation, CoalesceStreamingUpdates is true by default; consecutive updates are merged into a single update before being stored. The result: a streaming answer from the cache arrives not word by word but in large chunks, almost instantly. If your UI's "typing" animation depends on it, users will notice the difference. That's usually not a problem, but in a UI that relies on events such as usage and thinking from post 4 you need to test the stream coming from the cache too. Streaming also raises the partial-stream question: if a stream is cancelled midway, an incomplete answer must not be written to the cache; verify this with a cancellation test in your own setup.

In Production

Keep the accounting somewhere durable and shared. The in-memory TokenLedger resets when the application restarts, and with multiple instances each keeps its own ledger. In production use a counter per user and day in Redis (INCRBY with a TTL ending at the end of the day) or a table in the database.

Show the budget to the user. A "you have reached your daily limit" message that doesn't say how much was used and when it resets comes back to the support team as a ticket. An endpoint that shows the remaining quota from the ledger data is a cheap investment.

Measure the cache hit ratio. Without knowing how many requests were served from the cache, you can't know how much the cache cut costs. In post 8 we'll collect this as a metric with OpenTelemetry.

Common Pitfalls

First, writing answers containing personal data to a shared cache. The key is built only from the messages and options; separating users is your job.

Second, not giving the cache an expiry. By default entries stay in Redis forever; memory fills up and old answers keep coming back.

Third, rate limiting by request count alone. A single request that takes two minutes and a cache hit served in milliseconds count the same; a limit that doesn't count tokens doesn't limit cost.

Fourth, counting tokens outside the cache. An answer from the cache carries Usage too; if you count outside, you put tokens you never spent on your bill.

A Small Experiment

★☆☆ Add a second endpoint that calls /books/blurb's logic with Temperature = 0.7 and ask for the same book from both endpoints. How many keys are created in Redis? Why?

★★☆ Set up a per-user cache for personal answers: add the user id to CacheKeyAdditionalValues through UseDistributedCache(configure: c => ...). How do you provide this value per request when the pipeline is a singleton? Hint: you can derive from DistributedCachingChatClient and override GetCacheKey.

★★★ Move TokenLedger to Redis: one key per user and day, increments with INCRBY and a TTL ending at midnight. Run two application instances on different ports and show that the same user's budget is charged correctly on both.

With this post we've also closed the cost side of Block A: pipeline, DI, streaming, chat history, tools, and now caching, rate limiting and a token budget. In the next post we make all these layers visible: UseOpenTelemetry, the GenAI semantic conventions and the Aspire Dashboard, reading every model call, tool round and token in a trace.