Wiring IChatClient into ASP.NET Core: AddChatClient, Keyed Clients, Options and Secrets

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, Microsoft.Extensions.AI.OpenAI 10.11.0, OllamaSharp 5.5.2, Microsoft.Extensions.Http.Resilience 10.11.0 and a local Ollama 0.34.4 (gemma3:4b, gemma3:12b, nomic-embed-text).

Guess First

In the Bookstore API (Application Programming Interface) you registered the provider's HttpClient with IHttpClientFactory, and, as the team rule says for every outgoing HTTP (Hypertext Transfer Protocol) call, you added one line: .AddStandardResilienceHandler(). You didn't touch any settings. Then the "compare two books" endpoint sends a non-streaming request to gemma3:12b on an Ollama running on a shared machine, and the full answer takes longer than 10 seconds. What does the user see? How many requests reach the model?

And one more: does a [FromKeyedServices("smart")] IChatClient parameter get a new client on every HTTP request, or always the same instance?

The answer to the first question is an empty 500 after 30 seconds and three separate requests to the model; we will see the real logs below. The endpoint itself will answer the second question.

AddChatClient: Moving the Pipeline into the DI Container

In the previous post we built the pipeline by hand with ChatClientBuilder. In ASP.NET Core the same builder comes through DI (Dependency Injection): AddChatClient registers the client in the container and returns a ChatClientBuilder, to which you add .Use...() calls. The default lifetime is singleton; the method's last parameter is a ServiceLifetime and can be changed, but you need a good reason to change it. The factory overload (sp => ...) creates the client the first time it's requested, using other services in the container.

The builder coming from DI has a side effect: middleware such as UseLogging() and UseDistributedCache() take the service they need (ILoggerFactory, IDistributedCache) from the container when you don't pass it. If you call Build() outside a container, with an empty service provider, you see this as an error:

Unhandled exception. System.InvalidOperationException: No service for type 'Microsoft.Extensions.Logging.ILoggerFactory' has been registered.

In ASP.NET Core ILoggerFactory is already registered, so UseLogging() works without parameters.

Two Models, Two Keys: Keyed Clients

Bookstore has two different needs. The one-sentence blurb on a product page can be written by a cheap, fast model; comparing two books for a junior developer needs a stronger one. Keyed services, introduced in .NET 8, are made for this: you separate two registrations of the same type with a key.

// Two keyed clients: cheap/fast and strong/slow
builder.Services.AddKeyedChatClient("fast", sp => AiClientFactory.CreateChatClient(sp, o => o.FastModel))
    .ConfigureOptions(o => o.Temperature ??= 0)
    .UseLogging();

builder.Services.AddKeyedChatClient("smart", sp => AiClientFactory.CreateChatClient(sp, o => o.SmartModel))
    .UseLogging();

builder.Services.AddEmbeddingGenerator(AiClientFactory.CreateEmbeddingGenerator);

Each key has its own pipeline: "fast" defaults to temperature 0, while "smart" uses whatever settings the caller passes. The embedding generator is registered the same way with AddEmbeddingGenerator; VectorData will use it in post 9. Endpoints ask for the client by its key:

app.MapGet("/ai/models", (
    [FromKeyedServices("fast")] IChatClient fast,
    [FromKeyedServices("smart")] IChatClient smart,
    IEmbeddingGenerator<string, Embedding<float>> embeddings) => new
{
    fast = fast.GetService<ChatClientMetadata>()?.DefaultModelId,
    smart = smart.GetService<ChatClientMetadata>()?.DefaultModelId,
    embedding = embeddings.GetService<EmbeddingGeneratorMetadata>()?.DefaultModelId,
    sameInstance = ReferenceEquals(fast, app.Services.GetRequiredKeyedService<IChatClient>("fast")),
});

app.MapPost("/books/blurb", async (BlurbRequest request, [FromKeyedServices("fast")] IChatClient chat, CancellationToken ct) =>
{
    ChatResponse response = await chat.GetResponseAsync(
        $"Write a one-sentence shop blurb for the book '{request.Title}'. Reply with the sentence only.", cancellationToken: ct);
    return new { model = response.ModelId, text = response.Text.Trim() };
});

app.MapPost("/books/compare", async (CompareRequest request, [FromKeyedServices("smart")] IChatClient chat, CancellationToken ct) =>
{
    ChatResponse response = await chat.GetResponseAsync(
        $"Compare '{request.First}' and '{request.Second}' for a junior developer in three short bullet points.",
        new ChatOptions { Temperature = 0, Seed = 42 }, ct);
    return new { model = response.ModelId, text = response.Text.Trim() };
});

curl -s localhost:19620/ai/models
{"fast":"gemma3:4b","smart":"gemma3:12b","embedding":"nomic-embed-text","sameInstance":true}

curl -s -X POST localhost:19620/books/blurb -H 'Content-Type: application/json' -d '{"title":"Domain-Driven Design"}'
{"model":"gemma3:4b","text":"Unlock the secrets to building robust, maintainable software by mastering Domain-Driven Design, Eric Evans’s seminal guide to aligning your code with your business needs."}

sameInstance: true answers the second opening question: the client an endpoint gets is the same singleton as in the root container. The pipeline is built once and shared across all requests. That's why middleware must be thread-safe and carry per-request state in ChatOptions or parameters; it's the real reason we cloned the options in post 2.

A note: "Reply with the sentence only." at the end of the blurb prompt wasn't there on the first try. Despite the "one sentence" request, gemma3:4b returned three options and a follow-up question. With small models you need to state the output format explicitly.

Choosing the Provider: The Options Pattern

Model names and the provider live in configuration, not in code. appsettings.json:

"Ai": {
  "Provider": "Ollama",
  "Endpoint": "http://localhost:11434",
  "FastModel": "gemma3:4b",
  "SmartModel": "gemma3:12b",
  "EmbeddingModel": "nomic-embed-text"
}

We bind this section to a strongly typed options class and validate it at startup:

using System.ComponentModel.DataAnnotations;

namespace Bookstore.Assistant.Api;

public sealed class ChatModelOptions
{
    public const string SectionName = "Ai";

    [Required, AllowedValues("Ollama", "OpenAI")]
    public required string Provider { get; init; }

    [Required, Url]
    public required string Endpoint { get; init; }

    [Required]
    public required string FastModel { get; init; }

    [Required]
    public required string SmartModel { get; init; }

    [Required]
    public required string EmbeddingModel { get; init; }

    // Only needed for a cloud provider; goes into an environment variable or a secret store, not appsettings
    public string? ApiKey { get; init; }
}

builder.Services.AddOptions<ChatModelOptions>()
    .BindConfiguration(ChatModelOptions.SectionName)
    .ValidateDataAnnotations()
    .Validate(o => o.Provider != "OpenAI" || !string.IsNullOrWhiteSpace(o.ApiKey),
        "Ai:ApiKey is required when Ai:Provider is 'OpenAI'.")
    .ValidateOnStart();

All provider-specific code lives in a single static class. The rest of the application only sees IChatClient and IEmbeddingGenerator:

using System.ClientModel;
using System.ClientModel.Primitives;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;
using OllamaSharp;
using OpenAI;

namespace Bookstore.Assistant.Api;

public static class AiClientFactory
{
    public const string HttpClientName = "ai-provider";

    // All provider-specific code lives here; the rest of the app only sees IChatClient
    public static IChatClient CreateChatClient(IServiceProvider services, Func<ChatModelOptions, string> modelSelector)
    {
        ChatModelOptions options = services.GetRequiredService<IOptions<ChatModelOptions>>().Value;
        HttpClient http = services.GetRequiredService<IHttpClientFactory>().CreateClient(HttpClientName);
        string model = modelSelector(options);

        return options.Provider switch
        {
            "Ollama" => new OllamaApiClient(http, model),
            "OpenAI" => new OpenAIClient(
                    new ApiKeyCredential(options.ApiKey!),
                    new OpenAIClientOptions { Endpoint = new Uri(options.Endpoint), Transport = new HttpClientPipelineTransport(http) })
                .GetChatClient(model)
                .AsIChatClient(),
            _ => throw new InvalidOperationException($"Unknown provider '{options.Provider}'."),
        };
    }

    public static IEmbeddingGenerator<string, Embedding<float>> CreateEmbeddingGenerator(IServiceProvider services)
    {
        ChatModelOptions options = services.GetRequiredService<IOptions<ChatModelOptions>>().Value;
        HttpClient http = services.GetRequiredService<IHttpClientFactory>().CreateClient(HttpClientName);

        return options.Provider switch
        {
            "Ollama" => new OllamaApiClient(http, options.EmbeddingModel),
            "OpenAI" => new OpenAIClient(
                    new ApiKeyCredential(options.ApiKey!),
                    new OpenAIClientOptions { Endpoint = new Uri(options.Endpoint), Transport = new HttpClientPipelineTransport(http) })
                .GetEmbeddingClient(options.EmbeddingModel)
                .AsIEmbeddingGenerator(),
            _ => throw new InvalidOperationException($"Unknown provider '{options.Provider}'."),
        };
    }
}

The OpenAI branch compiles but was not run in this post; we use no API keys. What the two branches share is the same named HttpClient from IHttpClientFactory. OllamaSharp takes that client directly; the OpenAI SDK (Software Development Kit) builds its own HTTP layer on System.ClientModel, so we wrap the client with HttpClientPipelineTransport and pass it as Transport. That way timeouts and retries are defined in one place, independent of the provider.

Changing the model is now an environment variable. In ASP.NET Core, nested keys are written with a double underscore:

Ai__SmartModel=qwen3:4b dotnet run --urls http://localhost:19621
curl -s localhost:19621/ai/models
{"fast":"gemma3:4b","smart":"qwen3:4b","embedding":"nomic-embed-text","sameInstance":true}

Secrets and Validation at Startup

A cloud provider's API key doesn't go into appsettings.json. On a development machine you use User Secrets; on a server, an environment variable or a secret store such as Azure Key Vault; the key arrives in configuration as Ai__ApiKey and binds to ChatModelOptions.ApiKey. We saw how User Secrets works in the LLM (Large Language Model) Fundamentals post "Provider-Independent LLM Code"; the details of secret management are in the upcoming Configuration and Secret Management series.

The real gain here is ValidateOnStart(). A deployment with a forgotten key fails at startup instead of getting a 401 on the first user request:

Ai__Provider=OpenAI Ai__Endpoint=https://api.openai.com/v1 dotnet run --urls http://localhost:19620
fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      Microsoft.Extensions.Options.OptionsValidationException: Ai:ApiKey is required when Ai:Provider is 'OpenAI'.

A misspelled provider name is caught the same way:

Ai__Provider=Gemini dotnet run --urls http://localhost:19620
fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      Microsoft.Extensions.Options.OptionsValidationException: DataAnnotation validation failed for 'ChatModelOptions' members: 'Provider' with the error: 'The Provider field does not equal any of the values specified in AllowedValuesAttribute.'.

Timeouts and Retries: The Defaults Aren't Made for LLMs

Now to the opening question. I had registered the provider's HttpClient like this:

// The provider's HttpClient: timeouts and retries are managed here
builder.Services.AddHttpClient(AiClientFactory.HttpClientName, (sp, http) =>
    {
        http.BaseAddress = new Uri(sp.GetRequiredService<IOptions<ChatModelOptions>>().Value.Endpoint);
        http.Timeout = Timeout.InfiniteTimeSpan; // let the resilience handler manage the duration
    })
    .AddStandardResilienceHandler();

According to the table on Microsoft Learn, the standard resilience handler chains five strategies from the outside in: a rate limiter (1,000 concurrent requests), a 30-second total timeout, exponential retry with 3 attempts, a circuit breaker and a 10-second attempt timeout. These are sensible for a typical REST (Representational State Transfer) call. Not for an LLM call. The log of my request to the /books/compare endpoint:

Execution attempt. Source: 'ai-provider-standard//Standard-Retry', Operation Key: '', Result: 'The operation didn't complete within the allowed timeout of '00:00:10'.', Handled: 'True', Attempt: '0', Execution Time: 10010,25
Execution attempt. Source: 'ai-provider-standard//Standard-Retry', Operation Key: '', Result: 'The operation didn't complete within the allowed timeout of '00:00:10'.', Handled: 'True', Attempt: '1', Execution Time: 10003,62
Execution attempt. Source: 'ai-provider-standard//Standard-Retry', Operation Key: '', Result: 'The operation was canceled.', Handled: 'False', Attempt: '2', Execution Time: 4832,6666ms
Resilience event occurred. EventName: 'OnTimeout', Source: 'ai-provider-standard//Standard-TotalRequestTimeout', Operation Key: '', Result: ''
fail: Microsoft.Extensions.AI.LoggingChatClient[1784604714]
      Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.

The user waited 30 seconds and got a 500 with no body. What happened behind it is worse: three separate requests went to gemma3:12b, the first two were cut after 10 seconds each, and every retry started the generation from scratch. On a local model that's wasted GPU (Graphics Processing Unit) time; on a cloud provider it's a token bill even for the cut requests, because the provider had already started processing the prompt.

My first fix attempt was to raise only the attempt timeout to 2 minutes. The application didn't start:

Microsoft.Extensions.Options.OptionsValidationException: The sampling duration of circuit breaker strategy needs to be at least double of an attempt timeout strategy’s timeout interval, in order to be effective. Sampling Duration: 30s,Attempt Timeout: 120s

The circuit breaker measures the failure ratio over a time window; if the window is shorter than a single attempt the measurement becomes meaningless, and Polly rejects it at startup. The final version:

    .AddStandardResilienceHandler(r =>
    {
        // An LLM call can take minutes, not seconds: the default 10 s / 30 s limits don't work here
        r.AttemptTimeout.Timeout = TimeSpan.FromMinutes(5);
        r.TotalRequestTimeout.Timeout = TimeSpan.FromMinutes(6);
        // The circuit breaker's sampling duration must be at least twice the attempt timeout (Polly validation)
        r.CircuitBreaker.SamplingDuration = TimeSpan.FromMinutes(10);
        // Retry only on connection errors, 429 and 5xx; don't restart a generation that timed out
        r.Retry.ShouldHandle = args => ValueTask.FromResult(args.Outcome switch
        {
            { Exception: HttpRequestException } => true,
            { Result.StatusCode: HttpStatusCode.TooManyRequests } => true,
            { Result.StatusCode: >= HttpStatusCode.InternalServerError } => true,
            _ => false,
        });
    });

I narrowed the retry decision on purpose. 429 (rate limit) and 5xx are transient errors, and the provider usually rejected the request without processing it; retrying makes sense. A timeout, on the other hand, means "the model was working but didn't make it in time"; sending the same request again spends the same time again. The 5-minute limit is for a local, shared Ollama; with a cloud provider, set this value from the provider's own latency measurements. The same request with these settings:

Execution attempt. Source: 'ai-provider-standard//Standard-Retry', Operation Key: '', Result: 'The operation didn't complete within the allowed timeout of '00:05:00'.', Handled: 'False', Attempt: '0', Execution Time: 300020,2408ms
fail: Microsoft.Extensions.AI.LoggingChatClient[1784604714]
      GetResponseAsync failed.
      Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:05:00'.

# Ollama server log
level=INFO source=sched.go:556 msg="llama-server model predicted to exceed available memory, evicting" predicted="9.0 GiB"

To be honest: on this machine gemma3:12b didn't answer within five minutes either. Ollama runs one request at a time (OLLAMA_NUM_PARALLEL:1), and other work was using qwen3:4b continuously; to load the 9 GiB model it had to evict the others and wait for the queue to drain. But the important line in the log is Handled: 'False': no retry after the timeout, the model was called once and the user got a single clear error. Rather than a longer limit, the right answer here is not to run the large model on such a crowded server, or to offer a streaming endpoint that tells the user "you are in the queue" right away.

Retries on Streaming Requests

On a streaming call, what the resilience handler sees is the HTTP response headers, not the body. Retries only kick in for errors before the headers arrive: if the connection couldn't be established or the provider returned 429. If the connection drops after the model has started streaming tokens, the handler can't see it; the error surfaces as an exception in your await foreach loop. That's a good thing. Silently regenerating an answer the user has already half seen and appending a second text is far more confusing than reporting the error clearly. We'll see how to report a partial answer with Server-Sent Events in the next post.

The same boundary applies to timeouts, and that surprised me. The resilience handler is a DelegatingHandler; its work ends when SendAsync returns. On a non-streaming call Ollama sends the headers when the answer is complete (in the blurb request above the first attempt took 8003 ms, the whole generation), so the timeout covers the entire generation. On a streaming call the headers arrive as soon as the first chunk is ready. I verified this with a separate experiment: a small test server that sends headers right away and streams the body line by line over 8 seconds, and a standard handler with a 2-second attempt timeout and a 3-second total timeout:

using HttpResponseMessage res = await http.GetAsync("/", HttpCompletionOption.ResponseHeadersRead);
Console.WriteLine($"headers after {sw.ElapsedMilliseconds} ms");
using StreamReader reader = new(await res.Content.ReadAsStreamAsync());
while (await reader.ReadLineAsync() is string line)
{
    Console.WriteLine($"{sw.ElapsedMilliseconds} ms {line}");
}

headers after 27 ms
28 ms chunk0
1027 ms chunk1
2028 ms chunk2
3033 ms chunk3
4043 ms chunk4
5053 ms chunk5
6063 ms chunk6
7073 ms chunk7

The 3-second "total" timeout didn't stop the 7-second stream. For a streaming LLM answer, this handler's timeouts only bound "how long until the first token arrives"; if you want to bound the total duration of the stream, do it on the application side by putting a deadline on the request's CancellationToken with CancelAfter.

Lifetime: Singleton Client, Scoped Service

The client is a singleton; what about the services that use it? In Bookstore the chat service will use a DbContext on every request (when we store chat history in post 5). DbContext is scoped, so the service using it must be scoped too. A scoped service depending on the singleton IChatClient is fine; the reverse is the problem: a singleton middleware asking for a scoped service in its constructor fails ASP.NET Core's scope validation in the development environment, and in production it keeps that scoped service alive for the lifetime of the app. If middleware needs a per-request service, it should get it from IServiceProvider at request time or carry the data through ChatOptions.AdditionalProperties.

Then there's the lifetime of the HttpClient taken from the factory. Our clients are singletons, so they take an HttpClient from the factory once and hold it for the lifetime of the app; the factory's periodic handler rotation doesn't reach this instance. If the endpoint's DNS (Domain Name System) record can change, Microsoft's HttpClient guidelines recommend setting SocketsHttpHandler.PooledConnectionLifetime for long-lived clients.

In Production

Set timeouts from latency measurements. The "5 minutes" figure came from the real behavior of the shared Ollama on this machine. In production, measure p99 latency (with OpenTelemetry in post 8), set the attempt timeout a little above it, and show users progress with streaming on long tasks.

Take 429 seriously. Cloud providers enforce requests-per-minute and tokens-per-minute limits, and a 429 response usually carries a Retry-After header. The standard handler's retry strategy honors that header; but retries alone aren't enough, you also need per-user limits on your side. We'll do that with rate limiting in post 7.

Keep the number of keyed clients small. Use role names such as "fast" and "smart", not model names, as keys; endpoint code shouldn't change when the model does. As the number of roles grows, tagging which request went to which model in telemetry becomes important.

Common Pitfalls

First, creating a new client on every request. Writing new OllamaApiClient(...) inside an endpoint means a new HttpClient and new connections on every request; under load it goes as far as socket exhaustion, and it rebuilds the middleware every time too. Register the client once and get it from DI.

Second, adding the standard resilience handler to an LLM provider with default settings. We saw it above: a 10-second attempt timeout plus retry-on-timeout made the model generate a slow answer three times from scratch and showed the user an error after 30 seconds.

Third, hardcoding the model name. If the "gemma3:12b" string lives inside the endpoint, changing models means a deployment. Keep it in configuration and validate it at startup.

Fourth, writing the API key into appsettings.Development.json because "it's only for development anyway". That file goes into the repo. The key lives in User Secrets or an environment variable; once it has been in the repo, revoke it at the provider.

A Small Experiment

★☆☆ Remove the ValidateOnStart() line and start the application with Ai__Provider=Gemini. Does it start? When, and on which request, does the error appear?

★★☆ Add a third keyed client: "tools", model qwen3:4b, with UseFunctionInvocation() in the pipeline. Read the key and the model from configuration and add it to the /ai/models output.

★★★ Define a separate named HttpClient for streaming endpoints ("ai-provider-streaming"): retry only on 429, and an application-side deadline of 15 minutes on the stream. How do you decide which named client an IChatClient uses? Two separate keyed clients, or a choice by request type inside a single client?

Our clients are in DI, their settings in configuration, and provider calls sit behind a sensible resilience policy. In the next post we stream the answer to the user: a minimal API returning IAsyncEnumerable, .NET 10's TypedResults.ServerSentEvents, stopping the model when the user closes the tab, and telling the client about an answer cut in half.