Provider-Independent LLM Code: IChatClient, the OpenAI-Compatible /v1 and DI

Version note: The examples in this post were tested with Ollama 0.34 on a local model (gemma3:4b), using .NET 10 (Microsoft.Extensions.AI 10.11.0, Microsoft.Extensions.AI.OpenAI 10.11.0, OllamaSharp 5.5.2) and Node 22 (ollama 0.6.4, openai 7.53.0).

Guess First

The Bookstore assistant runs on Ollama in development. For production, the team decided to use a model in the cloud. How many lines of your code do you need to change? And one more: can you call Ollama with the official OpenAI SDK, or do you have to write two separate pieces of code?

By the end of this post the answer to the first question will be "zero lines, one environment variable". We will see the answer to the second with a real call.

The Abstraction: IChatClient

Since the start of the series, the C# examples have always used the same type: IChatClient. This interface comes from the Microsoft.Extensions.AI.Abstractions package and defines the two things we expect from a chat model: getting an answer in one go (GetResponseAsync) and getting it as a stream (GetStreamingResponseAsync). Messages are ChatMessage, settings are ChatOptions, the answer is ChatResponse, and token counts arrive under response.Usage whatever the provider.

Provider-specific code stays behind this interface. OllamaSharp's OllamaApiClient implements it directly. The OpenAI SDK's client is turned into an IChatClient by the AsIChatClient() extension method from the Microsoft.Extensions.AI.OpenAI package. Think of it as a plug adapter: your device (your code) always uses the same plug, and when the wall socket (the provider) changes, you put a different adapter in between. The middleware we used in earlier posts, such as caching and function invocation, also works through this interface, so it doesn't change with the provider.

Ollama's OpenAI-Compatible Door: /v1

Next to its own API (/api/chat), Ollama offers an endpoint that mimics OpenAI's Chat Completions format: http://localhost:11434/v1/. A client that speaks the OpenAI format can talk to Ollama once you point its address here:

curl -s http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ollama" \
  -d '{"model":"gemma3:4b","temperature":0,"seed":42,"messages":[{"role":"user","content":"Name one classic book about refactoring. Title only."}]}' \
  | python3 -m json.tool

{
    "id": "chatcmpl-207",
    "object": "chat.completion",
    "created": 1793997041,
    "model": "gemma3:4b",
    "system_fingerprint": "fp_ollama",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Refactoring: Improving the Design of Existing Code \n"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 21,
        "prompt_tokens_details": {
            "cached_tokens": 0
        },
        "completion_tokens": 12,
        "total_tokens": 33
    }
}

The shape of the response differs from Ollama's own API. The text sits under choices[0].message.content instead of message.content, the token counts are called usage.prompt_tokens/usage.completion_tokens instead of prompt_eval_count/eval_count, and the stop reason is in finish_reason. The key in the Authorization header is, in the words of the Ollama docs, "required but ignored". OpenAI SDKs don't accept an empty key, so you put any value there, usually ollama.

Choosing the Provider from Configuration

Now let's write a small minimal API for the Bookstore. Configuration, not code, decides which provider is used. First the default in appsettings.json:

{
  "Chat": {
    "Provider": "Ollama",
    "Endpoint": "http://localhost:11434",
    "Model": "gemma3:4b"
  }
}

Then Program.cs:

using System.ClientModel;
using Microsoft.Extensions.AI;
using OllamaSharp;
using OpenAI;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

ChatSettings settings = builder.Configuration.GetSection("Chat").Get<ChatSettings>()
    ?? throw new InvalidOperationException("Missing 'Chat' section.");

// The rest of the app only knows IChatClient
builder.Services.AddChatClient(CreateClient(settings));

WebApplication app = builder.Build();

app.MapPost("/books/recommend", async (RecommendRequest request, IChatClient chat, CancellationToken ct) =>
{
    List<ChatMessage> messages =
    [
        new(ChatRole.System, "You are a bookstore assistant. Recommend exactly one book in one sentence."),
        new(ChatRole.User, request.Question),
    ];

    ChatOptions options = new() { Temperature = 0, Seed = 42 };
    ChatResponse response = await chat.GetResponseAsync(messages, options, ct);

    return Results.Ok(new
    {
        provider = settings.Provider,
        answer = response.Text,
        inputTokens = response.Usage?.InputTokenCount,
        outputTokens = response.Usage?.OutputTokenCount,
    });
});

app.Run();

// All provider-specific code lives here
static IChatClient CreateClient(ChatSettings s) => s.Provider switch
{
    "Ollama" => new OllamaApiClient(new Uri(s.Endpoint), s.Model),
    "OpenAI" => new OpenAIClient(
            new ApiKeyCredential(s.ApiKey ?? throw new InvalidOperationException("Missing 'Chat:ApiKey'.")),
            new OpenAIClientOptions { Endpoint = new Uri(s.Endpoint) })
        .GetChatClient(s.Model)
        .AsIChatClient(),
    _ => throw new InvalidOperationException($"Unknown provider '{s.Provider}'."),
};

public sealed class ChatSettings
{
    public required string Provider { get; init; }
    public required string Endpoint { get; init; }
    public required string Model { get; init; }
    public string? ApiKey { get; init; }
}

public sealed record RecommendRequest(string Question);

The endpoint itself doesn't know the provider. It asks for an IChatClient parameter and gets it from the DI container, exactly what we did when injecting services into controllers in the .NET Core Web API series. AddChatClient registers the client in the container and returns a ChatClientBuilder; you add middleware such as caching or logging to the pipeline through that builder. All provider-specific code is inside CreateClient: whichever provider is chosen, the type that comes out is the same.

Same Code, Two Providers

Run the app with the default settings and send a request from a second terminal:

dotnet run
# in a second terminal
curl -s localhost:5238/books/recommend -H "Content-Type: application/json" \
  -d '{"question":"I liked Clean Code. What should I read next?"}'

{"provider":"Ollama","answer":"If you enjoyed “Clean Code,” you might find “The Pragmatic Programmer” a fantastic next read to deepen your understanding of software development principles and best practices.","inputTokens":41,"outputTokens":33}

Now switch the provider. We don't touch the code; we put the key into user secrets and override the provider and address with environment variables. In ASP.NET Core, nested settings are written with a double underscore in environment variables: the Chat:Provider setting becomes Chat__Provider.

dotnet user-secrets init
dotnet user-secrets set "Chat:ApiKey" "ollama"
dotnet user-secrets list
Chat:ApiKey = ollama

Chat__Provider=OpenAI Chat__Endpoint=http://localhost:11434/v1 dotnet run
# in a second terminal, the same request
curl -s localhost:5238/books/recommend -H "Content-Type: application/json" \
  -d '{"question":"I liked Clean Code. What should I read next?"}'

{"provider":"OpenAI","answer":"If you enjoyed “Clean Code,” you might find “The Pragmatic Programmer” a fantastic next read to deepen your understanding of software development principles and best practices.","inputTokens":41,"outputTokens":33}

This time the request went through OpenAI's official .NET SDK to Ollama's /v1 endpoint. With the same model, the same temperature and the same seed behind it, the answer and the token counts came out exactly the same. That answers both opening questions: you can call Ollama with the OpenAI SDK, and switching providers needs no code change. When you move to a real cloud provider, what changes is the Endpoint, the Model and a real ApiKey.

Where Should API Keys Live?

A cloud provider's API key is a password; if it leaks, the bill is written to your account. The rule is simple: the key never goes into code, into appsettings.json or into the repo. This is the LLM-side version of the "don't keep secrets in source code" rule from the OWASP posts in the Web Security series.

On a development machine, .NET's user secrets tool does the job. dotnet user-secrets keeps the setting outside the project folder, in a JSON file in your user profile; the file never enters the repo and is added to configuration automatically when the app runs in the Development environment. It is not encrypted, though, and is designed for development only. On a server or in a container, the key comes from an environment variable or from a secret management service such as Azure Key Vault. ASP.NET Core reads configuration sources in order, and a later one overrides an earlier one: appsettings.json, the environment-specific appsettings.{Environment}.json, user secrets in Development, environment variables, and finally command-line arguments. That is why the environment variable in the example above overrode the Ollama value in appsettings.json.

Also decide what happens when the key is missing. The CreateClient in the example fails at startup with Missing 'Chat:ApiKey'. if there is no key. That beats getting a 401 on the first user request: a misconfigured app is noticed at deploy time.

The Same Idea in TypeScript

On the TypeScript side there is no ready-made shared interface like Microsoft.Extensions.AI, but you can define as much as you need yourself. Let the app know only the ChatClient interface and keep both implementations behind it:

import { Ollama } from 'ollama';
import OpenAI from 'openai';

type Message = { role: 'system' | 'user' | 'assistant'; content: string };

// The only contract the app knows
interface ChatClient {
  complete(messages: Message[]): Promise<string>;
}

function createOllamaClient(host: string, model: string): ChatClient {
  const client = new Ollama({ host });
  return {
    async complete(messages) {
      const res = await client.chat({ model, messages, options: { temperature: 0, seed: 42 } });
      return res.message.content;
    },
  };
}

function createOpenAIClient(baseURL: string, apiKey: string, model: string): ChatClient {
  const client = new OpenAI({ baseURL, apiKey });
  return {
    async complete(messages) {
      const res = await client.chat.completions.create({ model, messages, temperature: 0, seed: 42 });
      return res.choices[0].message.content ?? '';
    },
  };
}

function createClient(): ChatClient {
  const provider = process.env.LLM_PROVIDER ?? 'ollama';
  const model = process.env.LLM_MODEL ?? 'gemma3:4b';
  switch (provider) {
    case 'ollama':
      return createOllamaClient(process.env.OLLAMA_HOST ?? 'http://localhost:11434', model);
    case 'openai': {
      const apiKey = process.env.LLM_API_KEY;
      if (!apiKey) throw new Error('LLM_API_KEY is not set.');
      return createOpenAIClient(process.env.LLM_BASE_URL ?? 'http://localhost:11434/v1', apiKey, model);
    }
    default:
      throw new Error(`Unknown provider '${provider}'.`);
  }
}

const chat = createClient();
const answer = await chat.complete([
  { role: 'system', content: 'You are a bookstore assistant. Recommend exactly one book in one sentence.' },
  { role: 'user', content: 'I liked Clean Code. What should I read next?' },
]);
console.log(`[${process.env.LLM_PROVIDER ?? 'ollama'}] ${answer}`);

npx tsx chat.ts
[ollama] If you enjoyed “Clean Code,” you might find “The Pragmatic Programmer” a fantastic next read to deepen your understanding of software development principles and best practices.

LLM_PROVIDER=openai LLM_API_KEY=ollama npx tsx chat.ts
[openai] If you enjoyed “Clean Code,” you might find “The Pragmatic Programmer” a fantastic next read to deepen your understanding of software development principles and best practices.

LLM_PROVIDER=openai npx tsx chat.ts
Error: LLM_API_KEY is not set.

The same answer as on the C# side; Ollama's own client and the openai package went to the same model. I kept the interface small on purpose: putting a capability the app doesn't use into the abstraction means writing it separately for every provider. When you need streaming or structured output, you add that method to the interface.

The Limits of the Abstraction

A shared interface covers shared capabilities; part of what makes each provider unique stays outside. If you switch without knowing this, the code compiles but the behavior changes silently.

The first example is num_ctx. As we saw in the context window post, in Ollama you can enlarge the context window per request with options.num_ctx. The OpenAI-compatible /v1 endpoint has no equivalent for this setting; the Ollama docs suggest creating a new model with PARAMETER num_ctx in a Modelfile instead. So an app that moves to /v1 can fall back to the default 4096 tokens on long prompts without getting an error.

The second example is model names. The model name lives in configuration, but every provider's catalog is different. When I passed a model name that isn't installed, the openai package threw this error:

LLM_PROVIDER=openai LLM_API_KEY=ollama LLM_MODEL=gemma3:1b npx tsx chat.ts
NotFoundError: 404 model 'gemma3:1b' not found

The third is provider-specific parameters: Ollama's think and keep_alive settings, and the differing structured output and tool calling details across providers. ChatOptions covers the shared ones (temperature, top-p, seed, max tokens, stop sequences, tools); for the rest there are escape hatches such as AdditionalProperties and RawRepresentationFactory. The moment you use them, your code gets tied to that provider a little; do it on purpose and keep it in one place, like CreateClient.

Common Pitfalls

First, writing the API key into appsettings.json "temporarily". The temporary key gets committed, the commit gets pushed, and it stays in history. Once a key has entered the repo, deleting it is not enough; revoke it at the provider and generate a new one.

Second, creating the client on every request. Instead of writing new OllamaApiClient(...) or new OpenAIClient(...) inside the endpoint, create it once and share it through DI; that way the HTTP connections and the middleware pipeline are managed in one place too.

Third, reading "OpenAI-compatible" as "identical in every way". Compatibility covers the fields listed in the docs. Ollama's /v1 endpoint, for example, does not support logprobs and doesn't take the num_ctx setting. Before you switch, check every field you use in the target provider's documentation.

Fourth, switching providers and leaving the prompt and the evaluation as they were. The code stays the same but the model changes; the same prompt can produce answers of a different length and shape on another model. Rerun the golden set from post 15 on every switch.

A Small Experiment

★☆☆ Run the minimal API again, this time with the Chat__Model=qwen3:4b environment variable. Did the model change without touching the code? Why is outputTokens so large despite a short answer? Remember the thinking topic from post 9.

★★☆ Add the UseDistributedCache middleware from post 16 to the builder returned by AddChatClient; you will need to call AddDistributedMemoryCache() first so that the container has an IDistributedCache. Send the same request twice. The second answer spent no tokens, so what do inputTokens and outputTokens show? Why does that matter when you write a cost report?

★★★ Add a stream(messages) method to the TypeScript ChatClient interface and write it for both implementations: turn the async iterator that Ollama returns with stream: true and the stream OpenAI returns with stream: true into the same AsyncIterable<string> type.

That brings us to the end of Block A: from tokens to the context window, from sampling to tool calling, from embeddings to RAG, we built the core structure by actually running it on local models. With the next post the series turns to history: from the 2017 "Attention Is All You Need" paper to the launch of ChatGPT, we will look at the steps that made today's LLMs possible.