The LLM Context Window: What the Model Remembers and What It Silently Cuts

Version note: The examples in this post were tested with Ollama 0.34 on a local model (gemma3:4b), using .NET 10 and Node 22.

Guess First

Two questions. First: in one request you tell the model "My name is Ada", and in the next request you ask "What is my name?" Does the model remember? Second: the ollama show output lists a context length of 131072 for Gemma 3. If you send the model a prompt of about 4500 tokens, does it read the whole prompt?

Most developers can guess the answer to the first question. The second is the real topic of this post, because the answer is "no", and Ollama does it without giving you any error.

The Model Does Not Remember: The API Is Stateless

LLM APIs are stateless. Every request stands on its own; the server does not store your previous request anywhere. Let's try it in C#. First we send two separate requests, then we repeat the same conversation while keeping the history ourselves:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
ChatOptions options = new() { Temperature = 0, Seed = 42 };

// 1) Two independent requests: the second knows nothing about the first
await client.GetResponseAsync("My name is Ada. Reply in one short sentence.", options);
ChatResponse forgotten = await client.GetResponseAsync("What is my name? Reply in one short sentence.", options);
Console.WriteLine($"Without history: {forgotten.Text}");
Console.WriteLine();

// 2) We keep the history and send all of it with every request
List<ChatMessage> history = [];
string[] turns =
[
    "My name is Ada. Reply in one short sentence.",
    "What is my name? Reply in one short sentence.",
    "Recommend one science fiction book for me. Reply in one short sentence.",
];

foreach (string turn in turns)
{
    history.Add(new ChatMessage(ChatRole.User, turn));
    ChatResponse response = await client.GetResponseAsync(history, options);
    // Add the model's answer to the history too, or it is lost on the next turn
    history.AddMessages(response);
    Console.WriteLine($"[input tokens: {response.Usage?.InputTokenCount}] {response.Text}");
}

Without history: I do not know your name, as I have no way of knowing personal information about you.

[input tokens: 21] Hello Ada, it’s nice to meet you!
[input tokens: 55] Your name is Ada.
[input tokens: 86] I recommend “Dune” by Frank Herbert for a sprawling, epic tale of politics, ecology, and adventure in a distant galaxy.

In the first attempt the model does not know the name, because nobody told it a name in the second request. In the second attempt it does, because we send the history list from start to finish every time. AddMessages(response) appends the model's answer to the list with the assistant role. Forget that line and the model never sees its own answers.

You know this from the web: HTTP is stateless too, and you keep session state on the server or in a cookie yourself. It is the same here. The chat history is your application's data; you decide where to store it and how much of it to send. The model is like a consultant who starts from zero on every request: it reads the file you put in front of it and knows nothing that is not in the file.

The Bill for History

Look at the input token counts in the output: 21, 55, 86. On every turn the model rereads not just the new question but the whole conversation. As we saw in the third post, input tokens drive the bill and the latency. So in a ten-turn chat the last request costs many times more than the first, and the total cost of the conversation grows faster than the number of turns.

There is also an upper limit. Since the history grows on every turn, at some point it will exceed what the model can see at once. That space is called the context window.

What Is a Context Window?

The context window is the maximum number of tokens a model can see in a single request. The prompt, the chat history and the answer the model will generate all have to fit into it. Gemma 3's 4B, 12B and 27B models support up to 128K tokens of input; that is the 131072 in the ollama show output.

But what the model can support and what Ollama allocates at the moment are not the same thing. Remember the ollama ps output from the second post: the CONTEXT column said 4096. Ollama reserves memory for the window when it loads the model and keeps it small by default. According to the Ollama docs, that default depends on GPU memory: 4K with less than 24 GiB of VRAM, 32K between 24 and 48 GiB, and 256K at 48 GiB and above. On the machine I ran these experiments on, the value is 4096. So a 128K model runs out of the box with a 4K window.

Silent Truncation: The 4096 Limit

Now let's try the second question. We put a store rule at the very beginning of the prompt: a discount code for members. A 220-line book catalog follows, and the question comes at the very end. We send the same prompt once with the default setting and once with num_ctx: 8192:

import ollama from "ollama";

const genres = ["fantasy", "history", "science fiction", "biography", "crime"];
const catalog = Array.from({ length: 220 }, (_, i) =>
  `Book ${i + 1}: "Sample Title ${i + 1}", ${genres[i % genres.length]}, ${10 + (i % 30)} USD`);

// The fact comes first, the question last
const prompt = [
  "Store policy: the discount code for members is MOON-7731.",
  "Catalog:",
  ...catalog,
  "Question: What is the discount code for members? If you cannot find it, say so.",
].join("\n");

async function ask(numCtx?: number): Promise<void> {
  const res = await ollama.chat({
    model: "gemma3:4b",
    messages: [{ role: "user", content: prompt }],
    options: { temperature: 0, seed: 42, ...(numCtx ? { num_ctx: numCtx } : {}) },
  });
  console.log(`num_ctx=${numCtx ?? "default"} prompt_eval_count=${res.prompt_eval_count}`);
  console.log(`  ${res.message.content.trim()}`);
}

await ask();
await ask(8192);

num_ctx=default prompt_eval_count=2051
  I cannot find any information about a discount code for members within the provided book list.
num_ctx=8192 prompt_eval_count=4498
  The discount code for members is MOON-7731.

The real size of the prompt is 4498 tokens; with the 8192 window the model read all of it and found the code. With the default window, prompt_eval_count is 2051. The prompt was only 400 tokens over the window, yet the part the model saw dropped to less than half. On top of that, the model said it could not find the code and the API returned no error. From the application's point of view everything looks normal: HTTP 200 and a well-formed answer.

Only Ollama's server log tells you what happened:

time=2026-11-06T23:24:14.062+03:00 level=WARN source=llama_server.go:320 msg="truncating input prompt" limit=2051 prompt=4498 keep=5 new=2051

Ollama kept the first few tokens (keep=5) of the prompt that did not fit and took the rest from the end, cutting the prompt down to 2051 tokens. In these experiments the limit was about half the window: 2051 for 4096 and 4099 for 8192. The part that got cut was the beginning of the prompt. The discount code sat right there, while the question survived because it was at the end. In a real application, what sits at the beginning is usually the most important part: your system prompt, your rules, the start of the document.

Setting num_ctx

In TypeScript options.num_ctx is enough, as you saw above. On the C# side the common properties of ChatOptions have no context window setting, because it is an Ollama-specific option. OllamaSharp provides an extension method for it:

using Microsoft.Extensions.AI;
using OllamaSharp;
using OllamaSharp.Models;

// Long prompts on a busy server can exceed the default 100-second timeout
HttpClient http = new()
{
    BaseAddress = new Uri("http://localhost:11434"),
    Timeout = TimeSpan.FromMinutes(5),
};
IChatClient client = new OllamaApiClient(http, "gemma3:4b");

string[] genres = ["fantasy", "history", "science fiction", "biography", "crime"];
IEnumerable<string> catalog = Enumerable.Range(1, 220)
    .Select(i => $"Book {i}: \"Sample Title {i}\", {genres[(i - 1) % genres.Length]}, {10 + (i - 1) % 30} USD");

string prompt = string.Join("\n",
[
    "Store policy: the discount code for members is MOON-7731.",
    "Catalog:",
    .. catalog,
    "Question: What is the discount code for members? If you cannot find it, say so.",
]);

ChatOptions options = new() { Temperature = 0, Seed = 42 };
// Ollama-specific num_ctx setting
options.AddOllamaOption(OllamaOption.NumCtx, 8192);

ChatResponse response = await client.GetResponseAsync(prompt, options);
Console.WriteLine($"input tokens: {response.Usage?.InputTokenCount}");
Console.WriteLine(response.Text);

input tokens: 4498
The discount code for members is MOON-7731.

The HttpClient part of the code comes from real experience too. I ran the first version of this program while other jobs were using the same Ollama server, and the request failed with a TaskCanceledException: "The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing." OllamaSharp's constructor that takes a Uri comes with the default 100-second timeout; for long prompts it is safer to pass your own HttpClient.

A bigger window has a price. Right after the request, ollama ps showed this:

$ ollama ps
NAME         ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma3:4b    a2af6cc3eb7f    3.9 GB    100% GPU     8192       4 minutes from now
qwen3:4b     359d7dd4bcda    3.2 GB    100% GPU     4096       4 minutes from now

The model takes 3.7 GB with a 4096 window and 3.9 GB with 8192; memory grows with the window. Also, when num_ctx changes, Ollama reloads the model. I measured this with a short "Hi" request: with the model loaded at 4096, a request sending num_ctx: 8192 had a load_duration of 1.09 seconds, and the next request, back on the default setting, took 1.2 seconds again. So requests that arrive with different num_ctx values slow each other down. That is why it is better to pick this value once for the application than to change it request by request. To set it server-wide there is the OLLAMA_CONTEXT_LENGTH environment variable. On the OpenAI-compatible /v1 endpoint we will use in the seventeenth post, num_ctx cannot be sent; the Ollama docs recommend creating a new model with PARAMETER num_ctx in a Modelfile instead.

Managing History: Trim or Summarize?

A bigger window only helps up to a point. In a long conversation you need to manage the history. There are two common methods: dropping old messages (trimming) or turning them into a short summary. Let's compare both on the same conversation:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
ChatOptions options = new() { Temperature = 0, Seed = 42 };

List<ChatMessage> history = [];
string[] turns =
[
    "My name is Ada. Reply in one short sentence.",
    "I mostly read science fiction and I dislike horror. Reply in one short sentence.",
    "My budget is 20 USD per book. Reply in one short sentence.",
    "What is the capital of France? Reply in one short sentence.",
];

foreach (string turn in turns)
{
    history.Add(new ChatMessage(ChatRole.User, turn));
    history.AddMessages(await client.GetResponseAsync(history, options));
}

const string question = "Recommend one book for me and greet me by name. Reply in two sentences.";

// Trimming: only the last two messages stay
List<ChatMessage> trimmed = KeepLast(history, 2);
trimmed.Add(new ChatMessage(ChatRole.User, question));
ChatResponse fromTrimmed = await client.GetResponseAsync(trimmed, options);
Console.WriteLine($"Trimmed [input tokens: {fromTrimmed.Usage?.InputTokenCount}]: {fromTrimmed.Text}");

// Summarizing: old messages turn into a single system message
List<ChatMessage> summarized = await SummarizeOld(client, history, 2, options);
Console.WriteLine($"Summary: {summarized[0].Text}");
summarized.Add(new ChatMessage(ChatRole.User, question));
ChatResponse fromSummary = await client.GetResponseAsync(summarized, options);
Console.WriteLine($"Summarized [input tokens: {fromSummary.Usage?.InputTokenCount}]: {fromSummary.Text}");

static List<ChatMessage> KeepLast(List<ChatMessage> history, int count) =>
[
    .. history.Where(m => m.Role == ChatRole.System),
    .. history.Where(m => m.Role != ChatRole.System).TakeLast(count),
];

static async Task<List<ChatMessage>> SummarizeOld(
    IChatClient client, List<ChatMessage> history, int keep, ChatOptions options)
{
    IEnumerable<ChatMessage> old = history.SkipLast(keep);
    string transcript = string.Join("\n", old.Select(m => $"{m.Role}: {m.Text}"));

    ChatResponse summary = await client.GetResponseAsync(
        $"Summarize what we know about the user in at most two sentences:\n\n{transcript}", options);

    return
    [
        new ChatMessage(ChatRole.System, $"Earlier conversation summary: {summary.Text.Trim()}"),
        .. history.TakeLast(keep),
    ];
}

Trimmed [input tokens: 58]: Hello Sarah, I would highly recommend "Pride and Prejudice" by Jane Austen – it’s a timeless classic filled with wit and romance! It’s a truly delightful read that I think you’ll absolutely love.
Summary: Earlier conversation summary: Ada enjoys reading science fiction and has a budget of $20 per book.
Summarized [input tokens: 84]: Hello Ada, I would highly recommend “Dune” by Frank Herbert – it’s a sprawling, epic science fiction novel that explores themes of politics, religion, and ecology. I think you’ll really enjoy its immersive world and complex characters!

In the trimmed history only the Paris question and its answer remained. The model does not know the name, but instead of saying so it made one up: "Sarah". Since it did not know about the science fiction preference either, it recommended a classic novel. With the summarized history, the model knows about Ada and science fiction and gives a fitting recommendation with 84 tokens. But look closely at the summary: the "I dislike horror" detail is gone. Summarizing compresses information, and like any compression it is lossy; the model decides what stays.

In practice the two are combined: the last few messages stay as they are, older ones turn into a summary, and critical facts (the user's name, order and preferences) are written into a system message from your database, independently of the chat text. Microsoft.Extensions.AI also has MessageCountingChatReducer and SummarizingChatReducer classes for this. In version 10.11.0, though, both are experimental; using them makes the compiler raise error MEAI001, and you have to suppress that diagnostic deliberately.

Common Pitfalls

The first pitfall is assuming the window the model supports and the window Ollama allocates are the same. "128K" on the model card does not mean your request goes into a 128K window. Look at the CONTEXT column in the ollama ps output.

The second is expecting truncation to raise an error. It does not. In any code that sends long prompts, compare prompt_eval_count with the size you expect. Count the prompt yourself and warn as you get close to the window. An unexpectedly small number is the only sign of truncation.

The third is putting the most important information at the very beginning of the prompt and assuming it is always safe there. In this experiment, the beginning was exactly what got cut. Keeping the prompt small enough to fit the window is more reliable than placement tricks.

The fourth is leaving no room for the answer. The window carries input and output together. A prompt that fills the window completely leaves no room for the answer.

The fifth is specific to C#: long prompts and a busy server can exceed the default 100-second timeout of HttpClient. Locally the first request also includes loading the model, so that time runs out faster than you think.

A Small Experiment

★☆☆ Set the number of catalog lines in truncation.ts to 150, 200 and 300. Each time, look at prompt_eval_count and the model's answer. Find the point where truncation starts.

★★☆ Move the discount code from the beginning of the prompt to just before the question and try again with the default window. Does the model find the code this time? Why is this result not a real fix?

★★★ Extend the P04Trim example: write a ReduceHistoryAsync method that turns the oldest messages into a summary once the history passes a given token count. Check the limit with Usage.InputTokenCount and print how the input tokens develop over a ten-turn conversation.

In the next post we will focus on the message roles we brushed past in this one: what system, user and assistant are for, how to write a good system prompt, why a chat history coming from the client cannot be trusted and how separating the text the user writes from your instructions works as a first step against prompt injection.