LLMs and RAG: Answering with Data the Model Has Never Seen

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b, nomic-embed-text), using .NET 10 (Microsoft.Extensions.AI 10.11.0, OllamaSharp 5.5.2).

Guess First

The Bookstore's return policy is clear: you can return a book within 21 days of delivery. That fact sits in our own document and has never been published anywhere. If you ask gemma3:4b, playing the Bookstore assistant, "How many days do I have to return a book?", what will it say? "I don't know", or a number?

The answer is in the first section. After that we will see how to make the model read data it doesn't know, by adding a chat model on top of the search we built in the embeddings post.

The Model Doesn't Know Your Policies

First let's ask without any context, with only the system prompt "You are the Bookstore support assistant.":

$ dotnet run -- "How many days do I have to return a book?" --no-rag

Okay, let's get that sorted out for you! 

You have **30 days** to return a book to us. 😊 

Do you have any other questions about our return policy, or perhaps you'd like to start the return process?
(input tokens: 34, output tokens: 55)

Confident, polite and wrong. The model has never seen our policy; 30 days is the return window it met most often in its training data, so that's what it said. In my first attempt, without any system prompt, it said "return policies vary greatly" and listed 30 days for Amazon and Barnes & Noble. Both are understandable; but if you tell a customer 30 days, you'll have a problem with the customer who asks for a return on day 25.

A model's knowledge is frozen at the date its training data ends, and it contains nothing specific to your company: policies, stock, contracts, internal documents. The most common way to fill that gap is RAG.

RAG: Retrieve, Augment, Generate

RAG (Retrieval-Augmented Generation) is named after its three steps. Retrieve: find the document chunks related to the question. Augment: add those chunks to the prompt. Generate: ask the model to produce the answer based on those chunks. The idea goes back to the 2020 paper by Lewis et al., but the simple form we use today needs far less: an embedding model, a chat model and a few lines of code.

The best analogy is an open-book exam. The model doesn't have to memorize everything; put the right page in front of it and it reads the answer from there. Finding the page is the embedding search's job, reading it and writing a sentence is the chat model's.

Why not fine-tuning? Fine-tuning changes the model's weights; it's good for style, format or a narrow task, but it doesn't reliably add new facts, and it needs a dataset and a GPU. When the policy changes you would have to train the model again. With RAG you update the document, and the next question is answered with the new information.

Splitting the Document into Chunks

Our source is a small markdown file:

# Shipping
Standard shipping takes 3 to 5 business days. Orders over 40 USD ship for free; otherwise standard shipping costs 4.50 USD.

Express shipping costs 9.90 USD and delivers on the next business day if you order before 2 pm.

# Returns
You can return a book within 21 days of delivery if it is unread and in its original condition. Refunds go back to the original payment method within 5 business days.

E-book purchases and opened gift wrap cannot be returned.

# Gift cards
Gift cards are valid for 3 years from the date of purchase. They can be used online and in the Ankara store, but they cannot be exchanged for cash.

# Loyalty program
Members earn 1 point per dollar spent. Every 100 points can be redeemed for a 5 USD discount on a future order.

We could embed this file in one piece, but then every question would get the whole file; as we saw in the embeddings post, the vector of a long text averages away its details. Instead we split the text into chunks. The simplest strategy: every paragraph is a chunk, and every chunk starts with its section title. The sentence "E-book purchases and opened gift wrap cannot be returned." is a bit of an orphan on its own; with Returns: in front, both the embedding and the model understand that it's a return rule. This file yields six chunks.

In real documents paragraphs are rarely this tidy. Common approaches are splitting by a fixed number of tokens (e.g. 300–500), leaving some overlap between neighbouring chunks, and keeping structure such as headings and tables. Which one works best depends on your data; don't decide without measuring.

The Whole Pipeline: Minimal RAG in C#

We open two OllamaApiClient instances, one for the embedding generator (nomic-embed-text) and one for the chat model (gemma3:4b). There's no vector database; keeping six vectors in a list is enough:

using System.Globalization;
using System.Numerics.Tensors;
using Microsoft.Extensions.AI;
using OllamaSharp;

CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;

Uri ollama = new("http://localhost:11434");
IEmbeddingGenerator<string, Embedding<float>> embedder = new OllamaApiClient(ollama, "nomic-embed-text");
IChatClient chat = new OllamaApiClient(ollama, "gemma3:4b");

// 1. Chunk: one chunk per paragraph, prefixed with its section title
List<string> chunks = Chunk(File.ReadAllText("policies.md"));

// 2. Index: embed every chunk once
GeneratedEmbeddings<Embedding<float>> vectors =
    await embedder.GenerateAsync(chunks.Select(c => $"search_document: {c}"));

string question = args[0];
bool useRag = !args.Contains("--no-rag");

List<ChatMessage> messages = [];
if (useRag)
{
    // 3. Retrieve: the 2 chunks closest to the question
    ReadOnlyMemory<float> q = await embedder.GenerateVectorAsync($"search_query: {question}");
    List<(string Text, float Score)> top = chunks
        .Select((text, i) => (text, TensorPrimitives.CosineSimilarity(q.Span, vectors[i].Vector.Span)))
        .OrderByDescending(x => x.Item2)
        .Take(2)
        .ToList();

    foreach ((string text, float score) in top)
    {
        Console.WriteLine($"[{score:F3}] {text}");
    }

    // 4. Augment: put the chunks into the prompt and set the rules
    string context = string.Join("\n\n", top.Select(t => t.Text));
    messages.Add(new ChatMessage(ChatRole.System, """
        You are the Bookstore support assistant.
        Answer using ONLY the facts inside <context>.
        If the answer is not in <context>, reply exactly: "I don't know based on our store policies."
        """));
    messages.Add(new ChatMessage(ChatRole.User, $"<context>\n{context}\n</context>\n\nQuestion: {question}"));
}
else
{
    // Same assistant, but it only has its training data
    messages.Add(new ChatMessage(ChatRole.System, "You are the Bookstore support assistant."));
    messages.Add(new ChatMessage(ChatRole.User, question));
}

// 5. Generate
ChatResponse response = await chat.GetResponseAsync(messages, new ChatOptions { Temperature = 0, Seed = 42 });
Console.WriteLine($"\n{response.Text}");
Console.WriteLine($"(input tokens: {response.Usage?.InputTokenCount}, output tokens: {response.Usage?.OutputTokenCount})");

static List<string> Chunk(string markdown)
{
    List<string> result = [];
    string section = "";
    foreach (string block in markdown.Split("\n\n"))
    {
        foreach (string line in block.Split('\n', StringSplitOptions.RemoveEmptyEntries))
        {
            if (line.StartsWith("# "))
            {
                section = line[2..];
            }
            else
            {
                result.Add($"{section}: {line}");
            }
        }
    }
    return result;
}

The five steps are numbered in the code. Two places deserve attention. The first is the system prompt: we tell the model to use only the information inside <context>, and spell out word for word which sentence to write if the answer isn't there. The second is separating the context from the user's question with delimiters: as we discussed in the prompt post, make the boundaries clear so the model doesn't mix up "document" and "question".

The Same Question, This Time with Context

$ dotnet run -- "How many days do I have to return a book?"
[0.907] Returns: You can return a book within 21 days of delivery if it is unread and in its original condition. Refunds go back to the original payment method within 5 business days.
[0.696] Returns: E-book purchases and opened gift wrap cannot be returned.

You can return a book within 21 days of delivery.
(input tokens: 134, output tokens: 14)

Retrieval put the right chunk on top with 0.907, and the model took the answer from there: 21 days. Look at the cost side too: the prompt grew from 34 to 134 tokens, because two chunks and the rules were added. The answer dropped from 55 tokens to 14; the model stopped chatting and answered the question. In a real system the number and length of chunks directly determine the token cost of every request.

Let's push a little harder. This question requires reading the rule and making a comparison:

$ dotnet run -- "My order is 35 USD. Do I pay for shipping?"
[0.742] Shipping: Standard shipping takes 3 to 5 business days. Orders over 40 USD ship for free; otherwise standard shipping costs 4.50 USD.
[0.719] Shipping: Express shipping costs 9.90 USD and delivers on the next business day if you order before 2 pm.

Otherwise standard shipping costs 4.50 USD.
(input tokens: 144, output tokens: 12)

The information is right but the sentence is odd: it starts with "Otherwise..." and never says "yes". The "use ONLY the facts inside the context" instruction pushed a small model towards copying sentences from the context. Strict rules reduce hallucination, but they can also reduce how natural the answer sounds. We will balance the two by measuring in the next post.

Teaching It to Say "I Don't Know"

The real value of RAG isn't only giving the right answer, it's noticing when it doesn't know the answer. Two questions that aren't in the document, and one edge case:

$ dotnet run -- "Do you ship to Germany?"
[0.618] Shipping: Standard shipping takes 3 to 5 business days. Orders over 40 USD ship for free; otherwise standard shipping costs 4.50 USD.
[0.591] Shipping: Express shipping costs 9.90 USD and delivers on the next business day if you order before 2 pm.

I don't know based on our store policies.
(input tokens: 136, output tokens: 12)

$ dotnet run -- "Can I turn my loyalty points into cash?"
[0.726] Loyalty program: Members earn 1 point per dollar spent. Every 100 points can be redeemed for a 5 USD discount on a future order.
[0.573] Gift cards: Gift cards are valid for 3 years from the date of purchase. They can be used online and in the Ankara store, but they cannot be exchanged for cash.

I don't know based on our store policies.
(input tokens: 148, output tokens: 12)

$ dotnet run -- "Can I return an e-book I bought yesterday?"
[0.807] Returns: E-book purchases and opened gift wrap cannot be returned.
[0.757] Returns: You can return a book within 21 days of delivery if it is unread and in its original condition. Refunds go back to the original payment method within 5 business days.

No, E-book purchases and opened gift wrap cannot be returned.
(input tokens: 134, output tokens: 15)

For the Germany question retrieval again returned the shipping chunks, because they were the closest thing it had; retrieval always returns something. But the model saw that those chunks contain no country information and wrote the sentence we asked for. The loyalty points question is more interesting: the policy says points can be turned into a discount, not whether they can be turned into cash. Instead of inferring "no", the model said "I don't know". That is cautious and, in this scenario, correct behaviour; but telling the user "you can use your points as a discount" would have been more helpful. For the e-book question the right chunk came first and the answer is clear.

Remember the negation trap from the embeddings post: the top score for the Germany question was 0.618, quite low next to the 0.907 of the return question. That gap can be used as an "I have no good source" signal. You can also put a threshold in front of the model and say "I don't know" without calling it at all; but you have to pick the threshold by measuring on your own data.

The Limits of RAG

If retrieval brings the wrong chunk, the model can't give the right answer; the weakest link in the RAG chain is often the search. For questions that need two chunks together (e.g. "Can I pay for express shipping with my gift card?"), Take(2) may not bring both. A bigger top-k increases recall but makes the prompt longer and distracts the model.

A list in memory is great for six chunks, not for six million. Real systems keep the vectors in a vector database or in a relational database with vector support, re-index documents as they change, and usually combine the search with keyword search. We will cover those in a separate RAG series, bringing in vector search with EF Core as well. The goal of this post is to show the mechanism: a search, a prompt and a rule.

Finally, RAG doesn't end hallucination, it reduces it. The model can misread the context, mix up two chunks or add a detail that isn't in the context. The way to catch that is to measure.

Common Pitfalls

The first is not writing the "I don't know" rule. Without it, when the model finds no answer in the context it makes something up from its training data, as in the opening example, and says it as if it had read it in the context.

The second is cutting chunks off from their context. A one-sentence chunk with no heading gets caught in the wrong searches and is misread by the model. Put the section title or the document name inside the chunk.

The third is never logging the retrieval result. For a wrong answer the first question should be "did the model misread, or did the wrong chunk arrive?". The program in this post prints the retrieved chunks with their scores; in production, store that somewhere too.

The fourth is updating the document but not the index. If the policy goes from 21 to 30 days and the old vectors still point to the old text, RAG will now state the old information with confidence.

The fifth is trusting document content just like user input. Every text that RAG puts into the prompt may be an instruction the model reads. If someone writes "ignore previous instructions" in a product review, that review enters the prompt as a chunk. The prompt injection we discussed in the tool calling posts applies here too.

A Small Experiment

★☆☆ Add an "International shipping" section to policies.md (e.g. "We ship to EU countries for 14 USD."). Run the program again and ask "Do you ship to Germany?". Does the model give the right answer now?

★★☆ Change Take(2) to Take(1) and to Take(4). How do the retrieved chunks and the answer change for "Can I pay for express shipping with a gift card?"? Note the input token count too.

★★☆ Delete the "If the answer is not in context..." line from the system prompt and ask "Do you ship to Germany?" and "Can I turn my loyalty points into cash?" again. What does the model make up?

★★★ Add a threshold: if the top retrieval score is below 0.65, print "I don't know based on our store policies." without calling the model at all. Using the five questions in this post, work out which questions the threshold filters correctly and which wrongly.

As you saw, RAG ties the model's answer to the right source, but it doesn't solve everything; the odd answer starting with "Otherwise..." and the cautious "I don't know"s are part of this system too. In the next post we will dig into why hallucinations happen, add source citations to the answer and verify those sources in code, and then compare our prompts with numbers using a small golden set.