Production RAG Architecture: Ingestion and Query Pipelines, and Why Minimal RAG Breaks

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0 and OllamaSharp 5.5.2, against local models (nomic-embed-text, gemma3:4b) on Ollama 0.34.4.

Guess First

In the LLM (Large Language Model) Fundamentals series we built RAG (Retrieval-Augmented Generation) over a six-paragraph policy file (the post "LLMs and RAG: Answering with Data the Model Has Never Seen"): split it into chunks, embed them, put the two closest chunks into the prompt. It said 21 days for the return question and "I don't know" for the Germany question; everything looked fine.

Now we point the same code at Bookstore's real document folder: five policy files, an HTML (HyperText Markup Language) page exported from the help center, the product catalog, a two-page PDF contract, two staff-only documents and a forgotten old shipping policy. Same code, same model; only the data became real. How many of 30 customer questions will get a correct answer? Write your guess down as a number, then keep reading.

Moving Minimal RAG onto a Real Document Folder

The document set we use throughout this series is Turkish and belongs to Bookstore. Customers ask in Turkish and the policies are written in Turkish, so the data and questions in the examples stay Turkish; I translate the important parts in the text. The folder looks like this:

corpus/
  policies/iade-politikasi.md        acl: public   updated_at: 2026-09-01
  policies/kargo-ve-teslimat.md      acl: public   updated_at: 2026-10-15
  policies/hediye-karti.md, sadakat-programi.md, odeme-secenekleri.md
  help/sss.html                      (help center export: menu, cookie banner, footer)
  products/urun-katalogu.md          14 products, one "## Title (BKS-xxxxx)" section each
  contracts/kurumsal-satis-sozlesmesi.pdf   2 pages, header/footer on every page
  internal/personel-indirimi.md      acl: staff
  internal/tedarikci-iade-sureci.md  acl: staff
  archive/kargo-ve-teslimat-2025.md  old version: free shipping from 300 TL

Every Markdown file starts with a small metadata block: id, title, acl (Access Control List; here just public or staff) and updated_at. Minimal RAG uses none of them; as in LLM Fundamentals, it turns every paragraph into a chunk prefixed with its section name, keeps them all in memory and puts the closest two into the prompt. The whole folder became 64 chunks. The class is almost exactly the old one:

namespace Bookstore.Rag;

// The minimal RAG from LLM Fundamentals 14, now over Bookstore's whole document folder
public sealed class MinimalRag(InMemoryIndex index, IChatClient chat, int topK = 2)
{
    public async Task<(List<Hit> Hits, string Answer)> AskAsync(string question)
    {
        List<Hit> hits = await index.SearchAsync(question, topK);
        string context = string.Join("\n\n", hits.Select(h => h.Chunk.Content));
        List<ChatMessage> messages =
        [
            new(ChatRole.System, """
                You are the Bookstore support assistant. Answer in Turkish.
                Answer using ONLY the facts inside <context>.
                If the answer is not in <context>, reply exactly: "Bu konuda bilgim yok."
                """),
            new(ChatRole.User, $"<context>\n{context}\n</context>\n\nQuestion: {question}")
        ];
        ChatResponse response = await chat.GetResponseAsync(messages, new ChatOptions { Temperature = 0, Seed = 42 });
        return (hits, response.Text.Trim());
    }
}

The only differences are that we ask for a Turkish answer and the "I don't know" sentence is Turkish ("Bu konuda bilgim yok."). Temperature 0, seed 42; the same question gets the same answer on every run, which makes the failures reproducible.

30 Questions, Five Failure Classes

Along with the documents I wrote a 30-question set. Every answer is in the documents, and for every question the set records which section of which document holds it. Later posts use that record to measure recall; today we only look at the answers. I ran the 30 questions one by one and classified each answer by hand:

20 of 30 correct: two thirds. Half of the failures come from retrieval, meaning the right text never reached the model. Outside the question set I tried two more failure classes, because random questions rarely hit them yet they are the most expensive ones in production: stale data and unauthorized documents.

Looking at the Failures One by One

Retrieval missed, and a chunk boundary

$ dotnet run -- minimal "Kaç liralık alışverişte kargo bedava oluyor?"
  [0.694] sss#10 (public) Stokta olmayan bir kitabı nasıl alabilirim?: Ürün sayfasındaki "Gelinc...
  [0.685] hediye-karti#22 (public) Kısıtlamalar: Kaybolan veya çalınan hediye kartları, satın alma fatura...
  => Bu konuda bilgim yok.

$ dotnet run -- minimal "Param ne zaman hesabıma geri yatar?"
  [0.658] kurumsal-satis-sozlesmesi#3 (public) 2. Ödeme koşulları: Kurum, fatura tarihinden itibaren 30 gün vadeli öd...
  [0.654] iade-politikasi#28 (public) Para iadesi: Hediye kartıyla ödenen siparişlerde iade tutarı hediye ka...
  => Ödeme havale veya EFT ile yapıldığında, iade tutarı hediye kartına yüklenir.

The first question asks "from what order amount is shipping free?" The answer is spelled out in the kargo-ve-teslimat document: standard shipping is free ("ücretsiz") from 400 TL. But the customer said "bedava" (the colloquial word for free) and the document says "ücretsiz"; the embedding did not consider them close enough, and an FAQ (Frequently Asked Questions) entry about out-of-stock books came out on top. The model behaved correctly: the answer was not in front of it, so it said it didn't know. For the customer the result is still bad.

The second one ("when will my money come back?") is sneakier. The refund section has two paragraphs: the first says the refund starts within 5 business days, the second covers orders paid with a gift card. Paragraph-based chunking split the section in two, retrieval brought the second half, and the model glued two unrelated sentences into a rule that does not exist. The answer is confident and wrong, which is worse than "I don't know".

Stale data and an unauthorized document

$ dotnet run -- minimal "İstanbul'da aynı gün teslimat yapıyor musunuz?"
  [0.698] kargo-ve-teslimat-2025#1 (public) Ekspres teslimat: Ekspres teslimat İstanbul'da geçerlidir. Saat 14:00'...
  [0.690] kargo-ve-teslimat#33 (public) Ekspres teslimat: Ekspres teslimat yalnızca İstanbul, Ankara ve İzmir'...
  => Saat 14:00'ye kadar verilen siparişler aynı gün teslim edilir. Ekspres teslimat ücreti 69,90 TL'dir.

$ dotnet run -- minimal "Çalışanlarınıza indirim yapıyor musunuz?"
  [0.711] sadakat-programi#47 (public) Puan kullanmak: Her 100 puan bir sonraki siparişte 25 TL indirim olara...
  [0.701] personel-indirimi#14 (staff) Personel indirimi: Bookstore çalışanları kendi kullanımları için yapac...
  => Evet, Bookstore çalışanları kendi kullanımları için yapacakları alışverişlerde %30 personel indirimi alır.

Same-day delivery in Istanbul existed a year ago; the new policy says next business day. Both documents are in the index, and the old one's score of 0.698 beat the new one's 0.690. The model took the first one and told the customer a cut-off time (14:00) and a fee (69,90 TL) that no longer apply. An embedding has no notion of "which one is current"; it is only asked about textual similarity.

The second answer ("do you give your employees a discount?" "Yes, 30%") is a security incident: the personel-indirimi document is marked acl: staff and literally says it must not be shared with customers. Minimal RAG never reads that metadata, so the document ended up in the customer's context and the model did its job.

The model misread the context

$ dotnet run -- minimal "Okulumuz için 120 adet kitap alırsak ne kadar indirim alırız?"
  [0.743] kurumsal-satis-sozlesmesi#2 (public) 1. Toplu alım indirimi: Aynı siparişte aynı kitaptan en az 50 adet ala...
  [0.696] sadakat-programi#47 (public) Puan kullanmak: Her 100 puan bir sonraki siparişte 25 TL indirim olara...
  => 200 ve üzeri siparişlerde indirim oranı %22'dir. Bu durumda 120 adet kitap için %22 indirim alırsınız.

Retrieval is flawless: the right section is on top with 0.743. The contract says "15% for 50-199 copies, 22% for 200 and above"; for a school ordering 120 copies, the model answered 22%. Expecting arithmetic slips from a 4-billion-parameter model is fair, but the same class also caught "can I turn my points into money?": the chunk says points cannot be converted to cash ("nakde çevrilemez"), and the model failed to connect "money" with "cash" and said it didn't know.

Two Pipelines: Ingestion and Query

All five failures seem to happen inside one function, but they actually come from different steps of two separate processes. In production RAG, separating them means knowing where to look for a failure.

The ingestion pipeline runs offline: it reads documents from their source, cleans them, extracts metadata, splits them into chunks, embeds them and writes them to the vector store. It can take minutes rather than seconds and runs as a background job, periodically or when a document changes. Stale data (3) and chunk boundary (2) failures are born here: it is ingestion that indexes the archive folder or leaves the vectors of a deleted document behind.

The query pipeline runs on every request and its latency budget is measured in seconds: embed the question, retrieve with a filter, rerank the candidates, skip the model entirely if the score is low, put the survivors into the prompt with their ids, generate the answer and validate the citations. Retrieval misses (1), unauthorized documents (4) and model misreads (5) happen here.

The practical consequence: in minimal RAG every question could have re-embedded all documents and nobody would notice. In a real system ingestion has its own schedule, its own error handling and its own metrics, and the query pipeline trusts the index ingestion left behind.

The Skeleton of the Query Pipeline

The rest of the series deepens every step separately, but the skeleton is clear already. The class below uses the in-memory index; we move to a vector database in the seventh post. The only change on the ingestion side is that the standard chunker now turns a whole section under a Markdown heading into one chunk instead of a paragraph, and it no longer indexes the archive/ folder:

public sealed record Caller(string UserId, string Tenant, IReadOnlyList<string> Roles);

public sealed record RagAnswer(string Text, IReadOnlyList<string> Sources, RagTrace Trace);

public sealed record RagTrace(string Question, IReadOnlyList<string> Retrieved, IReadOnlyList<float> Scores,
    bool AnsweredByModel, long RetrieveMs, long GenerateMs);

public sealed class QueryPipeline(InMemoryIndex index, IChatClient chat, float minScore = 0.70f, int topK = 3)
{
    public async Task<RagAnswer> AskAsync(string question, Caller caller)
    {
        Stopwatch sw = Stopwatch.StartNew();

        // 1. Retrieve: the permission filter goes INSIDE the search, not after it
        List<Hit> hits = await index.SearchAsync(question, topK,
            c => c.Tenant == caller.Tenant && (c.Acl == "public" || caller.Roles.Contains(c.Acl)));
        long retrieveMs = sw.ElapsedMilliseconds;

        // 2. Rerank and threshold: only a score threshold for now (reranker in 09)
        List<Hit> usable = hits.Where(h => h.Score >= minScore).ToList();
        if (usable.Count == 0)
        {
            RagTrace empty = new(question, hits.Select(h => h.Chunk.Id).ToList(),
                hits.Select(h => h.Score).ToList(), false, retrieveMs, 0);
            return new("Bu konuda bilgim yok.", [], empty);
        }

        // 3. Prompt: every chunk with its id, so the answer can cite
        string context = string.Join("\n\n", usable.Select(h => $"[{h.Chunk.Id}]\n{h.Chunk.Content}"));
        List<ChatMessage> messages =
        [
            new(ChatRole.System, """
                You are the Bookstore support assistant. Answer in Turkish.
                Answer using ONLY the facts inside <context>. Cite the ids you used like [doc#n].
                If the answer is not in <context>, reply exactly: "Bu konuda bilgim yok."
                """),
            new(ChatRole.User, $"<context>\n{context}\n</context>\n\nQuestion: {question}")
        ];

        // 4. Generate
        sw.Restart();
        ChatResponse response = await chat.GetResponseAsync(messages, new ChatOptions { Temperature = 0, Seed = 42 });
        RagTrace trace = new(question, usable.Select(h => h.Chunk.Id).ToList(),
            usable.Select(h => h.Score).ToList(), true, retrieveMs, sw.ElapsedMilliseconds);

        // 5. Citations: accept only ids we actually supplied
        List<string> sources = usable.Select(h => h.Chunk.Id).Where(id => response.Text.Contains(id)).ToList();
        return new(response.Text.Trim(), sources, trace);
    }

    public static string ToLogLine(RagTrace trace) => JsonSerializer.Serialize(trace, LogJson);

    private static readonly JsonSerializerOptions LogJson = new() { Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
}

Two lines matter architecturally. The first is passing the permission filter into SearchAsync: unauthorized chunks never become candidates, so all k results come from documents the user is allowed to see. The second is RagTrace: logging, as one JSON (JavaScript Object Notation) line per request, which chunks came back with which scores and whether the model was called. When a wrong answer shows up, the first question is "retrieval or model?", and this line answers it.

I first set the score threshold to 0.70, because in minimal RAG 15 of the 20 correctly answered questions had a top score above 0.70:

$ MIN_SCORE=0.70 dotnet run -- pipeline customer "Kaç liralık alışverişte kargo bedava oluyor?"
Bu konuda bilgim yok.
sources: 
{"Question":"Kaç liralık alışverişte kargo bedava oluyor?","Retrieved":["sss#2","kargo-ve-teslimat#5","kargo-ve-teslimat#1"],"Scores":[0.6895946,0.67578393,0.6729067],"AnsweredByModel":false,"RetrieveMs":26,"GenerateMs":0}

On the question retrieval had missed, the threshold said "I don't know" without calling the model and saved a model call of a few seconds. But the same threshold skipped the model for all five of the first questions I tried; in one of them, the same-day delivery question, the right section was on top with 0.677. When the chunker changed, the score distribution shifted too. More importantly, 7 of the 10 questions minimal RAG got wrong also had a top score above 0.70: a high score does not mean the right chunk arrived. You cannot pick a threshold once and forget it; in the tenth post we calibrate it with a measurement set. For now, 0.60:

$ MIN_SCORE=0.60 dotnet run -- pipeline customer "Çalışanlarınıza indirim yapıyor musunuz?"
Kurumsal Satış Sözleşmesi'ne [kurumsal-satis-sozlesmesi#0] göre, aynı siparişte aynı kitaptan en az 50 adet alan Kurum, liste fiyatı üzerinden %15 indirim alır. 200 adet ve üzeri siparişlerde ise indirim oranı %22'dir.
sources: kurumsal-satis-sozlesmesi#0
{"Question":"Çalışanlarınıza indirim yapıyor musunuz?","Retrieved":["sadakat-programi#1","kurumsal-satis-sozlesmesi#0","sadakat-programi#2"],"Scores":[0.69410163,0.66159636,0.6609422],"AnsweredByModel":true,"RetrieveMs":18,"GenerateMs":3222}

$ MIN_SCORE=0.60 dotnet run -- pipeline staff "Çalışanlarınıza indirim yapıyor musunuz?"
Evet, çalışanlarımız kendi kullanımları için alışverişlerde %30 personel indirimi alırlar [personel-indirimi#0].
sources: personel-indirimi#0
{"Question":"Çalışanlarınıza indirim yapıyor musunuz?","Retrieved":["personel-indirimi#0","sadakat-programi#1","personel-indirimi#2"],"Scores":[0.7003257,0.69410163,0.6704573],"AnsweredByModel":true,"RetrieveMs":18,"GenerateMs":1213}

Same question, two different users. The customer's search never saw the personel-indirimi chunk; the model described the closest thing it had, the corporate bulk discounts. Not an ideal answer, but not a leak either. The user with the staff role got the 30%, with its source.

$ MIN_SCORE=0.60 dotnet run -- pipeline customer "İstanbul'da aynı gün teslimat yapıyor musunuz?"
Evet, İstanbul'da ekspres teslimat ile aynı gün teslimat yapıyoruz [kargo-ve-teslimat#1]. Saat 12:00'ye kadar verilen siparişler ertesi iş günü teslim edilir.
sources: kargo-ve-teslimat#1
{"Question":"İstanbul'da aynı gün teslimat yapıyor musunuz?","Retrieved":["kargo-ve-teslimat#1","kargo-ve-teslimat#2","sss#4"],"Scores":[0.6768913,0.6651281,0.65260315],"AnsweredByModel":true,"RetrieveMs":96,"GenerateMs":2138}

This is where we need to be honest. With the archive out of the index the old 14:00 rule no longer appears, and retrieval found the right document. But the model said "Yes, same day" and then copied the "next business day" sentence from the context: it contradicted itself. The architecture closed the unauthorized-document and stale-data classes structurally; retrieval misses and model misreads remain. Those are fixed with measurement and better parts, not with architecture.

RAG, Long Context or Fine-Tuning?

This whole document set is 4,184 tokens with the o200k_base tokenizer. Most models today fit that into a single prompt. So why retrieval? Because the real Bookstore's documents do not stop at 14 products: thousands of product descriptions, hundreds of contracts, campaigns that change every week. Putting all of it into every prompt ties the cost and latency of each request to the size of the document set, and the model's tendency to miss information in the middle of a long context is a separate risk. For a small, rarely changing document set, "put everything in the prompt" is a perfectly reasonable start; we compare it with measurements in the thirteenth post.

Fine-tuning answers a different question. It is good for teaching a model a tone, a format or a narrow task; it does not reliably place a changing fact such as "the shipping threshold went from 300 to 400" into the model, and it needs retraining on every policy change. For information that changes, is subject to permissions and must be cited, RAG is the right tool, but as we just saw, only if each part is done well.

The Series Roadmap

Each post maps to one of the failure classes we saw today:

• Retrieval missed: choosing an embedding model (02), hybrid search (08), reranking and query transformation (09).
• Wrong chunk boundary: document ingestion (03) and chunking strategies (04).
• Stale data: incremental updates and removing deleted documents from the index (03).
• Unauthorized document: filtering in the vector database (05, 07) and authorization and security in RAG (11).
• Model misread the context: citations, context budget and the "I don't know" threshold (10), measurement (12).

Choosing a vector database (05) and vector search with EF (Entity Framework) Core (06, 07) sit in between, because there is no point discussing filters and indexes before moving the in-memory list to production. The last post covers where classic RAG is not enough: agentic RAG and GraphRAG.

Production Notes

Do not improve without measuring. The 30-question set in this post is small but classified by hand; only with a set like this can you see whether a change helped. We saw the score distribution shift when the chunker changed; without a measurement set we would have learned that from a customer complaint.

Log retrieval and generation separately. In the RagTrace lines, RetrieveMs is between 18 and 96 ms and GenerateMs between 1.2 and 3.2 seconds. Almost all latency is in the model call; the effort you spend speeding up retrieval is small next to what you gain by not calling the model at all on a low-score question.

Put the permission filter into retrieval, not into the prompt. Writing "do not show these documents to the user" in the system prompt means placing the unauthorized text in front of the model and asking it to keep quiet. When the filter is inside the query, the unauthorized text is never loaded.

Design ingestion as a separate process. Which folders go into the index, when each document was last embedded and with which embedding model should be recorded somewhere; a stale-data incident often starts with "who added that folder?"

Common Pitfalls

First, deciding with a demo document. On the six-paragraph policy file minimal RAG was flawless; on the real folder it missed 10 of 30 questions. Evaluate with your own data and the words your users actually use ("bedava", "turn into money").

Second, blaming every wrong answer on the model. Six of the ten answers that were not correct came from retrieval or chunking; a bigger model does not fix them, because the right text never reaches the model.

Third, leaving metadata for later. The acl and updated_at fields were in the documents, but they could not be used because they were not carried into the chunks. Store metadata with the chunk from the first day of ingestion; adding it later means reprocessing everything.

Fourth, copying a threshold from another system. 0.70 was reasonable with the paragraph chunker and eliminated everything with the section chunker. Scores vary with the embedding model, the chunk size and the language.

Fifth, counting "I don't know" as success. On the question retrieval missed, the model behaved correctly and said it didn't know, but the answer was in the documents. In measurement that is a failure, not good behavior.

A Small Experiment

★☆☆ Raise MinimalRag's topK from 2 to 4 and ask "Param ne zaman hesabıma geri yatar?" (when will my money come back?) again. Do both halves of the refund section arrive, and does the answer get fixed? Note how many tokens the prompt grew by.

★★☆ Add a document date rule to QueryPipeline: of two documents with the same title, only chunks of the one with the newer updated_at become candidates. Put the archive folder back into the index and try the same-day delivery question.

★★★ Run the 30 questions through QueryPipeline with thresholds of 0.55, 0.60 and 0.65. At each threshold, how many correct answers were dropped before the model, and how many wrong answers were prevented? Choose the threshold from the balance of those two numbers.

In this post we saw the failure classes and built the skeleton. In the next one we go back to the most basic part, the embedding model: how good nomic-embed-text is on Turkish questions, how prefixes and vector dimensions change the result, and what happens to the old vectors when you want to switch models.