Agentic RAG and GraphRAG: When Classic RAG Is Not Enough

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, OllamaSharp 5.5.2, Entity Framework Core 10 and PostgreSQL 17 + pgvector 0.8.7; the models are qwen3:4b (tool calling), gemma3:4b and nomic-embed-text on Ollama.

Guess First

A customer asks: "BKS-50518 kodlu ürünü indirdikten sonra iade edebilir miyim?" (can I return product BKS-50518 after downloading it?). The answer is spread across two documents: the product catalog says BKS-50518 is "Sesli Kitap: Kürk Mantolu Madonna" (an audiobook), and the return policy says audiobooks can't be returned once downloaded. What does the series' classic RAG (Retrieval-Augmented Generation) pipeline (hybrid search, top three chunks, gemma3:4b) say? And what does a model (qwen3:4b) do when it gets search as a tool and decides itself when and with what to search?

$ dotnet run -- classic "BKS-50518 kodlu ürünü indirdikten sonra iade edebilir miyim?"
Q: BKS-50518 kodlu ürünü indirdikten sonra iade edebilir miyim?
   retrieved: urun-katalogu#13, kurumsal-satis-sozlesmesi#3, iade-politikasi#2
   model calls: 1, input tokens: 323, output tokens: 31, 1,7s
   A: Kurumsal Satış Sözleşmesi'ndeki 4. maddeye göre, uygulama üzerinden indirilen sesli kitaplar iade edilemez.
$ dotnet run -- agentic "BKS-50518 kodlu ürünü indirdikten sonra iade edebilir miyim?"
Q: BKS-50518 kodlu ürünü indirdikten sonra iade edebilir miyim?
   search 1: BKS-50518 indirilen ürün iade
   search 2: indirilen ürünler iade politikası
   model calls: 3, input tokens: 3684, output tokens: 1540, 54,9s
   A: BKS-50518 kodlu ürünü indirdikten sonra iade edemezsiniz. Bookstore'un iade politikasına göre, e-kitaplar, sesli kitaplar ve dijital kodlar indirildikten veya etkinleştirildikten sonra iade edilemez. ...

Classic RAG reached the right conclusion by the wrong route: it found the product, missed the relevant section of the return policy, and attributed a sentence to clause 4 of the corporate contract that isn't there at all. The agent noticed the missing information after its first search, wrote a second query and based the answer on the right source. The price: 3 model calls, 11 times the input tokens, and 54.9 seconds instead of 1.7. This final post of the series asks about that trade-off: when is classic RAG not enough, and when it isn't, which of agentic RAG, GraphRAG or long context (putting everything into the prompt) really solves the problem?

Classic RAG's Single Shot

The pipeline we built in this series is a "single shot": the question is embedded once, searched once, and the result goes into the prompt once. That is the right design for the vast majority of questions whose answer lives in one place, and the measurements in post 12 show it. Three kinds of questions strain it, though. Multi-hop questions are those where part of the answer determines the query for a second search: first "what is BKS-50518?", then "can audiobooks be returned?". Global questions ask for a summary spread across the corpus: "in which cases do I get a refund?". And there are questions that need the whole context, more than fits in top-k. Each has a different tool.

Making Retrieval a Tool: Agentic RAG

The essence of agentic RAG is simple: you take search out of the pipeline as a fixed step and give it to the model as a tool. The logic of tool calling is in posts 11–12 of the LLM (Large Language Model) Fundamentals series, and the details of the agent loop are in the upcoming AI (Artificial Intelligence) Agents and MCP (Model Context Protocol) series. Here we only cover the retrieval side:

// Giving retrieval to the model as a tool: the model decides when and with what to search
public sealed class SearchTool(Retriever retriever)
{
    public List<string> Calls { get; } = [];
    [Description("Searches the Bookstore help center (policies, FAQ, product catalog, corporate sales contract). Returns the 3 most relevant sections.")]
    public async Task<string> SearchDocs([Description("A short Turkish search query about one topic")] string query)
    {
        Calls.Add(query);
        List<Candidate> hits = await retriever.HybridAsync(query, 3);
        return string.Join("\n\n", hits.Select(h => $"[{h.Id}]\n{h.Content}"));
    }
}
SearchTool tool = new(retriever);
IChatClient agent = new ChatClientBuilder(Models.Chat("qwen3:4b"))
    .UseFunctionInvocation(configure: f => f.MaximumIterationsPerRequest = 5)
    .Build();
response = await agent.GetResponseAsync(
[
    new ChatMessage(ChatRole.System, """
        You are the Bookstore help-center assistant. Answer in Turkish.
        Use the SearchDocs tool to find facts; search again with a different query if the first results are not enough.
        Answer only from the search results.
        """),
    new ChatMessage(ChatRole.User, question)
], new ChatOptions { Temperature = 0, Seed = 42, Tools = [AIFunctionFactory.Create(tool.SearchDocs)] });

The tool uses the hybrid search from the previous posts unchanged, which means the authorization filter from post 11 must stay inside the tool too. We don't leave user identity to the model's arguments; the code that creates the tool uses a DbContext built with the identity from the request's token. MaximumIterationsPerRequest = 5 matters as well: if the model never stops searching, we cut the loop. I asked the same agent two more questions:

Q: Okulumuz 120 adet kitap alacak; indirimimiz ne olur ve kredi kartıyla taksit yapabilir miyiz?
   search 1: okul kitap indirim kredi kartı taksit
   model calls: 2, input tokens: 2121, output tokens: 2560, 107,2s
   A: Okulumuz 120 adet kitap alırsa %15 indirim alırsınız. Kredi kartı ile taksit yapabilmek için toplam fiyat 750 TL ve üzeri olmalıdır.
Q: Kargo ücreti ne kadar?
   search 1: Kargo ücreti ne kadar?
   model calls: 2, input tokens: 1123, output tokens: 864, 20,7s
   A: Standart kargo ücreti 400 TL altındaki siparişler için **49,90 TL**dir. ...

The table shows agentic RAG isn't magic. On the second question the agent searched with a single combined query and applied the individual customers' instalment rule instead of the corporate contract's "no credit card instalments on corporate orders" clause. Classic RAG, on the same question, got the instalment part right and the discount rate wrong (the same 22% error as in post 10). The agent's real advantage shows only in the first question, a genuine two-step chain of information. The third question shows the agent's cost on a simple FAQ (Frequently Asked Questions) question: two model calls, three times the tokens and 20 times the time for the same answer. The timings reflect the long reasoning output qwen3:4b produces in thinking mode and a machine where other jobs share the same Ollama; look at the ratios, not the absolute seconds.

GraphRAG: Graphs and Community Summaries for Global Questions

Agentic RAG is one answer to multi-hop questions; but a question like "Hangi durumlarda para iadesi alırım, hangi durumlarda alamam?" (in which cases do I get a refund and in which don't I?) is a different challenge. The answer isn't in one chunk; it's spread across four sections: the return policy's "refund" and "damaged or wrong product" sections, the shipping page's "undeliverable parcels" section and the gift card's "validity" section (the balance of an expired card isn't refunded). I measured how much of that classic retrieval covers:

$ dotnet run -- global
top-3: 2/4 relevant  [iade-politikasi#3, iade-politikasi#2, kurumsal-satis-sozlesmesi#2]
top-5: 3/4 relevant  [iade-politikasi#3, iade-politikasi#2, kurumsal-satis-sozlesmesi#2, kargo-ve-teslimat#5, kurumsal-satis-sozlesmesi#0]
top-10: 3/4 relevant  [..., kurumsal-satis-sozlesmesi#3, personel-indirimi#0, iade-politikasi#4, iade-politikasi#0, kurumsal-satis-sozlesmesi#1]

Even the top 10 lacks the gift card section, because that section talks about the card's validity period, not "refunds"; similarity search doesn't find it close to a "refund" question. That is the typical fate of global questions: the relevant pieces don't resemble each other, they are connected to each other. Microsoft Research's GraphRAG (Edge et al., 2024) was designed for exactly this problem. At indexing time it splits the corpus into small text units, uses an LLM to extract entities, relationships and key claims from each unit, partitions the resulting graph into hierarchical communities with the Leiden algorithm, and has a summary written for each community bottom-up. On the query side there are four modes: "global search" over community summaries, "local search" that fans out to an entity's neighbours, DRIFT search that adds community information to local search, and "basic search", which is classic top-k.

To see the cost on a small scale, I ran GraphRAG's first step, entity and relationship extraction, over every public chunk in the index using gemma3:4b with structured output:

// GraphRAG's first step: LLM-based entity and relationship extraction for every chunk
public sealed class GraphExtractor(IChatClient chat)
{
    public async Task<(Extraction Result, long InputTokens, long OutputTokens)> ExtractAsync(string text)
    {
        ChatResponse<Extraction> r = await chat.GetResponseAsync<Extraction>(
        [
            new ChatMessage(ChatRole.System, """
                Extract the key entities and the relations between them from the bookstore policy text.
                Use short lowercase Turkish noun phrases for entities (e.g. "hediye kartı", "para iadesi", "ekspres teslimat").
                Relation types: applies_to, costs, takes, excludes, refunds_to, requires, allows.
                """),
            new ChatMessage(ChatRole.User, text)
        ], new ChatOptions { Temperature = 0, Seed = 42 });
        return (r.Result, r.Usage?.InputTokenCount ?? 0, r.Usage?.OutputTokenCount ?? 0);
    }
}

$ dotnet run -- graph
[kargo-ve-teslimat#5] entities: kargo, teslim edilemeyen gönderiler, adrese kimse, kargo firması, en yakın şube, paket, sipariş, ücret iadesi, kargo ücreti
   kargo -applies_to-> teslim edilemeyen gönderiler
   kargo firması -takes-> paket
   kargo firması -attempts-> teslim dener
   kargo firması -waits-> en yakın şube
   paket -refunds_to-> ücret iadesi
   sipariş -results_in-> iptal edilerek
   ücret iadesi -costs-> kargo ücreti
   kargo firması -excludes-> kargo ücreti
...
48 chunks, 379 distinct entities, 349 relations, input 7978 + output 14484 tokens, 338s
  iade: iade-politikasi#1, iade-politikasi#3, kurumsal-satis-sozlesmesi#3
  para iadesi: iade-politikasi#2, sss#3, urun-katalogu#13
  iade tutarı: iade-politikasi#2

The model extracted 349 relations and 379 distinct entities from 48 chunks; it spent 7,978 input and 14,484 output tokens and took 338 seconds. Note that output exceeds input: graph extraction is closer to rewriting the text than to reading it, and cloud providers usually charge more for output tokens. Quality is a problem too. I gave seven relation types in the prompt; the model invented its own, such as attempts, waits and results_in, and produced meaningless edges such as kargo firması -excludes-> kargo ücreti. The real problem is entity fragmentation: the same concept became four separate nodes, "para iadesi", "ücret iadesi", "iade tutarı" and "iade" (all variants of refund). Two of the three chunks linked to the "para iadesi" node (sss#3, urun-katalogu#13) have nothing to do with the global question, while the shipping and gift card sections aren't linked to this node at all. GraphRAG's value depends on merging these nodes correctly (entity resolution) and on the quality of community summaries; with a small local model even the first step isn't reliable.

And this is only the first step. Real GraphRAG merges entities (different spellings of the same concept), finds communities and generates a separate summary for each; so the number of calls exceeds the number of chunks. The project's own documentation warns "GraphRAG can consume a lot of LLM resources!" and recommends starting with small data and cheap models. LazyGraphRAG, announced by Microsoft Research in November 2024, reports skipping up-front summarisation entirely and bringing indexing cost down to the level of classic vector RAG (about 0.1% of full GraphRAG), moving the heavy work to query time. On our 48-chunk corpus this cost isn't worth it. Try cheaper routes for the global question first: an agent that searches the topics separately, or a reranker that filters a wider candidate pool. Measuring those is one of this post's exercises; remember that even the top 10 couldn't catch the gift card section.

Long Context: Putting Everything in the Prompt

gemma3:4b's context window is 128K tokens. Our entire public corpus is under 5,000 tokens. So why retrieve at all? I asked the same 10 questions two ways: classic RAG (top three chunks) and long context with the whole corpus in the system prompt. Since Ollama's default context is 4,096 tokens, I raised num_ctx to 16,384 for the long-context requests; otherwise, as we saw in the LLM Fundamentals post on context windows, the beginning of the prompt would be silently cut.

// Put the whole standard index into the prompt: no retrieval
List<SourceChunk> all = Corpus.LoadStandard().Where(c => c.Acl == "public").ToList();
string corpus = string.Join("\n\n", all.Select(c => $"<doc id=\"{c.Id}\">\n{c.Content}\n</doc>"));
...
ChatResponse r = await chat.GetResponseAsync(
[
    new ChatMessage(ChatRole.System, "You are the Bookstore help-center assistant. Answer in Turkish in one sentence, using only the documents.\n\n" + corpus),
    new ChatMessage(ChatRole.User, q.Question)
], new ChatOptions { Temperature = 0, Seed = 42, AdditionalProperties = new() { ["num_ctx"] = 16384 } });

$ dotnet run -- classic-batch
q01 + input=386 3,3s  Satın aldığın basılı kitabı teslim tarihinden itibaren 21 gün içinde iade edebilirsin.
q08 - input=324 1,3s  5.000 TL ve üzeri alışverişte Kitapsever seviyesine geçerek ... ücretsiz standart kargodan yararlanabilirsiniz.
q10 - input=452 1,3s  Ekspres teslimat yalnızca İstanbul, Ankara ve İzmir'in merkez ilçelerinde geçerli olduğu için ...
q11 - input=338 0,5s  Bu konuda dokümanlarda bilgi bulunmamaktadır.
q29 + input=379 0,8s  Belediye faturaları genellikle 60 gün içinde havale veya EFT ile ödenmelidir.
...
$ dotnet run -- longcontext
q01 + input=4794 5,7s  Basılı bir kitabı satın aldıktan sonra teslim tarihinden itibaren 21 gün içinde iade edebilirsiniz.
q08 + input=4797 0,5s  400 TL ve üzeri alışverişlerde standart kargo ücretsizdir.
q10 - input=4797 0,8s  Standart kargo ile siparişiniz 3 ila 5 iş günü içinde Van'ın bir köyüne teslim edilebilir.
q11 - input=4794 1,0s  Şu an için Almanya'ya kitap göndermekteyiz, ancak ekspres teslimat seçeneği yalnızca ...
q29 - input=4798 0,5s  Fatura tarihinden itibaren 30 gün vadeli ödeme yapabilirsiniz.
...

The program's automatic check (does the answer contain the expected phrase) was too strict here; it missed "3 ila 5" (3 to 5) because it was looking for "3-5". I graded the ten answers by hand: both approaches got 7 of 10 right, but on different questions. Long context answered q08 ("kargo bedava") and q10 (the region table) correctly, which retrieval had missed, because the right section was in the prompt. On the other hand it skipped the public-body 60-day exception in q29, which classic RAG got right, and in q11 it said "yes, we ship books to Germany" while the shipping page's "we only ship to KKTC (Northern Cyprus)" sentence sat right there in the prompt. Seeing everything doesn't mean looking at the right thing.

The cost side is clear: about 4,790 input tokens per question instead of 330–450, i.e. 12 times. Latency is better than expected: the first request took 5.7 seconds, the following ones 0.5–1.6 seconds. The reason is most likely that Ollama reuses the part a previous request already computed for prompts with the same prefix; since the corpus sits unchanged in the system prompt, only the question itself is reprocessed. Cloud providers' prompt caching features rest on the same idea, but pricing and expiry rules vary by provider; in this post I tried only a local model, not a cloud provider. At this scale long context really is a reasonable option: the corpus is small, it doesn't change and traffic is low. When the corpus grows to 50,000 pages, tenants and per-document permissions, the picture flips: sending the whole corpus with every question is expensive and makes the authorization filter from post 11 impossible.

Which One When?

Turning what we measured throughout the series into a decision list:

• Classic RAG (hybrid + rerank + citations): the vast majority of questions whose answer lives in one or a few sections. This should be the default; the others are added only for a failure class you have measured.
• Agentic RAG: genuine multi-hop questions, clarifying ambiguous questions, combining different sources (documents, an orders API (Application Programming Interface)). The cost is call count, latency and loss of control; a step limit, tool permissions and a trajectory log are mandatory.
• GraphRAG: "what does it say overall?" style global questions over a large, interconnected corpus. Indexing cost is high; not worth it if the corpus changes often and questions are mostly pinpoint.
• Long context: a small, stable, single-tenant corpus and low traffic. It removes retrieval errors but not reading errors, and you can't enforce permissions per document.

Production Notes

Log every step of the agent: which query it searched with, which chunks came back, how many iterations it took, how many tokens it spent. In classic RAG a failure was resolved with "wrong chunk or wrong reading?"; agentic RAG adds a third question: "did it write the wrong query?" The instalment error in the second question is exactly that. I re-ran the agent's single query ("okul kitap indirim kredi kartı taksit") through the same hybrid search: the top three are odeme-secenekleri#1 (the individual instalment rule), kurumsal-satis-sozlesmesi#0 and hediye-karti#2; kurumsal-satis-sozlesmesi#1, which holds the "no credit card instalments on corporate orders" sentence, stayed in fifth place and, since the tool returns only the top three, never reached the model. The details of evaluating agents (trajectory, tool call accuracy) are in the upcoming AI Agents and MCP series.

Set hard limits on cost and time: an iteration limit, a per-request token budget and a timeout. Reasoning models (here qwen3:4b in thinking mode) produce hundreds of tokens of thought at every step; most of the 2,560 output tokens in the second question were thinking, not answer.

Keep routing cheap. Instead of sending every question to the agent, answer most questions in a single shot with a simple classifier (or the confidence signals the measurements in post 12 showed), and send only low-confidence or multi-topic questions down the agentic path.

Common Pitfalls

Building agentic RAG for a simple FAQ bot. The shipping fee question spent 20 times the time and 3 times the tokens for the same answer. If most questions are single-shot, the agent is pure cost.

Leaving authorization to the agent. Putting a "user id" parameter on the tool undoes all the work from post 11. The tool takes identity from the request context; the model only chooses the query.

Building GraphRAG for a small or fast-changing corpus. Entity extraction and community summaries have to be regenerated on every document change; for a 48-chunk corpus that cost buys nothing.

Saying "the context window is big, no need for retrieval". On this set long context tied classic RAG at 7/10 and stated "we do" about something the document plainly says "we don't". Removing retrieval doesn't remove errors; it moves them.

A Small Experiment

★☆☆ Add the line "Search each topic separately" to the agent's system prompt and ask the 120 copies + instalments question again. Does the agent make two separate searches, and is the instalment part fixed?

★★☆ Ask the agent the global question and count how many searches it makes and how many of the four relevant sections it finds. Compare with a classic pipeline that answers the same question with top-10 + the LLM reranker.

★★☆ Repeat the long-context experiment without the num_ctx setting (default 4,096). How do the input token count and the answers change? Which documents are silently cut?

★★★ Normalise the entities in the graph output (e.g. "kredi kartı iadelerinin" → "kredi kartı"), build an adjacency list from the entity → chunk mapping, and for the global question fetch the chunks in the one- and two-step neighbourhood of the "para iadesi" entity. Can you catch the gift card section?

This post ends the RAG series. We started by classifying why minimal RAG breaks; we chose an embedding model, turned documents into clean text and chunked them, searched with pgvector and Entity Framework Core (EF Core), fixed ranking with hybrid search and reranking, answered with citations, put authorization into the query and measured all of it. In this post, the moment we handed retrieval to the model as a tool, we stopped writing a pipeline and started writing an agent. Building the agent loop from scratch, tool design, context engineering and exposing tools over the Model Context Protocol are the subject of the upcoming AI Agents and MCP series.