Measuring RAG: Recall@k, MRR, nDCG, Faithfulness and Groundedness

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI.Evaluation.Quality and .Reporting 10.11.0, xUnit v3 4.0.2 (on Microsoft.Testing.Platform), Entity Framework Core 10 and PostgreSQL 17 + pgvector 0.8.7; the models are gemma3:4b (answers), gemma3:12b (judge) and nomic-embed-text on Ollama.

Guess First

We have an LLM (Large Language Model) judge, gemma3:12b, score two answers through Microsoft's GroundednessEvaluator; the scale is 1–5 and anything below 4 counts as "failed". The first answer: "You can return the book within 30 days of delivery." The context says 21 days. The second answer, to the school asking about 120 books: "You get a 22% discount on orders of 200 copies and more." That sentence appears word for word in the contract, but it doesn't answer the question; the right rate for 120 copies is 15%. What does each one score?

q28          G=5 Rel=2 Ret=-  200 adet ve üzeri siparişlerde %22 indirim alırsınız.
q01-tampered G=3 Rel=2 Ret=5  Kitabı teslim tarihinden itibaren 30 gün içinde iade edebilirsin.

The wrong number of days scored 3 and counted as failed; good. The sentence that misses the question but "is written in the context" got a full 5 for groundedness, and only relevance caught it. Over the last four posts we saw every change improve one thing and break another. In this post we catch that with numbers instead of by eye: measuring retrieval separately from generation, computing recall@k, MRR (Mean Reciprocal Rank) and nDCG (normalized Discounted Cumulative Gain) in C#, looking at LLM-as-judge evaluators and their limits, and turning all of it into a test that runs on every commit.

The Evaluation Set: Question, Expected Source, Reference Answer

Everything starts with an evaluation set. The 30 questions we've used since the start of the series are exactly that: each line holds the question, the expected document, the expected section and a short reference answer.

{ "id": "q04", "question": "Param ne zaman hesabıma geri yatar?", "doc": "iade-politikasi", "section": "Para iadesi",
  "answer": "Ürün depoda kontrol edildikten sonra 5 iş günü içinde başlatılır; kredi kartında 10 iş gününü bulabilir." }

The expected source lets us measure retrieval independently of the model; the reference answer does the same for generation. Separating them is essential: for a wrong answer, first ask "did the right chunk get into the prompt?". If not, the problem is retrieval; if it did, it's the model or the prompt.

I defined relevance as graded rather than binary: the expected section is 2, other sections of the same document are 1, everything else is 0. For a refund question, retrieving the "damaged product" section of the return policy is a better mistake than retrieving the shipping page, and nDCG can tell the difference.

Computing Recall@k, MRR and nDCG in C#

Three metrics, three different questions. Recall@k: is the right chunk in the top k results? MRR: at what position is the right chunk, on average; first place scores 1, second 0.5, fifth 0.2. nDCG@k: graded relevance with a gain that shrinks logarithmically as rank grows, normalised against the ideal ordering.

// The result of one query: ordered chunk ids and the relevance grade of each id (0, 1, 2)
public sealed record RankedResult(string QuestionId, IReadOnlyList<string> Ranking, IReadOnlyDictionary<string, int> Relevance);
public static class RetrievalMetrics
{
    // Recall@k: is the most relevant (grade 2) chunk within the first k?
    public static double RecallAt(RankedResult r, int k) =>
        r.Ranking.Take(k).Any(id => r.Relevance.GetValueOrDefault(id) == 2) ? 1 : 0;
    // Reciprocal rank: the inverse of the most relevant chunk's position; 0 if absent
    public static double ReciprocalRank(RankedResult r)
    {
        int index = r.Ranking.ToList().FindIndex(id => r.Relevance.GetValueOrDefault(id) == 2);
        return index < 0 ? 0 : 1.0 / (index + 1);
    }
    // nDCG@k: graded relevance, gain decreasing with log2 as rank grows
    public static double NdcgAt(RankedResult r, int k)
    {
        double Dcg(IEnumerable<int> gains) => gains.Take(k)
            .Select((g, i) => (Math.Pow(2, g) - 1) / Math.Log2(i + 2)).Sum();
        double dcg = Dcg(r.Ranking.Select(id => r.Relevance.GetValueOrDefault(id)));
        double ideal = Dcg(r.Relevance.Values.OrderByDescending(g => g));
        return ideal == 0 ? 0 : dcg / ideal;
    }
    public static (double RecallAt1, double RecallAt5, double Mrr, double NdcgAt5) Summarize(IReadOnlyList<RankedResult> results) => (
        results.Average(r => RecallAt(r, 1)),
        results.Average(r => RecallAt(r, 5)),
        results.Average(ReciprocalRank),
        results.Average(r => NdcgAt(r, 5)));
}

The gain formula is 2^g - 1: 3 for grade 2, 1 for grade 1. So an exact hit is worth three times a neighbouring section of the same document. I ran the same 30 questions with the four retrieval methods we built in this series; the reranker calls go through a disk cache (explained below):

$ dotnet run -- retrieval
method          recall@1 recall@5    MRR  nDCG@5
vector             0,533    0,833  0,682   0,625
keyword            0,567    0,867  0,702   0,638
hybrid             0,500    0,867  0,671   0,646
hybrid+rerank      0,767    0,900  0,821   0,715
  worse after rerank: q14 RR 0,50 -> 0,20
  worse after rerank: q16 RR 1,00 -> 0,25
  worse after rerank: q20 RR 0,50 -> 0,17
elapsed: 112,4s

Hybrid trails vector and keyword on MRR but beats both on nDCG@5: it hits first place less often, but fills the top five better with neighbouring sections of the relevant document. Look at a single metric and you could draw two opposite conclusions. The reranker leads clearly on every metric; still, the per-question breakdown shows it broke three questions (q14, q16, q20). Don't ignore them just because the average went up; as we saw in the reranking post, each is a separate failure class. Vector MRR is 0.682; the in-memory measurement in the series' second post gave 0.668. The candidate list here is 20 results from pgvector's HNSW (Hierarchical Navigable Small World) index; that cutoff and the index are most likely the source of the small difference.

Measuring Generation: Microsoft.Extensions.AI.Evaluation

Retrieval metrics don't tell you the answer is right. There are two ways to measure the answer: compare it with the reference answer (NLP (Natural Language Processing) metrics such as BLEU (Bilingual Evaluation Understudy) and F1 (the harmonic mean of precision and recall), or an LLM's "same meaning?" verdict), or judge it against the context and the question. The Microsoft.Extensions.AI.Evaluation.Quality package ships ready-made evaluators for the latter; three stand out for RAG (Retrieval-Augmented Generation):

• GroundednessEvaluator: is the information in the answer present in the given context? Takes the context via GroundednessEvaluatorContext.
• RelevanceEvaluator: does the answer answer the question? Needs no extra context.
• RetrievalEvaluator: how relevant are the retrieved chunks to the question? Takes the chunk list via RetrievalEvaluatorContext.

Mapped to the concepts of the Ragas library: groundedness corresponds to Ragas's faithfulness (the share of the answer's claims supported by the context), and the retrieval evaluator is roughly an LLM-judged version of context precision. Ragas's context recall checks whether the information in the reference answer is found in the retrieved context; our recall@k answers the same question against a labelled source, without an LLM.

// gemma3:4b answers, gemma3:12b judges; both models' responses are in the disk cache
ReportingConfiguration generation = DiskBasedReportingConfiguration.Create(
    storageRootPath: store, evaluators: [], chatConfiguration: new ChatConfiguration(Models.Chat("gemma3:4b")),
    enableResponseCaching: true, executionName: "generation");
ReportingConfiguration judge = DiskBasedReportingConfiguration.Create(
    storageRootPath: store,
    evaluators: [new GroundednessEvaluator(), new RelevanceEvaluator(), new RetrievalEvaluator()],
    chatConfiguration: new ChatConfiguration(Models.Chat("gemma3:12b")),
    enableResponseCaching: true, executionName: "judge");
await using ScenarioRun generator = await generation.CreateScenarioRunAsync("Answer.Generation");
foreach (EvalQuestion q in questions.Where(q => ids.Contains(q.Id)).Append(Tampered(questions[0])))
{
    List<Candidate> top = await retriever.HybridAsync(q.Question, 3);
    string context = string.Join("\n\n", top.Select(c => c.Content));
    List<ChatMessage> messages =
    [
        new(ChatRole.System, "You are the Bookstore help-center assistant. Answer in Turkish, briefly, using only the context."),
        new(ChatRole.User, $"Context:\n{context}\n\nQuestion: {q.Question}")
    ];
    ChatResponse response = q.Id.EndsWith("-tampered")
        ? new ChatResponse(new ChatMessage(ChatRole.Assistant, q.Answer))
        : await generator.ChatConfiguration!.ChatClient.GetResponseAsync(messages, new ChatOptions { Temperature = 0, Seed = 42 });
    await using ScenarioRun run = await judge.CreateScenarioRunAsync($"Rag.{q.Id}");
    EvaluationResult result = await run.EvaluateAsync(messages, response,
    [
        new GroundednessEvaluatorContext(context),
        new RetrievalEvaluatorContext(top.Select(c => c.Content))
    ]);
    ...
}

I chose a judge larger than the model producing the answers; a model judging its own answer tends to find its own mistakes "reasonable", which is worth avoiding. I also added a deliberately broken answer to the list: 30 days instead of 21 for q01. The result:

$ dotnet run -- generation -v
q01          G=5 Rel=4 Ret=5  Satın aldığınız basılı kitabı teslim tarihinden itibaren 21 gün içinde iade edebilirsiniz.
q04          G=2 Rel=3 Ret=1  Siparişin kargoya verilmeden önce iptal edebilirsiniz. Kargoya verilmiş siparişler iptal e
   reason(G): The response doesn't answer the question about when the money will be returned. It only discusses order cancellation policies.
q07          G=5 Rel=4 Ret=5  400 TL altındaki siparişlerde 49,90 TL.
q09          G=5 Rel=4 Ret=5  Evet, İstanbul, Ankara ve İzmir'in merkez ilçelerinde saat 12:00'ye kadar verilen siparişl
q13          G=5 Rel=4 Ret=5  Hediye kartları satın alındığı tarihten itibaren 3 yıl geçerlidir.
q17          G=5 Rel=4 Ret=5  Her 100 puandan 25 TL indirim sağlanır.
q21          G=5 Rel=4 Ret=-  750 TL ve üzeri alışverişlerde anlaşmalı bankaların kredi kartlarına 3 taksit imkânı vardı
q24          G=5 Rel=4 Ret=-  Ankara Kızılay mağazamızdan ücretsiz teslim alma seçeneği mevcut.
q28          G=5 Rel=2 Ret=-  200 adet ve üzeri siparişlerde %22 indirim alırsınız.
q29          G=5 Rel=4 Ret=5  Kamu kurumları için ödeme süresi 60 gündür.
q01-tampered G=3 Rel=2 Ret=5  Kitabı teslim tarihinden itibaren 30 gün içinde iade edebilirsin.
   reason(G): The response provides an incorrect timeframe for returning a book, contradicting the information provided in the context. ...
elapsed: 687,6s

q04 is the most honest row in the table: retrieval brought the wrong chunks (Ret=1) and the model talked about order cancellation from them. Groundedness's reasoning is really doing relevance's job, but the judge saw the problem correctly. Three rows show Ret=-; the value is null. The diagnostics in the result file say why:

"Failed to parse score for 'Retrieval' from the following evaluation response:
<S0>Let's think step by step:
The query asks \"Taksit yapıyor musunuz?\" which translates to \"Do you offer installments?\". The"

The evaluators' prompts expect a specific output format; according to Microsoft's documentation these evaluators were tested on Microsoft's Copilot experiences. A local 12-billion-parameter model failed to hit that format in three of ten attempts. Count the nulls separately so that a missing score doesn't silently enter the average.

The Limits of LLM-as-Judge

Back to the two lines from the opening. The made-up 30 days was caught, but only scored 3, not 1. The wrong answer in q28 got 5 for groundedness, because the sentence really is written in the context. By definition groundedness asks "is what you said in the source?", not "is it right as applied to the question?". In this run relevance (2) caught it, but don't trust that blindly either: the same relevance gave q09's correct but long answer a 4, not a 5.

That's why you need to measure the judge too. A practical way is to make a few deliberately broken answers (wrong number, wrong source, invented condition) a permanent part of the evaluation set. If the judge can't catch those, you can't trust its 5s either. And always keep LLM-free signals such as labelled sources and reference answers alongside: no judge can fool recall@k.

Regression in CI: xUnit and the Response Cache

Running the measurements once and looking at the table isn't enough; when you change chunking, the embedding model or the prompt, you should learn that the score dropped from a failing test. Since LLM calls are slow and expensive, the key piece here is the response cache: DiskBasedReportingConfiguration stores every request to the model (messages, options, model) as a key and returns the answer from disk when the same request comes again.

public sealed class RagQualityTests
{
    static readonly IReadOnlyList<EvalQuestion> Questions = Corpus.Questions();
    static readonly List<SourceChunk> Chunks = Corpus.LoadStandard();
    static readonly ReportingConfiguration Judge = DiskBasedReportingConfiguration.Create(
        storageRootPath: Path.Combine(Path.GetDirectoryName(Corpus.Root)!, "eval-store"),
        evaluators: [new GroundednessEvaluator()],
        chatConfiguration: new ChatConfiguration(Models.Chat("gemma3:12b")),
        enableResponseCaching: true,
        executionName: Environment.GetEnvironmentVariable("EVAL_EXECUTION") ?? "local");
    [Fact]
    public async Task Hybrid_retrieval_does_not_regress()
    {
        ...
        var summary = RetrievalMetrics.Summarize(results);
        // Thresholds sit slightly below today's measurement: the test breaks if chunking or the model changes
        Assert.True(summary.RecallAt5 >= 0.80, $"recall@5 = {summary.RecallAt5:F3}");
        Assert.True(summary.Mrr >= 0.60, $"MRR = {summary.Mrr:F3}");
    }
    [Theory]
    [InlineData("q01")]
    [InlineData("q13")]
    [InlineData("q21")]
    public async Task Answer_is_grounded_in_retrieved_context(string questionId)
    {
        ...
        CancellationToken ct = TestContext.Current.CancellationToken;
        await using ScenarioRun run = await Judge.CreateScenarioRunAsync($"Grounded.{questionId}", cancellationToken: ct);
        // The same scenario's chat client: the answer also comes from the cache, so CI doesn't repeat the model call
        ChatResponse response = await run.ChatConfiguration!.ChatClient.GetResponseAsync(messages, new ChatOptions { Temperature = 0, Seed = 42 }, ct);
        EvaluationResult result = await run.EvaluateAsync(messages, response, [new GroundednessEvaluatorContext(context)], ct);
        NumericMetric groundedness = result.Get<NumericMetric>(GroundednessEvaluator.GroundednessMetricName);
        Assert.False(groundedness.Interpretation!.Failed, $"{questionId}: {groundedness.Value} - {groundedness.Reason}");
    }
}

I ran the tests twice. The first run really calls the judge, the second reads from the cache:

$ dotnet test
Test run summary: Passed!
  total: 4
  failed: 0
  succeeded: 4
  skipped: 0
  duration: 1m 42s 115ms
$ dotnet test
Test run summary: Passed!
  total: 4
  failed: 0
  succeeded: 4
  skipped: 0
  duration: 1s 128ms

From 1 minute 42 seconds to 1.1 seconds. That's what makes LLM-based tests runnable on every commit in a CI (Continuous Integration) pipeline. Because the cache key is built from the request content, changing retrieval, chunking or the prompt changes the prompt, misses the cache and really calls the model again; unchanged tests stay free. One note: with the .NET 10 SDK (Software Development Kit), the old VSTest mode of dotnet test doesn't work with Microsoft.Testing.Platform, which xUnit v3 uses; my first attempt failed with "Testing with VSTest target is no longer supported", and I fixed it by adding "test": { "runner": "Microsoft.Testing.Platform" } to global.json. In CI you need to keep the cache folder as an artifact and restore it for the next run; the mechanics are in the evaluation post of the upcoming AI (Artificial Intelligence) with .NET series.

Production Notes

The cache cuts both ways. The good side is above; the bad side is that if the provider updates the weights behind the same model name, cached tests won't notice. Give the cache a TTL (Time To Live) (timeToLiveForCacheEntries), add information such as the model or prompt version to the key with cachingKeys, and do an uncached run once a week.

Offline measurement doesn't represent production. 30 questions don't reflect the distribution of questions real users write. Pull sample questions from production logs (after permission and personal-data cleanup), label them and add them to the set. Collect online signals too: if the user rephrased the same question after an answer, clicked "not helpful" or opened a support ticket, that question is a candidate for the set.

Store scores together with versions. "recall@5 0.867" means nothing on its own; which chunking, which embedding model and which candidate count it was measured with must sit next to it. executionName exists exactly for that.

Common Pitfalls

Generating the evaluation set automatically from the documents and memorising it. When you ask an LLM to "write a question from this chunk", the question usually carries the chunk's words and makes keyword search look unfairly good. Questions in our set such as "the book arrived torn" or "kargo bedava" (free shipping) represent real user language that doesn't use the document's words, and the most instructive failures came from them.

Looking at a single average. Hybrid's MRR was low while its nDCG was high; the reranker raised the average while breaking three questions. Always produce the per-question breakdown as well and review the "got worse" list separately.

Counting the judge's parse failures as zero, or ignoring them. If null scores don't enter the average, it looks better than it is; if they count as zero, worse. Report them with a separate counter.

Judging with the model that produced the answer. The same model is likely to find the same mistake "reasonable". If you can, use a larger judge or one from a different family, and test the judge with broken answers.

A Small Experiment

★☆☆ Add PrecisionAt(k) to RetrievalMetrics: the share of the top k results with grade 1 or 2. Is the difference between hybrid and hybrid+rerank larger than at recall@5?

★★☆ Add five more deliberately broken answers to the evaluation set (wrong fee, wrong duration, a condition from another document, an invented country, a missing exception). How many does the gemma3:12b judge mark as failed?

★★☆ Split chunks at 120 tokens instead of 300 during ingestion (or drop the heading prefix) and run the Hybrid_retrieval_does_not_regress test. Does it fail, and which metric drops by how much?

★★★ Compare every answer with the reference answer in the set using EquivalenceEvaluator and put the result next to groundedness. Which questions do they disagree on (grounded but wrong, or right but outside the context)?

We can now say with numbers whether a change is good or bad. In the last post of the series we look at the limits these measurements reveal: multi-step questions, "global" questions that span the whole corpus, and cases where giving the model everything at once beats RAG. Agentic RAG, GraphRAG and long context: when should they replace classic RAG?