Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
Measuring RAG: Recall@k, MRR, nDCG, Faithfulness and Groundedness
Measuring RAG means measuring retrieval against labelled sources (recall@k, MRR, nDCG) and generation with an LLM judge (groundedness, relevance) separately. We compare four retrieval methods on 30 questions in C#, show Microsoft.Extensions.AI.Evaluation giving 5 to an answer that misses the question, and cut a 1 min 42 s test to 1.1 s with response caching.
Evaluating Agents: Trajectory, Tool Call Accuracy, Task Adherence and Production Monitoring
To evaluate an agent you measure the outcome, the path it took (trajectory) and the cost separately, and report a success rate by running the same scenario N times with fake tools; LLM-as-judge comes only after code checks, with its reasoning read. We look at our own trajectory recording, deterministic scenarios, real results of Microsoft.Extensions.AI.Evaluation's agent evaluators with local models, and the path from OpenTelemetry spans to a regression set.
Testing Code That Uses IChatClient: Fake Clients, Deterministic Tests and Microsoft.Extensions.AI.Evaluation
You test the correctness of code that uses IChatClient with a fake client, without a model and in milliseconds; you measure the model's answer quality with Microsoft.Extensions.AI.Evaluation, in separate, cached evaluation runs. In real runs we see judge timeouts, reasoning hitting an 800-token cap, a cache returning stale answers, and a Relevance score that catches a prompt bug.