Testing Code That Uses IChatClient: Fake Clients, Deterministic Tests and Microsoft.Extensions.AI.Evaluation

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, Microsoft.Extensions.AI.Evaluation.Quality and .Reporting 10.11.0, Microsoft.Extensions.AI.Evaluation.NLP 10.11.0-preview.1.26513.7, xUnit.net v3 4.0.2, OllamaSharp 5.5.2 and Ollama 0.34.4 (qwen3:4b).

Guess First

You want to write a test for the Bookstore assistant: for "I want to change old, untested code without breaking it, which book should I read?" it should recommend Working Effectively with Legacy Code. You wired the test to the real model and wrote Assert.Contains("Legacy Code", answer). Does this test give the same result on every CI (Continuous Integration) run? How many seconds does it take? What happens if the model answers "Feathers' book"?

And think about this: when this test turns red, is the problem in your code, in the model, or in the prompt?

In this post we answer two separate questions in two separate places. "Is my code correct?" is the job of unit tests that run with a fake IChatClient, in milliseconds, on every build. "Does the model answer well?" is the job of evaluations with Microsoft.Extensions.AI.Evaluation, which are slow and costly and run separately. Mixing the two, as we'll see at the end of this post, means a test suite that is both slow and fragile.

The Code Under Test

We have two classes. BookRecommender gets a structured recommendation from the model and checks it against the catalog:

using Microsoft.Extensions.AI;

namespace Bookstore.Assistant;

public sealed record BookRecommendation(string Title, string Reason);

public sealed class BookRecommender(IChatClient chat)
{
    public async Task<BookRecommendation?> RecommendAsync(string question, CancellationToken ct = default)
    {
        List<ChatMessage> messages =
        [
            new(ChatRole.System,
                $"Recommend exactly one book from this catalog: {string.Join("; ", Catalog.Titles)}."),
            new(ChatRole.User, question),
        ];

        ChatResponse<BookRecommendation> response = await chat.GetResponseAsync<BookRecommendation>(
            messages, new ChatOptions { Temperature = 0, Seed = 42 }, cancellationToken: ct);

        // The model may invent a book outside the catalog or return broken JSON
        if (!response.TryGetResult(out BookRecommendation? recommendation))
        {
            return null;
        }

        return Catalog.Titles.Contains(recommendation.Title, StringComparer.OrdinalIgnoreCase)
            ? recommendation
            : null;
    }
}

What's worth testing here isn't the model's answer but our code's decision logic: return the recommendation if the model picks a book from the catalog, return null if it invents one or returns broken JSON (JavaScript Object Notation), and send the request with deterministic settings. The idea behind structured output was covered in the LLM (Large Language Model) Fundamentals series post "Structured Output with LLMs".

BookstoreAssistant is a plain version of this series' order assistant: a system prompt, a tool and a model id that can be chosen per request.

using Microsoft.Extensions.AI;

namespace Bookstore.Assistant;

public sealed class BookstoreAssistant(IChatClient chat, OrderTools tools, string? modelId = null)
{
    public const string SystemPrompt =
        "You are the Bookstore assistant. Use tools for order questions. Answer in at most three sentences.";

    public async Task<(List<ChatMessage> Messages, ChatResponse Response)> AskAsync(
        string question, CancellationToken ct = default)
    {
        List<ChatMessage> messages = [new(ChatRole.System, SystemPrompt), new(ChatRole.User, question)];

        ChatOptions options = new()
        {
            ModelId = modelId,
            Temperature = 0,
            Seed = 42,
            Tools = [AIFunctionFactory.Create(tools.GetOrderStatus)],
        };

        ChatResponse response = await chat.GetResponseAsync(messages, options, ct);
        return (messages, response);
    }
}

A Fake IChatClient: A Model That Returns the Next Answer

IChatClient is a small interface; writing your own implementation for tests takes a few lines:

using Microsoft.Extensions.AI;

namespace Bookstore.Assistant.Tests;

// Returns the next canned response and records every incoming request
public sealed class ScriptedChatClient(params ChatResponse[] responses) : IChatClient
{
    private readonly Queue<ChatResponse> _responses = new(responses);

    public List<(List<ChatMessage> Messages, ChatOptions? Options)> Calls { get; } = [];

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        Calls.Add(([.. messages], options));
        return Task.FromResult(_responses.Dequeue());
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        ChatResponse response = await GetResponseAsync(messages, options, cancellationToken);
        foreach (ChatResponseUpdate update in response.ToChatResponseUpdates())
        {
            yield return update;
        }
    }

    public object? GetService(Type serviceType, object? serviceKey = null) =>
        serviceType.IsInstanceOfType(this) ? this : null;

    public void Dispose() { }

    public static ChatResponse Text(string text) => new(new ChatMessage(ChatRole.Assistant, text));
}

ScriptedChatClient does two things: it returns the next canned response and records every incoming request, with its messages and options, in the Calls list. The streaming method splits the same canned response into chunks with ToChatResponseUpdates(), so code that uses streaming can be tested with the same fake. You could do the same with a mocking library like NSubstitute. But GetResponseAsync<T> is an extension method and can't be mocked; what actually gets called is GetResponseAsync. That's why a small hand-written fake reads better than a mock here. The difference between mocks and fakes is covered in detail in the Unit Testing series.

Testing That the Code Is Correct

A test project with xUnit.net v3, running on Microsoft Testing Platform as in the Unit Testing series:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <OutputType>Exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <Using Include="Xunit" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="xunit.v3.mtp-v2" Version="4.0.2" />
    <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Quality" Version="10.11.0" />
    <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Reporting" Version="10.11.0" />
    <PackageReference Include="Microsoft.Extensions.AI.Evaluation.NLP" Version="10.11.0-preview.1.26513.7" />
    <PackageReference Include="OllamaSharp" Version="5.5.2" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\Bookstore.Assistant\Bookstore.Assistant.csproj" />
  </ItemGroup>

</Project>

Four behaviors of BookRecommender:

using Microsoft.Extensions.AI;

namespace Bookstore.Assistant.Tests;

public sealed class BookRecommenderTests
{
    [Fact]
    public async Task Returns_recommendation_when_model_picks_a_catalog_book()
    {
        ScriptedChatClient chat = new(ScriptedChatClient.Text(
            """{"title":"Working Effectively with Legacy Code","reason":"It shows how to get old code under test."}"""));

        BookRecommendation? result = await new BookRecommender(chat)
            .RecommendAsync("How do I change old code safely?", TestContext.Current.CancellationToken);

        Assert.Equal("Working Effectively with Legacy Code", result?.Title);
    }

    [Fact]
    public async Task Returns_null_when_model_invents_a_book()
    {
        ScriptedChatClient chat = new(ScriptedChatClient.Text(
            """{"title":"Legacy Code Mastery","reason":"A great book."}"""));

        BookRecommendation? result = await new BookRecommender(chat)
            .RecommendAsync("How do I change old code safely?", TestContext.Current.CancellationToken);

        Assert.Null(result);
    }

    [Fact]
    public async Task Returns_null_when_model_returns_broken_json()
    {
        ScriptedChatClient chat = new(ScriptedChatClient.Text("Sure! I recommend Refactoring."));

        BookRecommendation? result = await new BookRecommender(chat)
            .RecommendAsync("Any book on refactoring?", TestContext.Current.CancellationToken);

        Assert.Null(result);
    }

    [Fact]
    public async Task Sends_catalog_and_deterministic_options()
    {
        ScriptedChatClient chat = new(ScriptedChatClient.Text("""{"title":"Refactoring","reason":"r"}"""));

        await new BookRecommender(chat).RecommendAsync("Any book on refactoring?", TestContext.Current.CancellationToken);

        (List<ChatMessage> messages, ChatOptions? options) = Assert.Single(chat.Calls);
        Assert.Equal(ChatRole.System, messages[0].Role);
        Assert.Contains("Designing Data-Intensive Applications", messages[0].Text);
        Assert.Equal(0f, options?.Temperature);
        Assert.IsType<ChatResponseFormatJson>(options?.ResponseFormat);
    }
}

The second and third tests cover situations that are nearly impossible to produce with a real model: the model inventing a book that doesn't exist, and returning plain text instead of JSON. The real model won't tell you when it will do that; with a fake you make it happen whenever you want. The fourth test looks at the request itself: does the system message contain the catalog, is Temperature 0, did GetResponseAsync<T> add a JSON response format to the request?

Tools have two levels. The tool function itself is a plain C# method; it has nothing to do with the model and is tested directly. That the tool is wired into the pipeline correctly, meaning the call the model asks for is really made and the result goes back to the model, we test with a fake model turn:

using Microsoft.Extensions.AI;

namespace Bookstore.Assistant.Tests;

public sealed class OrderAssistantTests
{
    [Theory]
    [InlineData(1042, "Shipped on 2026-11-06, 2 items, tracking BKS-1042.")]
    [InlineData(7, "Order not found.")]
    public void GetOrderStatus_returns_status_without_any_model(int orderId, string expected) =>
        Assert.Equal(expected, new OrderTools().GetOrderStatus(orderId));

    [Fact]
    public async Task Tool_call_from_model_runs_the_tool_and_sends_the_result_back()
    {
        // Round 1: the model asks for a tool; round 2: the model writes the answer
        ScriptedChatClient model = new(
            new ChatResponse(new ChatMessage(ChatRole.Assistant,
                [new FunctionCallContent("call-1", "GetOrderStatus", new Dictionary<string, object?> { ["orderId"] = 1042 })])),
            ScriptedChatClient.Text("Your order 1042 has shipped."));

        IChatClient pipeline = model.AsBuilder().UseFunctionInvocation().Build();
        BookstoreAssistant assistant = new(pipeline, new OrderTools());

        (_, ChatResponse response) = await assistant.AskAsync("Where is order 1042?", TestContext.Current.CancellationToken);

        Assert.Equal("Your order 1042 has shipped.", response.Text);
        Assert.Equal(2, model.Calls.Count);
        FunctionResultContent result = model.Calls[1].Messages
            .SelectMany(m => m.Contents).OfType<FunctionResultContent>().Single();
        Assert.Equal("call-1", result.CallId);
        Assert.Contains("BKS-1042", result.Result?.ToString());
    }
}

In the second test the fake model returns a FunctionCallContent in the first round, just like a real tool request from the LLM Fundamentals series' tool calling post. UseFunctionInvocation() runs the tool, and the messages sent to the model in the second round must contain a FunctionResultContent with the same CallId. This test verifies the tool name, the parameter name (orderId) and the result's way back to the model, without a model. If someone renames the method or its parameter, the test breaks.

We separate the evaluation tests with a trait and run only the unit tests:

dotnet test --project tests/Bookstore.Assistant.Tests --filter-not-trait "Category=Evaluation"

Test run summary: Passed!
  total: 7
  failed: 0
  succeeded: 7
  skipped: 0
  duration: 935ms

Seven tests in under a second. They give the same result with Ollama down and without a network. This is the part that should run on every commit in CI.

Measuring Answer Quality: Evaluation

Unit tests answer "if the model returns this, does my code behave correctly?". They don't tell you whether the model really answers well. For that we use the Microsoft.Extensions.AI.Evaluation libraries. The evaluators in Microsoft.Extensions.AI.Evaluation.Quality (RelevanceEvaluator, CoherenceEvaluator and others) have an LLM, a "judge" model, score the answer. F1Evaluator from Microsoft.Extensions.AI.Evaluation.NLP uses no LLM: it looks at the words the answer shares with a reference answer. The NLP (Natural Language Processing) package is still preview. Microsoft.Extensions.AI.Evaluation.Reporting writes the results to disk and caches LLM responses.

using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.NLP;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Microsoft.Extensions.AI.Evaluation.Reporting;
using Microsoft.Extensions.AI.Evaluation.Reporting.Storage;
using OllamaSharp;

namespace Bookstore.Assistant.Tests;

[Trait("Category", "Evaluation")]
public sealed class AssistantEvaluationTests
{
    // Both the judge and the app use qwen3:4b: avoids waiting for a model swap on a shared server.
    // The evaluators' 800-token response cap is not enough for a thinking judge
    private static readonly IChatClient Judge = new ChatClientBuilder(new OllamaApiClient(
            new HttpClient { BaseAddress = new Uri("http://localhost:11434"), Timeout = TimeSpan.FromMinutes(10) },
            "qwen3:4b"))
        .ConfigureOptions(o => o.MaxOutputTokens = 4096)
        .Build();

    private static readonly ReportingConfiguration Reporting = DiskBasedReportingConfiguration.Create(
        storageRootPath: Environment.GetEnvironmentVariable("EVAL_STORAGE_PATH") ?? "eval-results",
        evaluators: [new RelevanceEvaluator(), new CoherenceEvaluator(), new F1Evaluator()],
        chatConfiguration: new ChatConfiguration(Judge),
        enableResponseCaching: true,
        executionName: Environment.GetEnvironmentVariable("EVAL_EXECUTION") ?? $"{DateTime.UtcNow:yyyyMMddTHHmmss}");

    [Theory]
    [InlineData("legacy-code",
        "I keep breaking old untested code when I change it. Which book should I read?",
        "Read Working Effectively with Legacy Code; it shows how to get untested code under test so you can change it safely.")]
    [InlineData("order-status",
        "Where is my order 1042?",
        "Order 1042 shipped on 2026-11-06 with 2 items; the tracking number is BKS-1042.")]
    public async Task Assistant_answer_is_relevant_and_coherent(string scenario, string question, string reference)
    {
        await using ScenarioRun run = await Reporting.CreateScenarioRunAsync(
            $"Bookstore.Assistant.{scenario}", cancellationToken: TestContext.Current.CancellationToken);

        // The app call through the cached client is cached too
        IChatClient app = run.ChatConfiguration!.ChatClient.AsBuilder().UseFunctionInvocation().Build();
        BookstoreAssistant assistant = new(app, new OrderTools(), modelId: "qwen3:4b");

        (List<ChatMessage> messages, ChatResponse response) =
            await assistant.AskAsync(question, TestContext.Current.CancellationToken);

        EvaluationResult result = await run.EvaluateAsync(messages, response,
            additionalContext: [new F1EvaluatorContext(reference)],
            cancellationToken: TestContext.Current.CancellationToken);

        NumericMetric relevance = result.Get<NumericMetric>(RelevanceEvaluator.RelevanceMetricName);
        NumericMetric coherence = result.Get<NumericMetric>(CoherenceEvaluator.CoherenceMetricName);
        NumericMetric f1 = result.Get<NumericMetric>(F1Evaluator.F1MetricName);

        TestContext.Current.TestOutputHelper?.WriteLine(
            $"{scenario}: relevance={relevance.Value} ({relevance.Interpretation?.Rating}), " +
            $"coherence={coherence.Value} ({coherence.Interpretation?.Rating}), f1={f1.Value:F2}");
        TestContext.Current.TestOutputHelper?.WriteLine($"answer: {response.Text}");

        Assert.False(relevance.Interpretation?.Failed ?? true, relevance.Reason);
        Assert.False(coherence.Interpretation?.Failed ?? true, coherence.Reason);
    }
}

A few design decisions. The test class is marked with [Trait("Category", "Evaluation")] so it runs separately from the unit tests. DiskBasedReportingConfiguration.Create brings together the evaluators, the judge's ChatConfiguration, the folder for results and enableResponseCaching: true. The execution name comes from an environment variable; as the Microsoft Learn tutorial suggests, in CI you can make it the build number. We call the assistant through run.ChatConfiguration.ChatClient: this client is cached, so the application's own model calls go into the cache too. The application model is chosen per request through ModelId.

Choosing the Judge: The Story of Four Attempts

The judge setup in the code wasn't like this on the first try. This section walks through the problems we hit in real runs, in order, because each of them is the kind you'll meet in production too.

On the first attempt we picked the bigger model, gemma3:12b, as the judge. A judge stronger than the model it evaluates seems sensible. On the shared local Ollama, though, the first call hit HttpClient's default 100-second timeout. We raised the timeout to 10 minutes; this time Relevance and Coherence came back with the diagnostic "The request was canceled due to the configured HttpClient.Timeout of 600 seconds elapsing". Making gemma3:4b the judge didn't change the result. The reason showed in the /api/ps output: only qwen3:4b was loaded on the server at the time, and gemma3's turn never came while other jobs' qwen3 requests kept arriving. When the application and the judge use different models, on a shared server the judge may have to wait for a model swap.

On the third attempt we made the judge the same model as the application: qwen3:4b. This time the evaluators complained about an empty response: "Failed to parse score for 'Relevance' from the following evaluation response". The raw response in the cache showed the cause: the evaluators cap the judge's response at 800 tokens. All 800 of qwen3:4b's tokens went to reasoning, finishReason was length, and the text part was empty. We tried turning thinking off (ReasoningEffort.None through ChatOptions.Reasoning). The model then wrote its thoughts straight into the text and was again cut off at 800 tokens.

On the fourth attempt we added ConfigureOptions(o => o.MaxOutputTokens = 4096) to the judge client. We ran the test and it finished in 1.7 seconds, but the results hadn't changed. The cache was returning the old broken responses. ConfigureOptions sits inside our client, while Reporting's cache layer sits outside it. The cache key is computed before our options change. We ran dotnet aieval clean-cache, but that command deletes only expired entries ("Deleting expired cache entries..."). Once we deleted the cache folder by hand, the real results came in:

dotnet test --project tests/Bookstore.Assistant.Tests --filter-trait "Category=Evaluation" --output detailed

failed ...Assistant_answer_is_relevant_and_coherent(scenario: "legacy-code", ...) (3m 37s 671ms)
  The response addresses the query but omits specific book recommendations, which are key details for the user.
  Standard output
    legacy-code: relevance=3 (Average), coherence= (Inconclusive), f1=0,09
    answer: I don't have access to book recommendations or programming resources through my current tools. My capabilities are limited to checking order statuses using the provided function. Please consult a programming book or resource for code stability best practices.
failed ...Assistant_answer_is_relevant_and_coherent(scenario: "order-status", ...) (5m 28s 450ms)
  Assert.False() Failure
  Expected: False
  Actual:   True
  Standard output
    order-status: relevance= (Inconclusive), coherence=4 (Good), f1=0,60
    answer: Your order #1042 was shipped on November 6, 2026, with 2 items. The tracking number is BKS-1042. You can track its delivery using this number.

Test run summary: Failed!
  total: 2
  failed: 2
  succeeded: 0

Total time: 9 minutes 7 seconds. The first result is exactly what you'd want from an evaluation: asked for a book recommendation, the assistant said "that's not my job". It got Relevance 3 (Average), and the reasoning is right: the answer relates to the question but doesn't provide what was actually asked, a book recommendation. F1 (F1 score: the harmonic mean of precision and recall over shared words) is 0.09, almost no words in common with the reference answer. The cause isn't the model but our system prompt: the sentence "Use tools for order questions" convinced the small model that it should only handle order questions. Our unit tests could never have caught this; the code works correctly, the product behavior is wrong.

The second result shows how unreliable the judge itself is. The answer is good (Coherence 4, F1 0.60), but Relevance is "Inconclusive" again: qwen3:4b didn't write the score in the format the evaluator expects. The test is red not because the answer is bad but because the judge made a formatting mistake. On top of that, in this setup the judge scores answers it produced itself; a model favoring its own output is a known bias. We accepted it here because of local constraints; in production the judge should be a different and stronger model than the application model.

Cache, Report and CI Cost

We ran the same test again without changing anything:

EVAL_EXECUTION=run8 dotnet test --project tests/Bookstore.Assistant.Tests --filter-trait "Category=Evaluation"
  total: 2
  failed: 2
  succeeded: 0
  duration: 767ms

767 milliseconds instead of 9 minutes. As long as prompts, options and model stay the same, every model call comes from the cache on disk. According to the Microsoft Learn tutorial, cache entries live for 14 days by default. If you keep this folder between CI builds, only scenarios whose prompts changed go to the model. For the report we install the Microsoft.Extensions.AI.Evaluation.Console tool not globally but through the project's tool manifest:

dotnet new tool-manifest
dotnet tool install Microsoft.Extensions.AI.Evaluation.Console --version 10.11.0
DOTNET_AIEVAL_TELEMETRY_OPTOUT=1 dotnet aieval report --path eval-results --output report.html -n 2
info: aieval[0] Storage root path: eval-results
info: aieval[0] Execution: run8 Scenario: Bookstore.Assistant.legacy-code Iteration: 1
info: aieval[0] Execution: run8 Scenario: Bookstore.Assistant.order-status Iteration: 1
info: aieval[0] Execution: run7 Scenario: Bookstore.Assistant.legacy-code Iteration: 1
info: aieval[0] Execution: run7 Scenario: Bookstore.Assistant.order-status Iteration: 1
info: aieval[0] Report: report.html [html]

The report is a single HTML (HyperText Markup Language) file (918 KB in this run). Scenarios are grouped into a tree by the dots in the scenario name; each metric's card shows the score, the interpretation and the judge's reasoning. DOTNET_AIEVAL_TELEMETRY_OPTOUT: the tool collects anonymous usage data by default, and this variable turns it off.

Thresholds and Flaky Evaluations

In these tests we asserted on Interpretation.Failed for each metric. In the library's default interpretation, on the 1–5 scale, 4 and above are "Good" or "Exceptional" and anything below fails. The Microsoft Learn tutorial also warns against turning individual evaluation results into build-breaking asserts: models and scores change over time; rather than failing on a single test, it's healthier to track trends in the reports and fail a build only when there's a significant drop across many scenarios.

Our run shows why in concrete terms. Of the two red tests, one is a real product bug and the other a judge formatting error. A practical setup: unit tests run on every commit and break the build. Evaluations run in a separate job, nightly or when a prompt changes, and their results go to the report. An "Inconclusive" result is treated as something to retry, not as a failure. And you set the threshold from the average of several runs, not as a one-off number.

In Production

• The judge should be separate from the model it evaluates, and preferably stronger. If the same model scores its own answer, bias creeps in; a small model may also not follow the score format.
• Plan the judge's quota and latency separately from the application's. On a shared model server judge requests compete with application traffic.
• Treat the cache folder like a build artifact: keep it, but clear it deliberately when the judge setup, the evaluator version or the prompt template changes.
• Evaluation data (questions, reference answers) is as valuable as code. Build it from real user questions with personal data removed, and version it.
• Evaluators specific to RAG (Retrieval-Augmented Generation) and agents (GroundednessEvaluator, RetrievalEvaluator, ToolCallAccuracyEvaluator, TaskAdherenceEvaluator) will be used in the related series.

Common Pitfalls

The most common pitfall is putting tests that hit a real model into the unit test suite. The two evaluation tests in this post took 9 minutes and wouldn't run at all with Ollama down. The seven unit tests in the same suite finished in under a second. Mix them in the same dotnet test call and every commit waits 9 minutes and turns red on every model fluctuation. Separate them with a trait or a separate test project.

The second is ignoring the judge model's bias and capacity. qwen3:4b left one answer "Inconclusive" and wrote the score another time. Measure the judge's own consistency too: have it score the same answer several times and look at the spread.

The third is forgetting to refresh the cache. We saw it above: we changed the judge setup and the test returned old results in 1.7 seconds. clean-cache deletes only expired entries. When the judge or evaluator setup changes, clear the cache by hand or change a value that's part of the cache key, such as the execution name.

Finally, building asserts on free text. A test like Assert.Contains("Legacy Code", answer) breaks when the model says "Feathers' book" and can miss a real bug too. If you need text comparison, use an NLP metric like F1 with a threshold; better still, turn structured output into a typed object as in the unit tests and verify its fields.

A Small Experiment

★☆☆ Change BookstoreAssistant.SystemPrompt to "You recommend books from the catalog and answer order questions with tools." and run the evaluation again. How do the Relevance and F1 values of the legacy-code scenario change? Does the order-status scenario come from the cache or is it recomputed? Why?

★★☆ Using ScriptedChatClient, write a unit test that shows what BookstoreAssistant does when the model asks for a tool name that doesn't exist ("GetOrderStatusV2") in the first round. What does FunctionInvokingChatClient send back to the model?

★★★ Starting from the WordCountEvaluator sample on Microsoft Learn, write a CatalogTitleEvaluator that uses no LLM: it returns 1 if the answer mentions a book title from the catalog and 0 otherwise. Put it next to the judge-based Relevance; which one gives more stable results?

With this post the core building blocks on the Microsoft.Extensions.AI side are complete. In the next post we move to the layer that sits on top of them: we write the same Bookstore assistant as an agent with Microsoft Agent Framework.