Evaluating Agents: Trajectory, Tool Call Accuracy, Task Adherence and Production Monitoring

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, Microsoft.Extensions.AI.Evaluation and .Quality 10.11.0 and OllamaSharp 5.5.2; the agent model is qwen3:4b on a local Ollama; qwen3:4b, gemma3:12b and gemma3:4b were tried as judges.

Guess First

You tell the Bookstore agent "Put two copies of Winter in Kyoto in my cart." Same model, same system prompt, same tools, same data. You run the agent three times, and all three times two books land in the cart. Did the three runs take the same path? The same number of steps, the same tokens, the same time?

No. The first run finished with search_books, check_stock, add_to_cart in 7 steps, 110 seconds and 5874 tokens. The second never looked at stock: search_books and add_to_cart in 5 steps, but 220 seconds and 3827 tokens. The third took 7 steps again, but 8384 tokens. The outcome is right in all three; path, cost and time differ. That is the first lesson of agent evaluation: looking at a single run and saying "passed" measures the outcome, not the agent. In the last post of the series we see how to measure agents with our own trajectory recording, deterministic fake tools, running N times, and the agent evaluators of Microsoft.Extensions.AI.Evaluation.

What to Measure: Outcome, Path and Cost

I split an agent's performance on a task into three questions:

Measuring only the outcome rewards an agent that reaches the right result by the wrong path: an answer that says "yes, it's in stock" without asking may be right today and wrong tomorrow. Measuring only the path punishes different but valid strategies: the second run that skipped check_stock still added the book to the cart correctly. That is why I write path checks as loose but critical rules such as "this tool must be called with these arguments" and "this tool must never be called".

Recording the Trajectory

To measure, you first have to record. As in post 2 of the series, FunctionInvokingChatClient drives the agent loop; to see every model call I put a DelegatingChatClient inside it:

using System.Diagnostics;
using System.Text.Json;
using Microsoft.Extensions.AI;

namespace Bookstore.Agent.Eval;

public sealed record Step(int Index, string Kind, string Name, string Detail, long StartMs, long DurationMs, long? InputTokens, long? OutputTokens);

// One run of the agent: every model call and every tool call, in order, with timings
public sealed class Trajectory
{
    private readonly Stopwatch _clock = Stopwatch.StartNew();
    public List<Step> Steps { get; } = [];

    public long Now => _clock.ElapsedMilliseconds;

    public void Add(string kind, string name, string detail, long startMs, long? inTokens = null, long? outTokens = null) =>
        Steps.Add(new Step(Steps.Count + 1, kind, name, detail, startMs, Now - startMs, inTokens, outTokens));

    public IEnumerable<Step> ToolCalls => Steps.Where(s => s.Kind == "tool");
}

public sealed class RecordingChatClient(IChatClient inner, Func<Trajectory> current) : DelegatingChatClient(inner)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        Trajectory trajectory = current();
        long start = trajectory.Now;
        ChatResponse response = await base.GetResponseAsync(messages, options, cancellationToken);
        string asked = string.Join(", ", response.Messages.SelectMany(m => m.Contents).OfType<FunctionCallContent>().Select(c => c.Name));
        trajectory.Add("model", "qwen3:4b", asked.Length == 0 ? "final answer" : $"asks {asked}", start,
            response.Usage?.InputTokenCount, response.Usage?.OutputTokenCount);
        return response;
    }
}

public static class Json
{
    public static string Of(object? value) => JsonSerializer.Serialize(value);
}

The order in the pipeline matters. In ChatClientBuilder the first one added ends up outermost; the RecordingChatClient added after .UseFunctionInvocation() sits inside the loop and runs once per model turn. In the reverse order I would see the whole loop as a single call. The tools record the tool side themselves.

The figure is drawn from the records of the first two runs in Guess First. Model call durations completely overshadow tool calls: the fake tools take under a millisecond, model turns between 20 and 77 seconds. The token count grows every turn, because each turn resends the whole history of the previous one; context growth from post 4, live.

Deterministic Scenarios with Fake Tools

If you test the agent against the real database and the real MCP (Model Context Protocol) server, you have two sources of uncertainty: the model and the world. Stock changes, orders ship, services slow down. To make measurement repeatable I freeze the world: fake tools that return the same data and record every call.

// Deterministic fakes: same data every run, every call is recorded
public sealed class FakeBookstore(Func<Trajectory> current)
{
    public IList<AITool> Tools =>
    [
        AIFunctionFactory.Create(SearchBooks, "search_books"),
        AIFunctionFactory.Create(CheckStock, "check_stock"),
        AIFunctionFactory.Create(GetOrderStatus, "get_order_status"),
        AIFunctionFactory.Create(AddToCart, "add_to_cart"),
        AIFunctionFactory.Create(CancelOrder, "cancel_order"),
    ];

    [Description("Adds copies of a book to the current customer's cart.")]
    private string AddToCart([Description("Book id from search_books")] int bookId, [Description("Number of copies")] int quantity) =>
        Record("add_to_cart", new { bookId, quantity }, () => $"Added {quantity} x book {bookId} to the cart.");

    private string Record(string name, object args, Func<string> body)
    {
        Trajectory trajectory = current();
        long start = trajectory.Now;
        string result = body();
        trajectory.Add("tool", name, Json.Of(args), start);
        return result;
    }
}

The remaining tools follow the same shape; the catalog and orders are the data we have used since the start of the series. The fake tools' names, descriptions and schemas must match those of the real MCP server; the model sees the definition, not the fake. You can turn the fingerprint gate from post 12 into a test here: if the fake's definition differs from the server's, the test goes red.

A scenario is a record that carries its expectations as data:

namespace Bookstore.Agent.Eval;

public sealed record ExpectedCall(string Tool, string ArgsJson);

public sealed record Scenario(
    string Name,
    string Prompt,
    ExpectedCall[] MustCall,        // in this order (other calls may sit in between)
    string[] MustNotCall,
    int MaxToolCalls,
    string AnswerMustContain);

public sealed record Verdict(bool Outcome, bool Path, bool Budget, string Why)
{
    public bool Passed => Outcome && Path && Budget;
}

public static class Checks
{
    public static Verdict Judge(Scenario s, Trajectory t, string answer)
    {
        List<Step> calls = [.. t.ToolCalls];
        int next = 0;
        foreach (Step call in calls)
        {
            if (next < s.MustCall.Length && call.Name == s.MustCall[next].Tool && call.Detail == s.MustCall[next].ArgsJson)
            {
                next++;
            }
        }

        bool outcome = answer.Contains(s.AnswerMustContain, StringComparison.OrdinalIgnoreCase);
        bool path = next == s.MustCall.Length && !calls.Any(c => s.MustNotCall.Contains(c.Name));
        bool budget = calls.Count <= s.MaxToolCalls;
        string why = !path
            ? $"missing {(next < s.MustCall.Length ? s.MustCall[next].Tool + s.MustCall[next].ArgsJson : "-")}, forbidden={calls.Any(c => s.MustNotCall.Contains(c.Name))}"
            : !outcome ? $"answer lacks '{s.AnswerMustContain}'" : !budget ? $"{calls.Count} tool calls" : "ok";
        return new Verdict(outcome, path, budget, why);
    }
}

The checks are deliberately simple and deterministic: the same trajectory always produces the same verdict. Asking an LLM (Large Language Model) "was this good" is the next section's topic; but first, check everything that does not need the model's opinion in code.

Running N Times: Success Rate

The main program runs the scenarios N times and writes each run's verdict, path and cost:

Scenario[] scenarios =
[
    new("order-status", "Where is my order 1043?",
        [new("get_order_status", """{"orderId":1043}""")], ["cancel_order", "add_to_cart"], 2, "TRK-88213"),
    new("stock-by-title", "Is 'Coffee and Compilers' in stock? How many copies?",
        [new("search_books", """{"query":"Coffee and Compilers"}"""), new("check_stock", """{"bookId":3}""")], ["add_to_cart"], 3, "4"),
    new("add-to-cart", "Put two copies of Winter in Kyoto in my cart.",
        [new("add_to_cart", """{"bookId":4,"quantity":2}""")], ["cancel_order"], 3, "Winter in Kyoto"),
    new("missing-book", "Add one copy of 'The Rust Orchard' to my cart.",
        [new("search_books", """{"query":"The Rust Orchard"}""")], ["add_to_cart", "cancel_order"], 3, "Rust Orchard"),
];

int runs = args.Length > 0 ? int.Parse(args[0]) : 3;
string judgeModel = args.Length > 1 ? args[1] : "qwen3:4b";
string[] only = args.Length > 2 ? args[2].Split(',') : [.. scenarios.Select(s => s.Name)];
Trajectory current = new();
FakeBookstore bookstore = new(() => current);

HttpClient http = new() { BaseAddress = new Uri("http://localhost:11434"), Timeout = TimeSpan.FromMinutes(10) };
IChatClient ollama = new OllamaApiClient(http, "qwen3:4b");
IChatClient agent = new ChatClientBuilder(ollama)
    .UseFunctionInvocation(configure: f => f.MaximumIterationsPerRequest = 6)
    .Use(inner => new RecordingChatClient(inner, () => current))
    .Build();

ChatMessage system = new(ChatRole.System,
    "You are the Bookstore assistant for customer 7. Use the tools for every fact about books, stock and orders. " +
    "Never cancel an order unless the user explicitly asks for it.");

List<object> log = [];
Dictionary<string, (List<ChatMessage> Messages, ChatResponse Response)> firstRuns = [];

foreach (Scenario scenario in scenarios.Where(s => only.Contains(s.Name)))
{
    int passed = 0;
    for (int run = 1; run <= runs; run++)
    {
        current = new Trajectory();
        List<ChatMessage> messages = [system, new(ChatRole.User, scenario.Prompt)];
        ChatResponse response = await agent.GetResponseAsync(messages, new ChatOptions { Tools = bookstore.Tools });
        Verdict verdict = Checks.Judge(scenario, current, response.Text);
        passed += verdict.Passed ? 1 : 0;
        firstRuns.TryAdd(scenario.Name, (messages, response));

        string path = string.Join(" > ", current.ToolCalls.Select(s => $"{s.Name}{s.Detail}"));
        long tokens = current.Steps.Sum(s => (s.InputTokens ?? 0) + (s.OutputTokens ?? 0));
        Console.WriteLine($"{scenario.Name} #{run}: {(verdict.Passed ? "PASS" : "FAIL")} ({verdict.Why}) " +
            $"steps={current.Steps.Count} tokens={tokens} time={current.Now / 1000.0:0.0}s path: {path}");
        log.Add(new { scenario = scenario.Name, run, verdict, answer = response.Text, steps = current.Steps });
    }

    Console.WriteLine($"{scenario.Name}: {passed}/{runs} passed");
}

File.WriteAllText($"trajectories-{judgeModel.Replace(':', '-')}.json", JsonSerializer.Serialize(log, new JsonSerializerOptions { WriteIndented = true }));

I deliberately did not set temperature or seed; I want to measure the behavior users will see, with the model's default sampling. The first run had the first three scenarios, three times each:

order-status #1: PASS (ok) steps=3 tokens=1558 time=169,1s path: get_order_status{"orderId":1043}
order-status #2: PASS (ok) steps=3 tokens=1682 time=112,5s path: get_order_status{"orderId":1043}
order-status #3: PASS (ok) steps=3 tokens=1552 time=156,4s path: get_order_status{"orderId":1043}
order-status: 3/3 passed
stock-by-title #1: PASS (ok) steps=5 tokens=3239 time=160,5s path: search_books{"query":"Coffee and Compilers"} > check_stock{"bookId":3}
stock-by-title #2: PASS (ok) steps=5 tokens=3097 time=236,0s path: search_books{"query":"Coffee and Compilers"} > check_stock{"bookId":3}
stock-by-title #3: PASS (ok) steps=5 tokens=4182 time=265,1s path: search_books{"query":"Coffee and Compilers"} > check_stock{"bookId":3}
stock-by-title: 3/3 passed
add-to-cart #1: PASS (ok) steps=7 tokens=5874 time=110,1s path: search_books{"query":"Winter in Kyoto"} > check_stock{"bookId":4} > add_to_cart{"bookId":4,"quantity":2}
add-to-cart #2: PASS (ok) steps=5 tokens=3827 time=219,8s path: search_books{"query":"Winter in Kyoto"} > add_to_cart{"bookId":4,"quantity":2}
add-to-cart #3: PASS (ok) steps=7 tokens=8384 time=236,1s path: search_books{"query":"Winter in Kyoto"} > check_stock{"bookId":4} > add_to_cart{"bookId":4,"quantity":2}
add-to-cart: 3/3 passed

9/9 passed; but that does not mean "the agent is reliable". Three runs are too few to estimate a rate: 3/3 also comes up with probability one in eight for an agent whose real success rate is 50 percent. More importantly, these three scenarios are easy; they don't push the agent's limits. The comma in the time column is a repeat of the culture lesson from post 9: my own console output was formatted with Turkish regional settings.

That is why I added a fourth scenario: adding a book that is not in the catalog to the cart. The right behavior is to search, find nothing, and not call add_to_cart. Three runs:

missing-book #1: PASS (ok) steps=3 tokens=1973 time=108,0s path: search_books{"query":"The Rust Orchard"}
missing-book #2: PASS (ok) steps=3 tokens=2390 time=91,5s path: search_books{"query":"The Rust Orchard"}
missing-book #3: PASS (ok) steps=3 tokens=2472 time=119,6s path: search_books{"query":"The Rust Orchard"}
missing-book: 3/3 passed

3/3 passed, and add_to_cart was never called. But the picture changes when you read the answers:

#1: I couldn't find "The Rust Orchard" in our catalog. Could you check the title spelling or try a different search term? For example, is it possible you meant "The Rusty Orchard" or another similar title?
#2: I couldn't find "The Rust Orchard" in our catalog. Could you check the title spelling or try a different book? For example, are you looking for "The Rust" by Michael C. Williams or another similar title?
#3: The book "The Rust Orchard" does not appear in our catalog. Please verify the title spelling or check if the author is specified (e.g., "The Rust Orchard by [Author]"). I cannot add books not present in the catalog.

The first two runs suggest books that are not in the catalog: "The Rusty Orchard" and "The Rust" by "Michael C. Williams". Both are made up; search_books came back empty and the model filled the gap itself. My checks did not see this, because they only measured that the forbidden tool was not called and that the title appeared in the answer; both were true. Code checks measure the rules you write; they don't catch the mistake you didn't write a rule for. There are two ways to catch it: a new rule ("every book named in the answer must appear in a search_books result") or letting a judge read the answer.

LLM-as-Judge: ToolCallAccuracy, TaskAdherence and IntentResolution

Code checks answer "what happened" well; they don't answer "did the answer meet the user's intent". The Microsoft.Extensions.AI.Evaluation.Quality package has three agent evaluators for this: ToolCallAccuracyEvaluator scores the relevance of tool calls to the conversation and the correctness of their parameters, TaskAdherenceEvaluator scores how well the response adheres to the assigned task and instructions, and IntentResolutionEvaluator scores whether the user's intent was understood and resolved. All three use an LLM as the judge and want the tool definitions through a context object:

// LLM-as-judge on the first run of each scenario; the judge model is a parameter
IChatClient judgeClient = new JudgeProbe(new OllamaApiClient(http, judgeModel));
ChatConfiguration judge = new(judgeClient);
IEvaluator[] evaluators = [new ToolCallAccuracyEvaluator(), new TaskAdherenceEvaluator(), new IntentResolutionEvaluator()];
EvaluationContext[] context =
[
    new ToolCallAccuracyEvaluatorContext(bookstore.Tools),
    new TaskAdherenceEvaluatorContext(bookstore.Tools),
    new IntentResolutionEvaluatorContext(bookstore.Tools),
];

foreach ((string name, (List<ChatMessage> messages, ChatResponse response)) in firstRuns)
{
    foreach (IEvaluator evaluator in evaluators)
    {
        EvaluationResult result = await evaluator.EvaluateAsync(messages, response, judge, context);
        foreach (EvaluationMetric metric in result.Metrics.Values)
        {
            string value = metric switch
            {
                NumericMetric n => $"{n.Value}",
                BooleanMetric b => $"{b.Value}",
                _ => "?",
            };
            string diagnostics = string.Join("; ", metric.Diagnostics?.Select(d => d.Message) ?? []);
            Console.WriteLine($"judge {name} | {metric.Name} = {value} | {metric.Interpretation?.Rating} | {diagnostics}");
        }
    }
}

The first build stopped with an error, not a warning:

error AIEVAL001: 'Microsoft.Extensions.AI.Evaluation.Quality.ToolCallAccuracyEvaluator' is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

In 10.11.0 the agent evaluators and their contexts are marked with this diagnostic; you have to state explicitly in the project file that you accept the API (Application Programming Interface) may change:

<PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <!-- Agent evaluators are marked "for evaluation purposes only" -->
    <NoWarn>$(NoWarn);AIEVAL001</NoWarn>
</PropertyGroup>

In the first attempt I used the agent itself, qwen3:4b, as the judge. The result:

judge order-status | Tool Call Accuracy =  | Inconclusive | Failed to parse score for 'Tool Call Accuracy' from the following evaluation response:
judge order-status | Task Adherence =  | Inconclusive | Failed to parse score for 'Task Adherence' from the following evaluation response:
judge order-status | Intent Resolution = 0 | Inconclusive | The model failed to produce a valid evaluation response.

Everything after "following evaluation response" is empty. To see what happened I wrapped the judge client in a small DelegatingChatClient that prints the response before it is parsed:

using Microsoft.Extensions.AI;

namespace Bookstore.Agent.Eval;

// Shows what the judge model actually returned before the evaluator parses it
public sealed class JudgeProbe(IChatClient inner) : DelegatingChatClient(inner)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        ChatResponse response = await base.GetResponseAsync(messages, options, cancellationToken);
        int reasoning = response.Messages.SelectMany(m => m.Contents).OfType<TextReasoningContent>().Sum(r => r.Text.Length);
        Console.WriteLine($"  judge call: maxOutputTokens={options?.MaxOutputTokens} finish={response.FinishReason} " +
            $"text={response.Text.Length} chars reasoning={reasoning} chars outTokens={response.Usage?.OutputTokenCount}");
        return response;
    }
}

  judge call: maxOutputTokens=800 finish=length text=0 chars reasoning=3075 chars outTokens=800
judge order-status | Tool Call Accuracy =  | Inconclusive | Failed to parse score for 'Tool Call Accuracy' from the following evaluation response:
  judge call: maxOutputTokens=800 finish=length text=0 chars reasoning=3593 chars outTokens=800
judge order-status | Task Adherence =  | Inconclusive | Failed to parse score for 'Task Adherence' from the following evaluation response:
  judge call: maxOutputTokens=800 finish=length text=0 chars reasoning=3373 chars outTokens=800
judge order-status | Intent Resolution = 0 | Inconclusive | The model failed to produce a valid evaluation response.

The same picture in all three calls: the evaluator limits the output budget to 800 tokens, qwen3:4b spends the whole budget on thinking (over 3,000 characters of reasoning), and the response is cut off with finish=length before the answer text even begins. The judge doesn't give a bad score; it can't give a score at all. In post 9 of the LLM Fundamentals series we saw how reasoning models' thinking tokens count against budgets and the bill; here the same thing silently makes an evaluation tool useless. The fix is to separate the judge from the agent: a model without a thinking mode and ideally a larger one. First I tried making gemma3:12b on Ollama the judge:

Unhandled exception. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 600 seconds elapsing.

The first judge call with gemma3:12b exceeded the ten-minute HttpClient.Timeout: with other work sharing the same Ollama, the 12-billion-parameter model either could not be loaded or waited in the queue. I reran two scenarios once each with the smaller gemma3:4b, which has no thinking mode. The six metrics the judge produced (only the start of each explanation is shown):

judge order-status | Tool Call Accuracy =  | Inconclusive | Model's evaluation chain of thought: Let's think step by ste ...
judge order-status | Task Adherence =  | Inconclusive | Failed to parse score for 'Task Adherence' from the followin ...
judge order-status | Intent Resolution = 5 | Exceptional |
judge missing-book | Tool Call Accuracy =  | Inconclusive | Failed to parse score for 'Tool Call Accuracy' from the foll ...
judge missing-book | Task Adherence =  | Inconclusive | Failed to parse score for 'Task Adherence' from the followin ...
judge missing-book | Intent Resolution = 4 | Good |

Only two of the six metrics could be parsed: Intent Resolution in both scenarios, 5 and 4. In the other four the judge did write a response, but not in the format the evaluator expects. The end of the Tool Call Accuracy response for order-status:

... </S1>The tool call is directly relevant to the user's query and utilizes a tool defined in the TOOL DEFINITION. </S2>1</S0>

The score is there, but the tags are broken: opening tags are missing and the closing ones are jumbled. The real problem, though, is in the content of the unparsed texts. For missing-book, the judge's reasoning says searching for a book that does not exist is "not relevant to the user's need to add a book to the cart" and writes 0 for Tool Call Accuracy; in Task Adherence it finds the answer "barely adherent" because the book was not added, and gives 2. Yet not adding a book that does not exist is the correct behavior. On top of that, the answer in this run also suggested a "The Rust" book that is not in the catalog, and Intent Resolution called it 4, "Good"; the judge did not see the made-up suggestion either.

These results don't show that the evaluators are bad; the package's own documentation says these evaluators' prompts were tested against and tuned for GPT-4o from OpenAI's GPT (Generative Pre-trained Transformer) family, and that results can be especially poor with smaller, local models. I did not run a cloud model for this post; what I saw with local models is this: a judge's score is not a number to act on without reading its reasoning. Code checks first, then the judge, and always store the judge's score together with its raw response.

Production: From Traces to Trajectories

In the test environment we kept the trajectory with our own recording. In production the same information may already be in OpenTelemetry traces. UseOpenTelemetry() produces spans following the GenAI semantic conventions; FunctionInvokingChatClient also publishes tool executions as spans from the same source. An ActivityListener is enough to see them without setting up an exporter:

using System.Diagnostics;
using Bookstore.Agent.Otel;
using Microsoft.Extensions.AI;
using OllamaSharp;

// Print every finished span: in production an OTLP exporter sends these to your tracing backend
using ActivityListener listener = new()
{
    ShouldListenTo = source => source.Name == "Bookstore.Agent",
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllDataAndRecorded,
    ActivityStopped = a => Console.WriteLine(
        $"{a.DisplayName,-28} {a.Duration.TotalSeconds,6:0.0}s parent={a.Parent?.DisplayName ?? "-"} " +
        string.Join(" ", a.Tags.Where(t => t.Key is "gen_ai.tool.name" or "gen_ai.usage.input_tokens" or "gen_ai.usage.output_tokens")
            .Select(t => $"{t.Key}={t.Value}"))),
};
ActivitySource.AddActivityListener(listener);

Trajectory current = new();
FakeBookstore bookstore = new(() => current);
HttpClient http = new() { BaseAddress = new Uri("http://localhost:11434"), Timeout = TimeSpan.FromMinutes(10) };
IChatClient agent = new ChatClientBuilder(new OllamaApiClient(http, "qwen3:4b"))
    .UseFunctionInvocation()
    .UseOpenTelemetry(sourceName: "Bookstore.Agent")
    .Build();

ChatResponse response = await agent.GetResponseAsync(
    [new(ChatRole.System, "You are the Bookstore assistant for customer 7. Use the tools for every fact."),
     new(ChatRole.User, "Is 'Coffee and Compilers' in stock? How many copies?")],
    new ChatOptions { Tools = bookstore.Tools });
Console.WriteLine(response.Text);

chat qwen3:4b                  84,5s parent=orchestrate_tools
execute_tool search_books       0,0s parent=orchestrate_tools gen_ai.tool.name=search_books
chat qwen3:4b                  60,7s parent=orchestrate_tools
execute_tool check_stock        0,0s parent=orchestrate_tools gen_ai.tool.name=check_stock
chat qwen3:4b                  29,6s parent=orchestrate_tools
orchestrate_tools             174,8s parent=-
Yes, 'Coffee and Compilers' is in stock with 4 copies available.

The trajectory is here too: an orchestrate_tools root span with model and tool spans in order beneath it. I tried to print the token tags as well, but in this setup they did not appear on the chat spans; I take tokens from ChatResponse.Usage in RecordingChatClient. The details belong to the observability post of the .NET and AI series; what matters here is how a production trace connects to evaluation. My loop looks like this:

• Store traces and the audit log, as in post 13, with the user and conversation id.
• Flag failed conversations: tool spans ending in errors, step counts over budget, negative user feedback, the same tool called again with the same arguments.
• Turn a flagged conversation into a scenario: the user message, expected and forbidden tool calls, the information the answer must contain. Scrub personal data.
• Add the scenario to the regression set and run it N times on every model, prompt or tool change.

In Production

First, cost. The 16 agent runs I made for this post took 41 minutes in total on the local model; judge calls and one call that timed out after ten minutes came on top of that. A cloud model shortens the time, but the same number of calls means a bill. The Reporting package of Microsoft.Extensions.AI.Evaluation can cache model responses and reuse them for unchanged inputs, and the dotnet aieval tool in the Console package generates reports from stored results. Split the scenario set in two: a small, fast set that runs on every commit, and a broad set that runs nightly.

Second, validate the judge itself. A judge is a model too; it can score wrongly and give different scores to the same input. Score a few trajectories by hand and compare them with the judge's scores; if they don't agree, don't make decisions based on the judge's scores. The package documentation also says plainly that these evaluators can perform especially poorly with smaller, local models.

Third, a dose of reality. Fake tools make measurement repeatable, but they don't show the real server's latency, errors and paginated responses. Alongside the evaluation with fakes, run a small number of end-to-end scenarios against the real MCP servers.

Common Pitfalls

First, calling it "passed" after a single run. The three runs in Guess First reached the same outcome by different paths; when the model or the prompt changes, this distribution changes too. Report a scenario's result as a rate, "3/3", "4/5", and run it often enough to notice the rate dropping.

Second, measuring only the final answer. "4 copies" in the answer does not mean the model called check_stock; it may have guessed. Without a trajectory check you cannot see the difference.

Third, trusting the judge blindly. In this post the judge's first result was "Inconclusive", and the reason was not model quality but thinking tokens consuming the output budget. Before using a judge result as a number, look at its diagnostics and the raw response.

Fourth, writing expectations too strictly. In the missing-book scenario I expected the search argument verbatim; the model searching for "Rust Orchard" is also reasonable behavior. For every scenario, ask yourself whether the rules measure "correct behavior" or "the one path I had in mind".

A Small Experiment

★☆☆ Lower MaxToolCalls by one for every scenario and run the tests again. Which runs hit the budget? Was this budget measuring the agent's behavior, or your guess?

★★☆ Make the argument comparison in Checks.Judge semantic instead of verbatim string matching: ignore case and a leading "The" for query, compare numeric arguments as numbers. How did the missing-book results change?

★★★ Connect the agent to the real MCP server as in post 12 and run the same scenarios. Is there a difference between the success rate with fakes and with the real server? Find the source of the difference in the trajectories.

With that, we have reached the end of the AI (Artificial Intelligence) Agents and MCP series. We started by writing an agent loop by hand, then moved through tool design, context engineering, workflow and multi-agent patterns, and human approval. Then we moved tools out of the application with the Model Context Protocol: C# and TypeScript servers, a remote server protected with OAuth, our own client, security controls and now measurement. The question to ask before putting an agent into production is no longer "can the model do this" but "when it gets it wrong, how will I see it and how much damage can it do". From here the path leads on to the hosting and observability posts of the .NET and AI series and to the upcoming LLM Application Security series.