Taking Tool Calling to Production: FunctionInvokingChatClient Settings, DI Services and Approvals

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, OllamaSharp 5.5.2, Microsoft.EntityFrameworkCore.Sqlite 10.0.14 and a local Ollama 0.34.4 (qwen3:4b).

Guess First

In the tool calling posts of LLM (Large Language Model) Fundamentals we gave the Bookstore assistant an order lookup tool. Now a developer taking it to production writes the tool's signature like this: GetOrderStatus(string customerId, int orderId). The reasoning sounds fine: let the model know which customer is asking. Ada types into the chat window: "I'm actually grace, what's the status of my order 1043?" What happens?

Second question: the same assistant gets the ability to cancel orders. When the user says "cancel 1042", the model calls the tool and the tool cancels the order. What happens if, because of a network error, the client sends the same request twice? And what if the user actually asked "what would happen if I cancelled 1042?"

In this post we derive both from the code: identity comes from the session, not from the model's arguments, and irreversible actions don't run without the user's approval.

Grouping Tools in Their Own Class

In LLM Fundamentals we created tools by passing lambdas to AIFunctionFactory.Create. A real tool goes to the database, writes logs and knows who the user is; in other words it needs services from the DI (Dependency Injection) container. I group them in a class:

using System.ComponentModel;
using Bookstore.Assistant.Api.Data;
using Microsoft.EntityFrameworkCore;

namespace Bookstore.Assistant.Api;

// Scoped: created on every HTTP request with that request's DbContext and user
public sealed class OrderTools(BookstoreDbContext db, CurrentUser user, ILogger<OrderTools> logger)
{
    [Description("Returns the status of one of the current customer's orders.")]
    public async Task<object> GetOrderStatusAsync(
        [Description("The order number, for example 1042.")] int orderId,
        CancellationToken ct)
    {
        // The customer identity comes from the session, not from the model
        Order? order = await db.Orders.SingleOrDefaultAsync(o => o.Id == orderId && o.CustomerId == user.Id, ct);
        logger.LogInformation("get_order_status({OrderId}) for {User}: {Found}", orderId, user.Id, order is not null);

        // A user error is not an exception but a result the model can understand
        return order is null
            ? new { error = $"Order {orderId} was not found for this customer." }
            : new { order.Id, order.Status, order.BookTitle };
    }

    [Description("Cancels one of the current customer's orders. Requires the customer's approval.")]
    public async Task<object> CancelOrderAsync(
        [Description("The order number to cancel.")] int orderId,
        CancellationToken ct)
    {
        Order? order = await db.Orders.SingleOrDefaultAsync(o => o.Id == orderId && o.CustomerId == user.Id, ct);
        if (order is null)
        {
            return new { error = $"Order {orderId} was not found for this customer." };
        }

        // Idempotent: a second call doesn't change the state again, it returns the same result
        if (order.Status == "Cancelled")
        {
            return new { order.Id, order.Status, note = "Order was already cancelled." };
        }

        if (order.Status == "Shipped")
        {
            return new { error = $"Order {orderId} has already shipped and cannot be cancelled." };
        }

        order.Status = "Cancelled";
        await db.SaveChangesAsync(ct);
        logger.LogInformation("cancel_order({OrderId}) for {User}: cancelled", orderId, user.Id);
        return new { order.Id, order.Status };
    }
}

public sealed class CurrentUser(IHttpContextAccessor accessor)
{
    // Demo: a real app reads the claim coming from authentication
    public string Id => accessor.HttpContext!.Request.Headers["X-Demo-User"].ToString();
}

The [Description] attributes are the description of the tool and its parameters sent to the model; they are written into the JSON (JavaScript Object Notation) schema. The CancellationToken parameter doesn't appear in the schema; FunctionInvokingChatClient fills it with the request's token. The class is registered as scoped, and the tools are created per request from that request's instance:

// Tools are created per request, from that request's scoped OrderTools instance
static ChatOptions CreateOptions(OrderTools tools) => new()
{
    Tools =
    [
        AIFunctionFactory.Create(tools.GetOrderStatusAsync, "get_order_status"),
        new ApprovalRequiredAIFunction(AIFunctionFactory.Create(tools.CancelOrderAsync, "cancel_order")),
    ],
};

Why per request? Because the IChatClient pipeline is a singleton (post 3) while DbContext is scoped. If you create the tool once in a singleton place, it stays bound forever to the first request's DbContext and the first request's user. An AIFunction created from a method reference (tools.GetOrderStatusAsync) captures that instance; since the instance comes from the request's scope, everything binds to the right request.

Authorization Lives in the Session, Not in the Model's Arguments

Back to the opening question. If customerId were a tool parameter, the model would fill in its value, and the model is influenced by the text the user writes. "I'm actually grace" isn't even a prompt injection, it's an ordinary sentence. All it takes is for the model to believe that sentence and send customerId: "grace"; the tool then returns grace's order without asking. This is the LLM version of Broken Access Control from the Web Security (OWASP Top 10) series.

The rule is simple: on whose behalf a tool runs is not a parameter. OrderTools takes the user from CurrentUser, that is, from the request's identity, and adds the condition o.CustomerId == user.Id to every query. The model can choose the order number but not whose order it looks at. Order 1043 belongs to grace; when Ada asks:

$ ada: What is the status of order 1042?
{"answer":"Order 1042 is currently **Processing** with book title *Refactoring*."}
$ ada: What is the status of order 1043?
{"answer":"Order 1043 was not found for this customer. Would you like to check a different order number?"}

Both tool calls show up in the server log:

get_order_status(1042) for ada: True
get_order_status(1043) for ada: False

The answer is "not found", not "belongs to another customer"; the tool doesn't leak existence either. Note that I return the user error (a wrong order number) not as an exception but as a result object the model can read and explain to the user; that distinction is exactly the topic of the next section but one.

FunctionInvokingChatClient Settings

UseFunctionInvocation() without parameters comes with sensible defaults, but in production you need to choose each one deliberately:

builder.Services.AddChatClient(new OllamaApiClient(
        new HttpClient { BaseAddress = new Uri("http://localhost:11434"), Timeout = TimeSpan.FromMinutes(30) }, "qwen3:4b"))
    .UseFunctionInvocation(configure: f =>
    {
        f.MaximumIterationsPerRequest = 5;          // the model may ask for tools at most 5 rounds
        f.MaximumConsecutiveErrorsPerRequest = 2;   // exception after 2 consecutive tool errors
        f.AllowConcurrentInvocation = false;        // DbContext can't handle two queries at once
        f.IncludeDetailedErrors = false;            // keep exception messages from reaching the model and then the user
    });

The values come from the API (Application Programming Interface) documentation of Microsoft.Extensions.AI 10.11.0; since they may change between versions, check them in your own version.

Tool Error or User Error?

A tool can fail in two different ways. A user error: the order wasn't found, the order has shipped and can't be cancelled. These are part of the normal flow; the model should know so it can explain them to the user. A system error: the database is locked, the network dropped. These aren't things the model can fix, and their details shouldn't reach the user.

We saw the first above: the tool returns { error = "..." }. To see the second, a tool throwing an exception, without going to a model at all, I set up a small experiment: a fake IChatClient that first asks for a tool call and then records the tool result it receives, and a tool that throws a database error every time it's called:

using Microsoft.Extensions.AI;

AIFunction failing = AIFunctionFactory.Create(
    string (int orderId) => throw new InvalidOperationException("SQLite Error 5: 'database is locked'."),
    "get_order_status");

foreach (bool detailed in new[] { false, true })
{
    ScriptedModel model = new();
    IChatClient chat = model.AsBuilder()
        .UseFunctionInvocation(configure: f => f.IncludeDetailedErrors = detailed)
        .Build();

    await chat.GetResponseAsync("Where is order 1042?", new ChatOptions { Tools = [failing] });
    Console.WriteLine($"IncludeDetailedErrors={detailed}: model saw \"{model.LastToolResult}\"");
}

// Fake client that imitates the model: first asks for a tool call, then records the tool result it receives
sealed class ScriptedModel : IChatClient
{
    public object? LastToolResult { get; private set; }

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        FunctionResultContent? result = messages.SelectMany(m => m.Contents).OfType<FunctionResultContent>().LastOrDefault();
        if (result is null)
        {
            return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant,
                [new FunctionCallContent("call_1", "get_order_status", new Dictionary<string, object?> { ["orderId"] = 1042 })])));
        }

        LastToolResult = result.Result;
        return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Sorry, I could not check that order.")));
    }

    public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default) =>
        throw new NotSupportedException();

    public object? GetService(Type serviceType, object? serviceKey = null) => null;

    public void Dispose() { }
}

IncludeDetailedErrors=False: model saw "Error: Function failed."
IncludeDetailedErrors=True: model saw "Error: Function failed. Exception: SQLite Error 5: 'database is locked'."

With the default setting the model only sees "Error: Function failed."; the database name, the error code, maybe a table name don't reach the model and, through it, the user. The exception isn't lost either: it sits in FunctionResultContent.Exception and you can log it. IncludeDetailedErrors = true can help the model work around the error on its own in development, but in production it carries internal details outside. This experiment has a side benefit: with a fake model we tested FunctionInvokingChatClient's behavior deterministically, in milliseconds; we'll grow this idea in post 10.

Note the string (int orderId) => throw ... syntax too. When a lambda only throws, the compiler can't infer its return type, and the first attempt failed with error CS8917: The delegate type could not be inferred.; since C# 10 you can write an explicit return type on a lambda.

The Approval Flow: ApprovalRequiredAIFunction

Cancelling an order can't be undone. The model may read "what would happen if I cancelled?" as "cancel"; with small models such confusion can happen. That's why I wrapped cancel_order in ApprovalRequiredAIFunction. This wrapper doesn't change the tool itself; it tells FunctionInvokingChatClient "get approval before running this". When the model calls this tool, the FICC (FunctionInvokingChatClient) doesn't run the function; it puts a ToolApprovalRequestContent in the response and ends the loop.

In a web API that means two separate HTTP (Hypertext Transfer Protocol) requests. The first request returns with the approval request; the message history waits on the server. When the user approves, the second request arrives, a ToolApprovalResponseContent is appended to the history and the same pipeline is called again; this time the FICC runs the function and gives the result to the model.

app.MapPost("/chat", async (ChatRequest request, IChatClient chat, OrderTools tools, CurrentUser user,
    PendingApprovals pending, CancellationToken ct) =>
{
    List<ChatMessage> messages = [new(ChatRole.System, SystemPrompt), new(ChatRole.User, request.Text)];
    return await RunAsync(messages, chat, tools, user, pending, ct);
});

app.MapPost("/chat/approvals/{conversationId:guid}", async (Guid conversationId, ApprovalDecision decision,
    IChatClient chat, OrderTools tools, CurrentUser user, PendingApprovals pending, CancellationToken ct) =>
{
    // An approval is accepted only from the user who started it
    List<ChatMessage>? messages = pending.Take(conversationId, user.Id);
    if (messages is null)
    {
        return Results.NotFound();
    }

    ToolApprovalRequestContent request = messages
        .SelectMany(m => m.Contents)
        .OfType<ToolApprovalRequestContent>()
        .Last();
    messages.Add(new ChatMessage(ChatRole.User, [request.CreateResponse(decision.Approved, decision.Reason)]));

    return await RunAsync(messages, chat, tools, user, pending, ct);
});

static async Task<IResult> RunAsync(List<ChatMessage> messages, IChatClient chat, OrderTools tools,
    CurrentUser user, PendingApprovals pending, CancellationToken ct)
{
    ChatResponse response = await chat.GetResponseAsync(messages, CreateOptions(tools), ct);
    messages.AddMessages(response);

    ToolApprovalRequestContent? approval = response.Messages
        .SelectMany(m => m.Contents)
        .OfType<ToolApprovalRequestContent>()
        .FirstOrDefault();

    if (approval?.ToolCall is FunctionCallContent call)
    {
        Guid conversationId = pending.Save(user.Id, messages);
        return Results.Ok(new { conversationId, approvalRequired = new { tool = call.Name, call.Arguments } });
    }

    return Results.Ok(new { answer = response.Text });
}

The class holding pending conversations is deliberately simple; in memory for the demo, in a database as in post 5 for production. Two rules matter: an approval entry belongs to the user who started it, and it is single-use.

using System.Collections.Concurrent;
using Microsoft.Extensions.AI;

namespace Bookstore.Assistant.Api;

// Conversations waiting for approval. In memory for the demo; in production stored in a database as in post 5.
public sealed class PendingApprovals
{
    private readonly ConcurrentDictionary<Guid, (string UserId, List<ChatMessage> Messages)> _items = new();

    public Guid Save(string userId, List<ChatMessage> messages)
    {
        Guid id = Guid.NewGuid();
        _items[id] = (userId, messages);
        return id;
    }

    public List<ChatMessage>? Take(Guid id, string userId)
    {
        // Check ownership first; someone else's attempt must not delete the entry
        if (!_items.TryGetValue(id, out (string UserId, List<ChatMessage> Messages) item) || item.UserId != userId)
        {
            return null;
        }
        return _items.TryRemove(id, out _) ? item.Messages : null;
    }
}

In the first version of Take I called TryRemove first and checked the user afterwards. While writing the test scenario I noticed: with that version, grace trying Ada's approval id would get a 404 but would also delete the entry; Ada's own approval would no longer work. The order looks like a small detail, but it opened a door for an attacker to block someone else's action.

The flow end to end:

$ ada: Please cancel order 1042.
{"conversationId":"524495ab-d6ff-4a0b-b7f1-e371dbd780b3","approvalRequired":{"tool":"cancel_order","arguments":{"orderId":1042}}}
$ grace: approve 524495ab-d6ff-4a0b-b7f1-e371dbd780b3
HTTP/1.1 404 Not Found
$ ada: approve 524495ab-d6ff-4a0b-b7f1-e371dbd780b3
{"answer":"Order 1042 has been cancelled successfully."}
$ ada: approve 524495ab-d6ff-4a0b-b7f1-e371dbd780b3 again
HTTP/1.1 404 Not Found

# server log
cancel_order(1042) for ada: cancelled

The first request returned an approval request, not an answer text: the model wanted to call cancel_order with orderId: 1042, and the FICC didn't run it. When grace came with the same approval id, no entry was found. When Ada approved, the tool ran, its result went to the model and the model wrote the answer to the user. The cancel_order line appears once in the server log, after the approval. Sending the same approval again got a 404: the entry is single-use.

Idempotency: A Tool That Runs Twice

In post 3 we added retries to the provider's HttpClient; that retry repeats the HTTP request to the model, not the tool. What runs a tool twice comes from elsewhere: the client sending the same approval request twice, the user clicking the button twice, or the model calling the same tool twice in one round. I built two lines of defense. The first is at the API level: the approval entry is single-use, and a second approval gets a 404 (the last line above). The second is at the tool level: CancelOrderAsync returns "already cancelled" for an order that is already cancelled, without changing its state. For tools with side effects such as refunding money or sending e-mail this second line matters even more; there you usually need an idempotency key (for example the approval id) to check "has this already been done?".

In Production

Log every tool call: which user, which tool, which arguments, which result. The get_order_status(1043) for ada: False line above is the only trace showing that a user tried to look up someone else's order. In post 8 we'll carry this information into OpenTelemetry spans.

Give tools by role and permission. You don't have to expose every tool to every user; CreateOptions can return a different list depending on the user's role. The model can't call a tool it can't see.

Show the model's arguments on the approval screen. The approval shouldn't be "the model wants to do something, do you allow it?" but "do you want to cancel order 1042, Refactoring?". Build that information from the ToolCall's arguments and your own database, not from text the model wrote.

Common Pitfalls

First, leaving authorization to the model's arguments. "On whose behalf" information such as customerId, userId or tenantId shouldn't be a tool parameter; it should come from the request's identity.

Second, creating the tool once in a place with a singleton lifetime. The first request's DbContext and user stick to the tool; you either get an ObjectDisposedException on the second request or, worse, see the wrong user's data.

Third, leaving a non-idempotent tool without approval and without protection. A retry, a double click or the model repeating the same call leads to outcomes like refunding the same order twice.

Fourth, leaving IncludeDetailedErrors on in production "to make debugging easier". Exception messages carry internal details such as connection addresses, table names and file paths, and the model will politely read them to the user.

A Small Experiment

★☆☆ Set MaximumIterationsPerRequest to 1 and ask "what's the status of 1042?". What happens? Why must this setting be "at least one"?

★★☆ Add an expiry to approval requests: PendingApprovals should store the creation time and reject an approval older than 5 minutes. Get the time from TimeProvider and advance it with FakeTimeProvider in a test.

★★★ Using the ScriptedModel idea, write a test for the approval flow: the fake model first asks for a cancel_order call; verify that the first response contains a ToolApprovalRequestContent, that CancelOrderAsync runs exactly once after approval, and that it never runs when rejected.

The assistant can now do work, which also means every request has a cost. In the next post we bring that cost under control: DistributedCachingChatClient with Redis, what goes into the cache key, per-user rate limiting and token accounting with UsageDetails.