Observing LLM Calls: UseOpenTelemetry, GenAI Semantic Conventions and the Aspire Dashboard

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, OllamaSharp 5.5.2, OpenTelemetry .NET 1.20.1, Aspire Dashboard 13.6.4 and Ollama 0.34.4 (qwen3:4b).

Guess First

Someone asks the Bookstore assistant "Where is my order 1042?" and the answer arrives 1 minute 54 seconds later. The answer is correct: the order has shipped and the tracking number is there. All you have is ASP.NET Core's request log, which shows a single line: POST /assistant/ask, 200, 114 seconds.

Where did those two minutes go? Is the model slow, is the tool slow, or did the model do the same work twice? And one more question: do you need to write the user's email address to a log to find out?

By the end of the post we will read that request's trace span by span. The answer is not "the model is slow", which is most people's first guess.

Why an LLM Call Deserves Its Own Observability

For a normal HTTP (Hypertext Transfer Protocol) dependency, duration and status code are usually enough. For an LLM (Large Language Model) call three more things matter: how many tokens went in and came out (that is where cost comes from), whether the model asked for tools and how many rounds it took, and which model was called with which parameters. Each of these appeared as a concept in the LLM Fundamentals series; here the goal is to collect them automatically on every request.

OpenTelemetry is the shared language for this: traces (a tree of spans for one request), metrics and logs. The LLM-specific naming comes from the GenAI (Generative Artificial Intelligence) semantic conventions. On the Microsoft.Extensions.AI side, the piece that implements this standard is OpenTelemetryChatClient, which enters the pipeline through UseOpenTelemetry(). The pipeline itself and the order of Use... calls were covered in this series' "The IChatClient Pipeline" post; here we look at it only from the telemetry angle.

Your First Trace with UseOpenTelemetry

We add the OpenTelemetry packages to the Bookstore.Assistant.Api project. The OTLP (OpenTelemetry Protocol) exporter will send telemetry to the dashboard, and the console exporter lets us see spans in the terminal:

<ItemGroup>
  <PackageReference Include="Microsoft.Extensions.AI" Version="10.11.0" />
  <PackageReference Include="OllamaSharp" Version="5.5.2" />
  <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.20.1" />
  <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.20.1" />
  <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.20.1" />
  <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.20.2" />
</ItemGroup>

The whole Program.cs:

using System.ComponentModel;
using Microsoft.Extensions.AI;
using OllamaSharp;
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

const string AiSource = "Bookstore.Assistant.AI";
bool sensitive = builder.Configuration.GetValue<bool>("Ai:EnableSensitiveData");

builder.Services.AddSingleton<OrderTools>();

// First Use is outermost: function invocation → telemetry → logging → Ollama
builder.Services
    .AddChatClient(new OllamaApiClient(new Uri("http://localhost:11434"), "qwen3:4b"))
    .UseFunctionInvocation()
    .UseOpenTelemetry(sourceName: AiSource, configure: c => c.EnableSensitiveData = sensitive)
    .UseLogging();

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("bookstore-assistant-api"))
    .WithTracing(t => t
        .AddSource(AiSource)
        .AddAspNetCoreInstrumentation())
    .WithMetrics(m => m
        .AddMeter(AiSource)
        .AddAspNetCoreInstrumentation())
    .WithLogging()
    .UseOtlpExporter();

// Only for seeing spans in the terminal locally
if (builder.Configuration.GetValue<bool>("Telemetry:Console"))
{
    builder.Services.ConfigureOpenTelemetryTracerProvider(t => t.AddConsoleExporter());
}

WebApplication app = builder.Build();

app.MapPost("/assistant/ask", async (AskRequest request, IChatClient chat, OrderTools tools, CancellationToken ct) =>
{
    ChatOptions options = new()
    {
        Temperature = 0,
        Seed = 42,
        Tools = [AIFunctionFactory.Create(tools.GetOrderStatus)],
    };

    ChatResponse response = await chat.GetResponseAsync(
        [
            new(ChatRole.System, "You are the Bookstore assistant. Use tools for order questions. Answer in one sentence."),
            new(ChatRole.User, request.Question),
        ],
        options, ct);

    return Results.Ok(new
    {
        answer = response.Text,
        inputTokens = response.Usage?.InputTokenCount,
        outputTokens = response.Usage?.OutputTokenCount,
    });
});

app.Run();

public sealed record AskRequest(string Question);

public sealed class OrderTools
{
    [Description("Returns the shipping status of a Bookstore order.")]
    public string GetOrderStatus([Description("The order number")] int orderId) =>
        orderId == 1042 ? "Shipped on 2026-11-06, 2 items, tracking BKS-1042." : "Order not found.";
}

Two details are critical. The first is sourceName: OpenTelemetryChatClient emits its spans through an ActivitySource and its metrics through a Meter, both with that name. If you don't write AddSource(AiSource) on the tracer and AddMeter(AiSource) on the meter provider, the SDK (Software Development Kit) never listens to that data and you see no error either; the telemetry just disappears. If you omit the name, the library default Experimental.Microsoft.Extensions.AI is used. The second is UseOtlpExporter(): it routes traces, metrics and logs to OTLP in one call and reads the target from the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

We run the app with the console exporter on and send a request:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:19651 Telemetry__Console=true \
  ASPNETCORE_URLS=http://localhost:19660 dotnet run

curl -s localhost:19660/assistant/ask -H "Content-Type: application/json" \
  -d '{"question":"Where is my order 1042?"}'
{"answer":"Your order 1042 is shipped on 2026-11-06 with 2 items and tracking number BKS-1042.","inputTokens":610,"outputTokens":546}

The first span in the terminal is the first call to the model:

Activity.TraceId:            a3411ea2ab92fe28dcd913bb4647b5f4
Activity.SpanId:             7af35b22a6b59e1f
Activity.ParentSpanId:       0c08514a277f5acb
Activity.DisplayName:        chat qwen3:4b
Activity.Kind:               Client
Activity.Duration:           00:00:05.8162760
Activity.Tags:
    gen_ai.operation.name: chat
    gen_ai.request.model: qwen3:4b
    gen_ai.provider.name: ollama
    server.address: localhost
    server.port: 11434
    gen_ai.request.seed: 42
    gen_ai.request.temperature: 0
    gen_ai.tool.definitions: [
  {
    "type": "function",
    "name": "GetOrderStatus"
  }
]
    gen_ai.response.finish_reasons: ["stop"]
    gen_ai.response.id: 2026-11-08T06:53:05.549717Z
    gen_ai.response.model: qwen3:4b
    gen_ai.usage.input_tokens: 169
    gen_ai.usage.output_tokens: 217
Instrumentation scope (ActivitySource):
    Name: Bookstore.Assistant.AI

The same trace has five spans in total. Following the ParentSpanId values in the console output gives this tree:

POST /assistant/ask                  22.95 s   (Microsoft.AspNetCore)
└─ orchestrate_tools                 22.90 s   input 610, output 546
   ├─ chat qwen3:4b                   5.82 s   input 169, output 217
   ├─ execute_tool GetOrderStatus     0.94 ms
   └─ chat qwen3:4b                  17.07 s   input 441, output 329

One HTTP request went to the model twice. The first call asked for a tool, FunctionInvokingChatClient ran the tool (under 1 millisecond), and the second call turned the result into a sentence. The orchestrate_tools span wraps the whole loop and sums the tokens of both calls: 169 + 441 = 610 input, 217 + 329 = 546 output. The inputTokens and outputTokens returned by the endpoint are that sum. The second call's input grew to 441 because the first round's messages and the tool result are sent again.

GenAI Semantic Conventions: Reading the Attributes

Span names and attributes are not arbitrary. Per the conventions, an inference span is named {gen_ai.operation.name} {gen_ai.request.model} and a tool span execute_tool {gen_ai.tool.name}. The attributes you will use most:

There is one thing to watch here: these conventions are still in "Development" status, and in 2026 they moved out of OpenTelemetry's main semantic-conventions repository into their own repository (open-telemetry/semantic-conventions-genai). The XML (Extensible Markup Language) documentation of Microsoft.Extensions.AI 10.11 states plainly that OpenTelemetryChatClient implements v1.41 of the conventions and that its output may change. You can see the gap in the metrics. The two GenAI metrics this app sends to the dashboard are gen_ai.client.operation.duration and gen_ai.client.token.usage. The draft in the new repository uses gen_ai.client.inference.duration for inference duration and splits tokens into separate counters such as gen_ai.client.inference.usage.input_tokens. If your alerts and dashboard queries depend on metric names, you will need to check them after a package update.

There is also a small Ollama-specific quirk: gen_ai.response.id is a timestamp. Ollama's response has no separate id, so OllamaSharp uses its created_at value. With cloud providers this holds the provider's response id.

Pipeline Order Changes the Trace

In the order above, UseFunctionInvocation() is outside and UseOpenTelemetry() is inside. The pipeline example on Microsoft Learn uses the same order. That way the telemetry layer sees every single call to the model. Let's flip it:

builder.Services
    .AddChatClient(new OllamaApiClient(new Uri("http://localhost:11434"), "qwen3:4b"))
    .UseOpenTelemetry(sourceName: AiSource, configure: c => c.EnableSensitiveData = sensitive)
    .UseFunctionInvocation()
    .UseLogging();

Asking the same question again, the trace shrinks to two spans:

Activity.DisplayName:        chat qwen3:4b
Activity.Duration:           00:00:48.9658390
    gen_ai.usage.input_tokens: 608
    gen_ai.usage.output_tokens: 562
Activity.DisplayName:        POST /assistant/ask
Activity.Duration:           00:00:49.0080990

The answer is still correct and the total tokens are still visible. But now two model rounds look like one "chat" span, and there is no execute_tool span. You can no longer answer any of these questions: how many rounds did the model take, how many milliseconds did the tool take, in which round did the prompt grow? When telemetry sits outside, the function invocation loop looks like a single call to it. The practical rule: keep UseOpenTelemetry() inside UseFunctionInvocation(), and also inside layers that might skip the call entirely, such as cache and rate limiting. A response served from cache should not produce a model span; every call that really reaches the model should.

Reading Traces Locally with the Aspire Dashboard

The console exporter is fine for one request, but you have to build the tree yourself. The standalone Aspire Dashboard container gives you a local OTLP receiver and a UI; you don't need an Aspire AppHost:

docker run -d --name aiwork2-aspire \
  -p 19650:18888 -p 19651:18889 \
  -e DASHBOARD__FRONTEND__AUTHMODE=Unsecured \
  mcr.microsoft.com/dotnet/aspire-dashboard:13.6.4

Inside the container, 18888 is the UI and 18889 is the OTLP/gRPC (gRPC Remote Procedure Calls) port. We mapped them to 19650 and 19651 and gave the app OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:19651. Unsecured is only for your own machine: the dashboard shows a warning on start that the endpoint is not secured, because any application can send data to the OTLP port.

On the Traces page a small GenAI icon appears next to each POST /assistant/ask row. The dashboard recognizes spans that carry gen_ai.* attributes; clicking the icon opens a GenAI details window that shows the system message, the user message, the assistant's tool calls and the span's token count as a chat view. On the Metrics page, the two metrics above sit under the Bookstore.Assistant.AI meter.

Where Did Two Minutes Go?

Back to the 114-second request from the opening. In its trace orchestrate_tools took 1 minute 53 seconds and the first chat qwen3:4b span took 1 minute 36 seconds. Is the model slow? The UseLogging() output of the same call also shows the timings OllamaSharp adds to the response:

"usage": {
  "inputTokenCount": 175,
  "outputTokenCount": 302,
  "totalTokenCount": 477
},
"additionalProperties": {
  "load_duration": "00:00:00.0019730",
  "total_duration": "00:01:36.7374836",
  "prompt_eval_duration": "00:00:00.1848700",
  "eval_duration": "00:00:06.8685780"
}

Generating 302 tokens took 6.9 seconds, processing the prompt 0.18 seconds, loading the model 2 milliseconds. For the remaining 89 seconds or so, the request waited for its turn inside Ollama. While this post was being prepared, other jobs were using the same Ollama. So neither the model nor the tool was slow; the bottleneck was the queue on a shared model server. To make that diagnosis you need two things: the span duration and the provider's own timing breakdown. OpenTelemetryChatClient always gives you the first. Ollama gives you the second, and it is added to the span as attributes too. With cloud providers the equivalent is usually a response header or a less detailed usage field.

The same trace shows one more thing: qwen3:4b is a thinking model, and most of the first round's 302 output tokens are reasoning. That reasoning goes back into the second round as input through the history, which feeds straight into cost and latency. You only notice something like this when you see the content. So what does seeing the content cost you?

EnableSensitiveData and UseLogging: The Price of Seeing Content

By default OpenTelemetryChatClient writes only metadata: model, tokens, duration, tool name. Even the gen_ai.tool.definitions attribute above contains only the tool's name and type, without its description or parameter schema. With EnableSensitiveData = true (or when the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true environment variable is set) the full messages go into the span. We ran the same app with Ai__EnableSensitiveData=true and asked a question containing an email:

    gen_ai.input.messages: [
  ...
  {
    "role": "user",
    "parts": [
      {
        "type": "text",
        "content": "Where is my order 1042? My email is ada@example.com"
      }
    ]
  }
]
...
    gen_ai.tool.call.arguments: {
  "orderId": 1042
}
    gen_ai.tool.call.result: "Shipped on 2026-11-06, 2 items, tracking BKS-1042."

The user's email address is now in the trace. You see the sentence verbatim in the Aspire Dashboard's GenAI details window as well. The model's reasoning also says "They provided their email as ada@example.com", so the address enters the span from at least two places. The tool result (order status, tracking number) is in gen_ai.tool.call.result.

UseLogging() carries the same risk on the log side. LoggingChatClient writes messages and options only at the Trace log level; the log output in this post was captured with Logging__LogLevel__Microsoft.Extensions.AI=Trace. At the default Information level it writes nothing. The library's own documentation says this level should never be enabled in production.

A healthy setup looks like this: in production EnableSensitiveData is off and comes from configuration; content is turned on only in development or in a short, access-restricted debugging session. Sampling keeps trace volume down:

double ratio = builder.Configuration.GetValue("Telemetry:SampleRatio", 1.0);

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t
        .SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(ratio)))
        .AddSource(AiSource)
        .AddAspNetCoreInstrumentation())
    .UseOtlpExporter();

ParentBasedSampler follows the upstream service's decision; when the trace starts at the root span, it samples ratio of requests. Sampling affects only traces; token metrics are still recorded on every request, so cost tracking is not affected by sampling.

In Production

• The token metric is the raw data behind LLM cost. Aggregate the gen_ai.client.token.usage histogram by model and gen_ai.token.type (input/output); this is the observability side of the budget calculation in this series' "Controlling LLM Cost" post.
• Compare span duration with the provider's own timings. If the gap is large, the problem is not the model but a queue, the network or a rate limit. If you run your own Ollama, that is your first scaling signal.
• Look at tool spans. If a tool is called noticeably more often than there are questions, the model keeps asking for the same tool. Fix the tool description or its return format.
• Track prompt growth through the increase in input tokens across rounds. In this trace it went from 169 to 441; it grows as chat history and reasoning accumulate.
• Keep telemetry settings in configuration, not code: the source name is fixed, but EnableSensitiveData, the sampling ratio and the OTLP endpoint change per environment.

Collectors, backend choice and log/trace correlation in detail belong to the upcoming Observability series.

Common Pitfalls

The most common pitfall is a mismatch between AddSource and sourceName. If you write UseOpenTelemetry(sourceName: "Bookstore.Assistant.AI") and give the tracer a different name, or none, the OpenTelemetry SDK never listens to the spans; the app runs, HTTP spans arrive, GenAI spans don't. Metrics need the same name in AddMeter.

The second is putting telemetry in the wrong place in the pipeline. We saw it above: placed outside UseFunctionInvocation(), tool rounds become invisible. Placed outside the cache, cached responses count as model calls too and your duration metric drifts down.

The third is carrying content logging settings from development to production. One line from appsettings.Development.json slipping into the main file means every user message flows into your telemetry backend. That data is usually accessible to a wider team than your application database. From a GDPR (General Data Protection Regulation) or KVKK (Turkey's Personal Data Protection Law) point of view it also becomes a separate data processing location.

The fourth is the API (Application Programming Interface) key leaking into telemetry. OpenTelemetryChatClient does not write it, but if HTTP client instrumentation or a log line of your own writes headers, the key leaks from there. The secret management from this series' "Wiring IChatClient into ASP.NET Core" post applies on the telemetry side too.

Finally, treating convention names as permanent. The GenAI conventions are in Development status. After a package update, don't trust your alert rules until you have checked the metric and attribute names in the dashboard.

A Small Experiment

★☆☆ Delete the AddSource(AiSource) line and send the same request. Which span remains in the console? Then put the line back and remove the sourceName argument; which name do you have to give the tracer to see the spans again?

★★☆ Add UseDistributedCache() to the pipeline (an in-memory IDistributedCache is enough). Put it first inside telemetry, then outside, and ask the same question twice. In which case does the second request produce a chat qwen3:4b span, and in which case not?

★★★ Write your own DelegatingChatClient that adds a bookstore.feature attribute to Activity.Current, reading the value from ChatOptions.AdditionalProperties. Where in the pipeline do you have to place it to see the attribute on the chat span, and where to see it on orchestrate_tools?

Now you can see every LLM call. But seeing it doesn't prove the answer is good. In the next posts we first build our first vector collection with Microsoft.Extensions.VectorData, then move on to testing code that uses IChatClient and measuring the quality of the model's answers.