How OpenTelemetry Is Structured and a First Setup in .NET: SDK, Instrumentation, OTLP, Resource and the Aspire Dashboard

Version note: The examples in this post were run with .NET 10, OpenTelemetry .NET 1.21 (OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol and the AspNetCore, Http, SqlClient and Runtime instrumentations 1.21.0) and the Aspire Dashboard 13.6.5 container.

Guess First

In the previous post we saw that checkout in Bookstore, the online bookstore used as the shared example across this site's series, sometimes takes 8 seconds. In this post we add OpenTelemetry to Bookstore.Api. In code you name the service bookstore-api, and at deployment you set the OTEL_SERVICE_NAME=bookstore-api-env environment variable. Under which name does the service show up in the telemetry backend? And another one: if the telemetry endpoint is mistyped and the telemetry can't go anywhere, how does the application tell you? We tried both; the answers are in the middle of the post.

The Parts of OpenTelemetry

OpenTelemetry is not a single library but a standard made of separate parts. The specification defines the data model and behavior of the signals (traces, metrics, logs). The API is the surface that application and library code call to produce telemetry; when nobody listens it costs almost nothing. The SDK is the actual implementation behind the API: sampling, aggregation, batching and export live here. Instrumentation libraries bridge frameworks that don't produce telemetry themselves, or don't produce it the way OpenTelemetry expects: ASP.NET Core, HttpClient, SqlClient and so on. Exporters send the data somewhere; the standard target is OTLP, on port 4317 over gRPC and 4318 over HTTP. The Collector is a separate process that receives OTLP, processes it and distributes it to backends (the subject of the fifth post). Semantic conventions are the shared vocabulary of names: the duration of an HTTP request is http.server.request.duration, its route http.route, the database system db.system.name. Backends and dashboards rely on these names.

What sets .NET apart from other languages is that most of the API layer is inside the platform. ILogger for logs, System.Diagnostics.Metrics.Meter for metrics, System.Diagnostics.ActivitySource and Activity for traces already exist in .NET, and ASP.NET Core and HttpClient produce telemetry with them. The OpenTelemetry .NET SDK listens to these platform APIs and turns the data into the OpenTelemetry model. In practice, your code can use Meter and ActivitySource without depending on an OpenTelemetry package; you set up the SDK once, at application startup.

Adding OpenTelemetry to Bookstore.Api

We added the packages with central package management (Directory.Packages.props):

<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.21.0" />
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.21.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.21.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.21.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.21.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.21.0" />

All of them are stable releases. That was not always the case: the SqlClient instrumentation stayed in beta for a long time, and its first stable release was 1.15.0 in January 2026. The Entity Framework Core instrumentation is still in beta (1.21.0-beta.1); since EF Core talks through SqlClient anyway, the SqlClient instrumentation is enough for SQL Server. One more note: 1.21.0, released on 11 November 2026, removed the net8.0 and net9.0 targets from the packages and targets .NET 10 and 11. A project on .NET 8 should read the changelog before moving to this version.

The whole setup is one extension method:

public static WebApplicationBuilder AddBookstoreObservability(this WebApplicationBuilder builder)
{
    string version = typeof(ObservabilityExtensions).Assembly
        .GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion ?? "unknown";

    builder.Services.AddOpenTelemetry()
        // Who is sending: these attributes are attached to every span, metric point and log record.
        .ConfigureResource(resource => resource
            .AddService(serviceName: "bookstore-api", serviceVersion: version)
            .AddAttributes([new("deployment.environment.name", builder.Environment.EnvironmentName.ToLowerInvariant())]))
        .WithTracing(tracing => tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation())
        .WithMetrics(metrics => metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation())
        .WithLogging()
        // One OTLP exporter for all three signals; the endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT.
        .UseOtlpExporter();

    return builder;
}

One line in Program.cs: builder.AddBookstoreObservability();. A few details become clear from the source code. On the metrics side, AddAspNetCoreInstrumentation() on .NET 8 and later doesn't measure anything itself; it subscribes to ASP.NET Core's built-in meters (Microsoft.AspNetCore.Hosting, Microsoft.AspNetCore.Server.Kestrel, Microsoft.AspNetCore.Routing, Authentication, Authorization, Identity and others), and on runtimes older than .NET 11 it also installs a small middleware that writes error.type on 5xx responses. AddRuntimeInstrumentation() likewise registers the runtime's own System.Runtime meter on .NET 9 and later. On the tracing side the instrumentation packages do real work: they write the semantic convention attributes on the spans. UseOtlpExporter() sets up a single OTLP exporter for logs, metrics and traces; it cannot be combined with the per-signal AddOtlpExporter on the same builder.

Resource: Who Is Sending the Telemetry?

The resource is the set of attributes describing what produces the telemetry: every span, every metric point and every log record travels with the same resource. The most important ones are service.name (the key that separates services in backends), service.version and deployment.environment.name. For the version we used the assembly's informational version; when the project is in a git repository, the .NET SDK also appends the commit hash to it during the build. That makes "which deployment did this slowdown start with" answerable from the telemetry; versioning itself is the subject of the Versioning and Release Management series. deployment.environment.name is the current semantic convention name of the older deployment.environment attribute; you may still meet dashboards and queries that use the old name.

OpenTelemetry also lets you set the resource through environment variables: OTEL_SERVICE_NAME and the comma-separated OTEL_RESOURCE_ATTRIBUTES. We tried the first question from Guess First with this code, and the result was not what we expected. The application running with OTEL_SERVICE_NAME=bookstore-api-env and OTEL_RESOURCE_ATTRIBUTES=service.namespace=bookstore,deployment.environment.name=local had this resource:

service.name=bookstore-api
service.instance.id=428f78b5-ae79-4675-b0e1-ae89d7c3ad74
deployment.environment.name=development
service.namespace=bookstore
service.version=1.0.0+9430f6b1c6d43694f75c9007a1730186013e51ea
telemetry.sdk.language=dotnet
telemetry.sdk.name=opentelemetry
telemetry.sdk.version=1.21.0

service.namespace from the environment variable was added to the resource, but for the same keys the code won: the service name stayed bookstore-api and the environment development. The reason is the order inside the resource builder. The SDK runs the environment variable detector first by default; then the values you give in code are added, and whatever comes later overrides what came before. Leave everything you want to be overridable by an environment variable as a default in code and add the detector again at the end:

.ConfigureResource(resource => resource
    .AddService(serviceName: "bookstore-api", serviceVersion: version)
    .AddAttributes([new("deployment.environment.name", builder.Environment.EnvironmentName.ToLowerInvariant())])
    // Code gives the defaults; OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES, read last, override them.
    .AddEnvironmentVariableDetector())

Running again with the same environment variables, the service arrived as bookstore-api-env and the environment as local. We didn't set service.instance.id: the SDK generates a random GUID per process, which is how two copies of the same service are told apart.

Seeing It Locally: The Standalone Aspire Dashboard

To see the telemetry you need a backend. For local development the shortest path is the Aspire Dashboard container: an OTLP receiver, the trace waterfall, structured logs and metric screens in one container. Your project doesn't need to use Aspire; the dashboard post of the .NET Aspire series covered this usage through the CLI, and we used the container:

docker run -d --name bookstore-dashboard -p 18888:18888 -p 4317:18889 -p 4318:18890 -e Dashboard__Api__AuthMode=ApiKey -e Dashboard__Api__PrimaryApiKey=<api-key> mcr.microsoft.com/aspire/dashboard:13.6.5

docker logs bookstore-dashboard
      Login to the dashboard at http://localhost:18888/login?t=<token>
      OTLP server is unsecured. Untrusted apps can send telemetry to the dashboard. ...

Inside the container OTLP/gRPC listens on 18889 and OTLP/HTTP on 18890; we published them as the standard 4317 and 4318, so OpenTelemetry's default endpoint works on the application side. The UI opens with the token from the log. The Dashboard__Api__* settings open the dashboard's telemetry API with an API key; the text outputs in this post come from that API. We started the application with these environment variables (PowerShell syntax):

$env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4317"
$env:OTEL_RESOURCE_ATTRIBUTES = "service.namespace=bookstore"
dotnet run --project src/Bookstore.Api

After a traffic run of 20 orders we opened the slowest POST /api/orders/ trace on the dashboard's Traces page. We give the waterfall as text, built from the spans read through the dashboard's telemetry API (columns: start ms, duration ms):

      0.0    5140.3  POST /api/orders/
      0.6       1.3    SELECT [c].[Id] [c].[CreatedAtUtc] [c].[UpdatedAtUtc] [ordering].[Carts] [ordering].[CartItems]
      2.2       1.0    SELECT [catalog].[Books]
      3.4       0.8    SELECT [inventory].[StockItems]
      4.3       0.6    SELECT [inventory].[StockItems]
      5.0       0.7    SELECT [inventory].[StockItems]
      6.4       0.8    UPDATE
      7.3       0.7    UPDATE
      8.2       0.7    UPDATE
      9.0       0.5    SELECT [ordering].[Customers]
     10.1       0.5    BookstoreDb
     10.7       0.9    DELETE
     11.9       1.3    DELETE
     15.9    2003.9    POST ERROR
   2539.7    2001.5    POST ERROR
   5042.1      91.1    POST
   5133.6       6.4    UPDATE

The answer to the previous post's "where did the time go?" is on one screen. The whole SQL part takes 13 ms. The first two POST calls to the payment provider were cut at 2,000 ms and marked as errors, with a 500 ms wait between them, and the third returned in 91 ms. Five of the 5.1 seconds are the retry loop. Something else shows up without writing any code: for a cart with three books, three separate SELECT and three separate UPDATE statements go to the stock table. Cheap for now, but 24 queries for a 12-book institutional order; we will come back to this later in the series.

Span names come from the semantic conventions. A server span is {method} {route}, a client span only the method (the full URL has high cardinality, so it lives in the url.full attribute rather than the name), and a SQL span the db.query.summary that the SqlClient instrumentation extracts from the statement. The span named BookstoreDb is a command whose summary could not be extracted: its db.query.text shows EF Core's SAVE TRANSACTION [__EFSavePoint];. The attributes of the attempt that timed out:

http.request.method=POST
server.address=localhost
server.port=5090
url.full=http://localhost:5090/v1/charges
error.type=System.Threading.Tasks.TaskCanceledException

Logs and Metrics with the Same Resource

Thanks to WithLogging(), ILogger records go out over OTLP too. The order's log record looks like this in the dashboard (shortened from the telemetry API's JSON):

{"severityText": "Information",
 "body": {"stringValue": "Order {OrderId} placed: {ItemCount} items, {Total} {Currency}, took {ElapsedMs} ms"},
 "attributes": [{"key": "OrderId", "value": {"stringValue": "408"}}, ... {"key": "ElapsedMs", "value": {"stringValue": "5139"}}],
 "traceId": "67906de9a63b40347b19898355db93e2", "spanId": "7b8d74d9fd18ed92"}

Two things to note. The record has traceId and spanId; those are the fields that let you jump from a log line to the trace with one click in the dashboard. And body is the message template, not the formatted message; the parameters are separate attributes. This makes filtering by OrderId in a log backend easy. If you also want the formatted text, WithLogging's options have IncludeFormattedMessage.

Without a single line of metric code, the Metrics page listed these meters: Microsoft.AspNetCore.Hosting (http.server.request.duration, http.server.active_requests), Microsoft.AspNetCore.Server.Kestrel (connection durations and queues), Microsoft.AspNetCore.Routing, Microsoft.AspNetCore.MemoryPool, System.Net.Http (including http.client.request.duration), System.Net.NameResolution and System.Runtime (GC, JIT, thread pool, exception counts). What they tell you, and our own metrics, are the subject of the next post.

What Happens When Telemetry Can't Get Through?

The second question from Guess First. We started the application with OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4999, a port nobody listens on, and sent a request:

curl -s -o /dev/null -w "%{http_code} %{time_total}\n" http://localhost:5080/api/categories
200 0.154831

The request returned normally and there is not a single line about an export failure on the application's console. This is deliberate: telemetry that can't be delivered must not slow down or stop the application. Exporter errors go to the SDK's own EventSource and only show up if you turn on OpenTelemetry's self-diagnostics. We put an OTEL_DIAGNOSTICS.json into the folder the application runs from and repeated the test:

{"LogDirectory": ".", "FileSize": 1024, "LogLevel": "Warning"}

[2026-11-14T19:13:16.5938510Z][518806][------][Warning]     OpenTelemetry-Exporter-OpenTelemetryProtocol: Transient HTTP error occurred when communicating with http://localhost:4999/opentelemetry.proto.collector.logs.v1.LogsService/Export. Exception: System.Net.Http.HttpRequestException: Connection refused (localhost:4999)

The SDK opened a file named otel-dotnet-<pid>-...log in that folder and wrote the error there. So when "the dashboard is empty", the first place to look is not the application's log; it is the endpoint, the port, the protocol (4317 gRPC, 4318 HTTP) and the API key, if there is one.

ServiceDefaults Without Aspire

The ServiceDefaults project in Aspire's project template does almost exactly the setup we wrote here: AddOpenTelemetry(), the ASP.NET Core, HttpClient and runtime instrumentations, UseOtlpExporter(), plus health check and service discovery settings. Even without an AppHost you can reference that project and call builder.AddServiceDefaults(); the only difference is that you provide the OTLP endpoint and service name through environment variables instead of the AppHost. We didn't do that in Bookstore.Api; we wrote our own extension method to see every line of the setup. Even if you set up ServiceDefaults, the resource ordering pitfall above applies just the same.

In Production

Keep the service name and the environment settable at deployment. If the same image runs in staging and production, the environment must come from an environment variable, not from code; check where the detector sits in the resource builder.

Locally everything is sampled. The SDK's default sampler records everything, following the parent's decision; in production keeping every span of every request is expensive. We handle the sampling decision in the Collector in the fifth post.

The dashboard's OTLP port is unprotected by default; the warning in the log says so. That's fine on a local machine, but if you open the dashboard in a shared environment, require an API key for OTLP too. The Aspire Dashboard is also not designed for production: by default it keeps a limited amount of data in memory and you can't define alerts.

db.query.text carries the text of the query. Because EF Core sends parameters by name, like @request_CartId, the values don't end up in the span; code that builds queries by string concatenation puts the values into the text as well. Check that your telemetry carries no personal data before going to production.

Common Pitfalls

Setting up the exporter and forgetting the instrumentation. UseOtlpExporter() on its own only sends what somebody produces; without AddAspNetCoreInstrumentation() there is no request span, without AddSqlClientInstrumentation() no SQL spans, and you see no error anywhere.

Taking prerelease packages to production without noticing. Some instrumentation packages (EF Core, the gRPC client) are still in beta; a -beta suffix in the version means attribute names and behavior can change. Check on NuGet and in the changelog which packages are stable.

Leaving the console exporter on in production. It is useful while trying things out, but writing every span and metric to stdout is expensive and creates a second copy of the telemetry in your log collector.

Not knowing that AddService in code overrides the environment variable. As we saw above, there is no error and no warning; the service just shows up under a different name than you expect.

Looking for errors in the application's log when the dashboard is empty. Export errors are silent; check the endpoint, the port and the protocol first.

A Small Experiment

★☆☆ Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf and point the endpoint at http://localhost:4318. Does data still reach the dashboard? What happens if you change only the protocol and leave the endpoint at 4317?

★★☆ Set IncludeFormattedMessage = true through WithLogging's configureOptions parameter and look at how the body of the log record in the dashboard changes. What does that do to the cost of a log backend?

★★★ Start the dashboard with Dashboard__Otlp__AuthMode=ApiKey and Dashboard__Otlp__PrimaryApiKey so that OTLP requires an API key as well. Run the application first without OTEL_EXPORTER_OTLP_HEADERS=x-otlp-api-key=<key>, then with it. Which error shows up in the self-diagnostics file the first time?

Telemetry is now produced and goes to a backend, but so far it is all what the frameworks give us. The next post focuses on metrics: writing Bookstore's own business metrics with Counter, Histogram, UpDownCounter and Gauge, what the built-in metrics tell you, unit testing a metric, and what the SDK does when the cardinality budget is exceeded.