Producing Metrics in .NET: Counters, Histograms, Gauges, Built-in Metrics and a Cardinality Budget

Version note: The examples in this post were run with .NET 10, OpenTelemetry .NET 1.21, Microsoft.Extensions.Diagnostics.Testing 10.11, xUnit v3 4.0 and dotnet-counters 10.0.

Guess First

We added a metric to Bookstore, the online bookstore used as the shared example across this site's series, that tells "how many checkouts are in progress right now": it goes up by one when a checkout starts and down by one when it ends. While traffic was running we attached dotnet-counters to the application and read this value:

bookstore.checkouts.active ({checkout}) = -3

How can the number of checkouts in progress be negative? Is the bug in the code or on the measuring side? The answer has to do with how instrument types work; it is in the middle of the post.

Instrument Types: Which One When?

In .NET's System.Diagnostics.Metrics API you record a metric with an instrument created from a Meter. Choosing the type means choosing how the value is aggregated in the backend:

Two distinctions matter. First, whether what you record is a change or a state: Counter and UpDownCounter record changes (+1, -1), Gauge records state (the current value is 2). Second, who produces the value: with synchronous instruments your code calls Add or Record when the event happens; with ObservableCounter, ObservableUpDownCounter and ObservableGauge the SDK calls your callback when it is time to collect. For durations you almost always use a histogram. Writing an average duration into a gauge is tempting, but as we saw in the previous post the average hides the tail and cannot be turned into a percentile later.

Ordering Metrics with IMeterFactory

Instead of creating the meter statically with new Meter(...), we take it from IMeterFactory in DI. The factory ties the meter to the DI container: the meter is closed when the container is disposed, and in tests the application's own meter can be listened to in isolation (we use that below).

public sealed class OrderingMetrics
{
    public const string MeterName = "Bookstore.Ordering";

    private readonly Counter<long> ordersPlaced;
    private readonly Histogram<double> checkoutDuration;
    private readonly UpDownCounter<long> activeCheckouts;

    public OrderingMetrics(IMeterFactory meterFactory)
    {
        Meter meter = meterFactory.Create(MeterName);
        ordersPlaced = meter.CreateCounter<long>(
            "bookstore.orders.placed", unit: "{order}", description: "Orders that were placed and paid.");
        checkoutDuration = meter.CreateHistogram(
            "bookstore.checkout.duration", unit: "s", description: "Duration of POST /api/orders, payment included.",
            advice: new InstrumentAdvice<double> { HistogramBucketBoundaries = [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 7.5, 10] });
        activeCheckouts = meter.CreateUpDownCounter<long>(
            "bookstore.checkouts.active", unit: "{checkout}", description: "Checkouts in progress.");
    }

    public void CheckoutStarted() => activeCheckouts.Add(1);

    // outcome: placed | out_of_stock | payment_failed | cart_empty | cart_not_found | cart_changed
    public void CheckoutFinished(string outcome, string orderType, TimeSpan elapsed)
    {
        activeCheckouts.Add(-1);
        KeyValuePair<string, object?>[] tags =
        [
            new("bookstore.checkout.outcome", outcome),
            new("bookstore.order.type", orderType)
        ];
        checkoutDuration.Record(elapsed.TotalSeconds, tags);
        if (outcome == "placed")
        {
            ordersPlaced.Add(1, new KeyValuePair<string, object?>("bookstore.order.type", orderType));
        }
    }
}

The naming follows OpenTelemetry's rules: lower case, dot-separated namespaces, your own prefix for your own metrics (bookstore.). The unit is not in the name but in a separate field, in UCUM format: s for durations, a curly-brace annotation like {order} for counted things. We record durations in seconds as double rather than milliseconds; .NET's built-in metrics and the semantic conventions do the same. On the Prometheus side this name becomes bookstore_checkout_duration_seconds; we will see the translation in the sixth post.

The advice parameter (added in .NET 9) is a suggestion for the histogram's bucket boundaries. The OpenTelemetry SDK's default boundaries (0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500…) were chosen with milliseconds in mind; for a duration in seconds almost every value would fall into the first bucket. We picked boundaries for checkout's real distribution: normal orders between 0.05 and 0.25, the retry steps around 2.5, 5 and 7.5. Advice is only a suggestion; whoever sets up the application can override it with a view in the SDK.

The tags are deliberately few and of low cardinality. bookstore.checkout.outcome takes one of six values, bookstore.order.type one of two (a cart with 10 or more lines counts as an institutional order). The order number, the customer and the book's ISBN are not tags. One line in the setup is enough to register the meter with OpenTelemetry; the wildcard also covers meters added later, like Bookstore.Inventory. We added Polly's meter on the same line, because that is where the retry metrics come from:

.WithMetrics(metrics => metrics
    // Bookstore.Ordering, Bookstore.Inventory… and Polly's resilience metrics (retries, timeouts)
    .AddMeter("Bookstore.*", "Polly")
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddRuntimeInstrumentation())

During traffic, dotnet-counters collect showed this in a 10-second window (Polly's tags shortened, some percentile values removed):

bookstore.orders.placed ({order} / 10 sec)[bookstore.order.type=retail] = 3
bookstore.orders.placed ({order} / 10 sec)[bookstore.order.type=institutional] = 1
bookstore.checkout.duration (s)[bookstore.checkout.outcome=placed;bookstore.order.type=retail;Percentile=95] = 5.078125
bookstore.checkout.duration (s)[bookstore.checkout.outcome=placed;bookstore.order.type=institutional;Percentile=95] = 2.63671875
bookstore.checkouts.active ({checkout}) = -3
bookstore.inventory.low_stock_books ({book}) = 2
resilience.polly.strategy.events (Count / 10 sec)[event.name=OnRetry;...;strategy.name=Retry] = 4
resilience.polly.strategy.events (Count / 10 sec)[event.name=OnTimeout;...;strategy.name=Timeout] = 4
resilience.polly.strategy.attempt.duration (ms)[attempt.handled=True;attempt.number=0;...;Percentile=95] = 2004

The answer to Guess First is in this output. An UpDownCounter records changes, not state; its value is the sum of the +1s and -1s since the listener started listening. When dotnet-counters attached, four checkouts were already in progress; it never saw their +1s, only the -1s when they finished. There is no bug in the code. The OpenTelemetry SDK starts with the application, so the value it sees is correct; but every listener that attaches late will see this shift. The Polly lines are valuable too: 4 retries and 4 timeouts in 10 seconds. The information we tried to extract from the logs by hand in the previous post is a ready-made number here.

Gauges and Observable Callbacks

The answer to "how many books have fewer than 5 copies in stock?" is a database query. The first thing that comes to mind is an ObservableGauge: the SDK calls the callback at every collection and the callback runs the query. The problem is that the callback is synchronous, runs on the SDK's collection thread and on every export; if Prometheus scrapes every 15 seconds, every time, and for each reader if there are several. A callback that goes to the database is the shortest path to telemetry slowing down the application. The Gauge<T> added in .NET 9 fits better here: a background service measures the value on its own schedule and sets it.

public sealed class InventoryMetrics(IMeterFactory meterFactory, IServiceScopeFactory scopes, ILogger<InventoryMetrics> logger)
    : BackgroundService
{
    public const string MeterName = "Bookstore.Inventory";
    private const int LowStockThreshold = 5;

    private readonly Gauge<int> lowStockBooks = meterFactory.Create(MeterName).CreateGauge<int>(
        "bookstore.inventory.low_stock_books", unit: "{book}", description: $"Books with fewer than {LowStockThreshold} available copies.");

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using PeriodicTimer timer = new(TimeSpan.FromSeconds(30));
        do
        {
            try
            {
                await using AsyncServiceScope scope = scopes.CreateAsyncScope();
                BookstoreContext db = scope.ServiceProvider.GetRequiredService<BookstoreContext>();
                int count = await db.StockItems.CountAsync(s => s.QuantityOnHand - s.QuantityReserved < LowStockThreshold, stoppingToken);
                lowStockBooks.Record(count);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                logger.LogWarning(ex, "Could not measure low stock");
            }
        }
        while (await timer.WaitForNextTickAsync(stoppingToken));
    }
}

The low_stock_books = 2 in the output above comes from this service: the two books seeded with 0 and 3 copies. Observable instruments are the right choice for values already held in memory: the number of items in a Channel, the size of a connection pool, values that take nanoseconds to read.

Built-in Metrics: What You Get Without Code

Let's briefly see what the meters the dashboard listed in the previous post tell you. http.server.request.duration (ASP.NET Core) is the request duration as the server sees it, tagged with route, method and status code. Almost the whole RED dashboard comes out of this one histogram. http.server.active_requests is the number of requests being handled right now. Kestrel's kestrel.active_connections, kestrel.queued_connections and kestrel.connection.duration show the connection layer, and http.client.request.duration from System.Net.Http shows outgoing calls; the payment provider's duration is read from there. The System.Runtime meter (.NET 9+) describes process health: GC count and duration, heap size, thread pool queue and thread count, exception count, lock contention. .NET 10 added aspnetcore.memory_pool.* metrics to ASP.NET Core, as well as metrics for authentication, authorization and Identity; Bookstore.Api has no authentication yet, so those meters are empty. The security metrics themselves are the subject of the ASP.NET Core Security series.

The value of built-in metrics is that their names match the semantic conventions. A ready-made ASP.NET Core dashboard in Grafana and the dashboard of a service written in another language both look for http.server.request.duration.

RED and USE: Deciding What to Measure

Instead of adding metrics at random, starting with one of two well-known methods makes things easier. The RED method (Rate, Errors, Duration) is the approach Tom Wilkie proposed for services that handle requests: for every endpoint, requests per second, how many of them end in an error, and the duration distribution. In Bookstore all three come from http.server.request.duration and, on the business side, from the outcome tag of bookstore.checkout.duration. The USE method (Utilization, Saturation, Errors) is Brendan Gregg's approach for resources: for each resource such as CPU, memory, the thread pool or a connection pool, how much it is used, how much it makes work wait, and whether it errors. In .NET the thread pool queue (dotnet.thread_pool.queue.length) and Kestrel's queued connections are typical signs of saturation.

They complement each other: RED says what the user experiences, USE says whether the cause is an exhausted resource. In our case RED would alert (checkout's p95 is 5 seconds) while USE would stay calm, because the problem is not in a resource but in the provider outside. That is why business metrics and metrics of external calls (like Polly's retry count) belong next to RED.

Testing a Metric: MetricCollector

Metrics are code too; if a tag's name changes during a refactoring, dashboards and alerts break silently. MetricCollector<T> from the Microsoft.Extensions.Diagnostics.Testing package collects an instrument's measurements so you can test them. Bookstore's integration tests start the real API with WebApplicationFactory; we bound the collector to the application's own IMeterFactory:

[Fact]
public async Task A_placed_order_is_counted_and_its_checkout_duration_is_recorded()
{
    // The collector listens to the meter created by the app's own IMeterFactory.
    IMeterFactory meterFactory = factory.Services.GetRequiredService<IMeterFactory>();
    using MetricCollector<long> placed = new(meterFactory, OrderingMetrics.MeterName, "bookstore.orders.placed");
    using MetricCollector<double> duration = new(meterFactory, OrderingMetrics.MeterName, "bookstore.checkout.duration");

    HttpResponseMessage cart = await client.PostAsync("/api/carts", null, ct);
    Guid cartId = (await cart.Content.ReadFromJsonAsync<CartView>(ct))!.Id;
    await client.PutAsJsonAsync($"/api/carts/{cartId}/items/{Emma}", new SetQuantityRequest(1), ct);
    HttpResponseMessage response = await client.PostAsJsonAsync("/api/orders",
        new PlaceOrderRequest(cartId, "Ada Lovelace", "ada@example.com"), ct);
    Assert.Equal(HttpStatusCode.Created, response.StatusCode);

    CollectedMeasurement<long> order = Assert.Single(placed.GetMeasurementSnapshot());
    Assert.Equal(1, order.Value);
    Assert.Equal("retail", order.Tags["bookstore.order.type"]);

    CollectedMeasurement<double> checkout = Assert.Single(duration.GetMeasurementSnapshot());
    Assert.Equal("placed", checkout.Tags["bookstore.checkout.outcome"]);
    Assert.InRange(checkout.Value, 0, 5);
}

On the first run the test was red:

failed Bookstore.Api.Tests.OrderingMetricsTests.A_placed_order_is_counted_and_its_checkout_duration_is_recorded (389ms)
  Assert.Single() Failure: The collection contained 2 items
  Collection: [CollectedMeasurement`1 { Tags = [["bookstore.order.type"] = "retail"], ..., Value = 1 }, CollectedMeasurement`1 { Tags = [["bookstore.order.type"] = "retail"], ..., Value = 1 }]

This test class shares the same API host with the other test classes, and xUnit runs classes in parallel. The collector also counted the order placed by OrderingTests at the same time. Binding the collector to the application's IMeterFactory keeps out meters of other hosts, but not measurements of other requests within the same host. We moved the telemetry tests into a collection that doesn't run in parallel:

// The API host is shared by all tests, so its meters see every order placed in parallel tests.
// Telemetry tests run alone; otherwise a measurement of another test lands in the snapshot.
[CollectionDefinition(DisableParallelization = true)]
public sealed class MetricsCollection;

[Collection(typeof(MetricsCollection))]
public sealed class OrderingMetricsTests(BookstoreApiFactory factory)

After this change the test stayed green across three consecutive dotnet test runs.

A Cardinality Budget

In the first post we said the cost of metrics grows with the number of time series. The OpenTelemetry .NET SDK protects against this with a limit: one metric keeps at most 2000 distinct tag combinations (according to the package's XML documentation, the point without tags and the overflow point are not counted). To see what happens we wrote a learning test and did the knowingly wrong thing, making the order number a tag:

[Fact]
public void An_order_id_tag_hits_the_cardinality_limit()
{
    List<Metric> exported = [];
    using Meter meter = new("Bookstore.CardinalityDemo");
    using MeterProvider provider = Sdk.CreateMeterProviderBuilder()
        .AddMeter(meter.Name)
        .AddInMemoryExporter(exported)
        .Build();

    Counter<long> placed = meter.CreateCounter<long>("bookstore.orders.placed");
    for (int orderId = 1; orderId <= 2500; orderId++)
    {
        // Wrong on purpose: every order becomes its own time series.
        placed.Add(1, new KeyValuePair<string, object?>("bookstore.order.id", orderId));
    }
    provider.ForceFlush();

    List<MetricPoint> points = [];
    foreach (MetricPoint point in exported.Single().GetMetricPoints())
    {
        points.Add(point);
    }
    MetricPoint overflow = points.Single(p => HasTag(p, "otel.metric.overflow"));

    // 2000 order ids got their own series; the remaining 500 were merged into the overflow series.
    Assert.Equal(2001, points.Count);
    Assert.Equal(500, overflow.GetSumLong());
}

The test passes with these two asserts. 2000 orders got their own series; the 500 measurements from the 2001st order on were merged into one series tagged otel.metric.overflow=true. No data was lost and the total is correct, but after the 2001st order you no longer know which order contributed what, and the SDK writes no error or warning log about it. The limit can be changed per instrument with a view (MetricStreamConfiguration.CardinalityLimit), but the fix is not a bigger limit; it is choosing the right tags. The limit protects memory in the SDK; the cost in the backend grows with every stored series, long before the limit.

A practical budgeting method: multiply the number of values each tag can take. bookstore.checkout.duration has 6 outcomes, 2 order types and 10 bucket boundaries, which means at most 120 bucket series per application instance. With 20 pods on Kubernetes, each producing its own series, that is 2,400. Doing this multiplication before adding a tag makes the decision concrete.

In Production

Choose histogram buckets for the real distribution. The default boundaries are meant for milliseconds; if you record seconds and don't give boundaries through advice or a view, every value lands in one bucket and the percentiles are meaningless.

Pick tag values from a bounded set. An exception message, user input, a full URL or an id will inevitably hit the cardinality limit. Use the exception type (error.type) instead of the message, and the route template instead of the URL.

Don't do I/O in observable callbacks. The callback runs at every collection and for every reader; if you need a database or HTTP call, measure the value in a background service and write it to a Gauge<T>.

Keep meter names consistent with the setup. The AddMeter("Bookstore.*") wildcard covers the meters of new modules automatically; someone who picks a different prefix like Ordering.Metrics ends up without metrics and without any error.

Common Pitfalls

Putting userId or orderId in a tag. The test above shows the result: everything after 2000 disappears into the overflow series, while the backend bill grows long before that. This information belongs in traces and logs.

Reporting an average duration with a gauge. A gauge keeps the last value; neither the average of several pods' averages nor a percentile over a time range can be computed from it.

Publishing the same metric in two different units. A checkout.duration recorded in milliseconds in one place and seconds in another means series that get mixed up in the backend; give the unit in the unit field, not in the name, and stick to one unit.

Treating an UpDownCounter's value as absolute. A listener that attaches late can see a negative value, as in Guess First. If you need an exact "current" value and the source has it at hand, an ObservableUpDownCounter is more accurate.

Running metric tests in the same host as tests that run in parallel. Measurements get mixed and the test turns red now and then; move telemetry tests into their own collection.

A Small Experiment

★☆☆ Remove the advice parameter from bookstore.checkout.duration and look at the p95 in the dashboard (or in dotnet-counters). Why is the value meaningless with the default buckets?

★★☆ Add a Histogram<int> that records the number of lines in a cart (bookstore.cart.lines, unit {line}) and write a MetricCollector test for it. Choose bucket boundaries that separate institutional orders.

★★★ Publish an in-memory value with an ObservableGauge, such as "messages waiting in OrderConfirmationQueue right now" (we add this queue in the next post). Count how many times the callback is called and measure the relationship between the OTLP export interval and the number of calls.

Metrics answer "how often, how bad". "Where did this slow order spend its time" is the job of a trace. The next post moves to distributed tracing: spans, parent/child relationships, the W3C traceparent header, adding our own spans to checkout with ActivitySource, and carrying the trace context over a queue to a background worker.