dotnet-counters and Built-in Metrics: The System.Runtime Meter, ASP.NET Core/Kestrel Metrics and Your Own Meter

Version note: The code in this post was built and run with .NET 10 (runtime 10.0.15 in the container), dotnet-counters 10.0.750501, with k6 2.4.0 generating load. The .NET 8 comparison used runtime 8.0.25.

Guess First

Bookstore.Api has been under load for a while: 50 virtual users place orders and search for books in turn, and the service answers around 950 requests per second. A few lines from the dotnet-counters screen:

[System.Runtime]
    dotnet.gc.heap.total_allocated (By)                             3.6738e+10
    dotnet.jit.compiled_methods ({method})                           1,123,822
    dotnet.thread_pool.queue.length ({work_item})                            0
    dotnet.thread_pool.thread.count ({thread})                               9

Three suspicions: is 36.7 GB of allocation a memory leak? Are 9 threads enough for 50 concurrent users, or is the thread pool stuck? Are 1.1 million compiled methods normal? Only one of these three lines points to a real problem. We will see which one, and why the other two mislead, in the middle of the post with the numbers from a baseline recording.

dotnet-counters monitor and the System.Runtime Meter

dotnet-counters does two jobs: monitor refreshes the values on screen every second, and collect writes the same values to a CSV or JSON file. If you pass no --counters, it watches System.Runtime. With .NET 9 the runtime metrics moved to the System.Diagnostics.Metrics API and are published as a Meter called System.Runtime; the names you see in a .NET 10 app are the dotnet.* names that follow the OpenTelemetry semantic conventions:

docker exec -it profseries-api dotnet-counters monitor -p 1 --counters System.Runtime

[System.Runtime]
    dotnet.assembly.count ({assembly})                                      92
    dotnet.gc.collections ({collection})
        gc.heap.generation
        ------------------
        gen0                                                                54
        gen1                                                               810
        gen2                                                                 5
    dotnet.gc.heap.total_allocated (By)                             3.6738e+10
    dotnet.gc.last_collection.heap.size (By)
        gc.heap.generation
        ------------------
        gen0                                                                 0
        gen1                                                         5,726,688
        gen2                                                        82,359,008
        loh                                                            258,440
        poh                                                          1,368,696
    dotnet.gc.last_collection.memory.committed_size (By)            1.5441e+08
    dotnet.gc.pause.time (s)                                             4.103
    dotnet.jit.compilation.time (s)                                    426.108
    dotnet.jit.compiled_il.size (By)                                1.6203e+08
    dotnet.jit.compiled_methods ({method})                           1,123,822
    dotnet.monitor.lock_contentions ({contention})                      38,985
    dotnet.process.cpu.count ({cpu})                                         4
    dotnet.process.cpu.time (s)
        cpu.mode
        --------
        system                                                         213.419
        user                                                           751.756
    dotnet.process.memory.working_set (By)                          3.2123e+08
    dotnet.thread_pool.queue.length ({work_item})                            0
    dotnet.thread_pool.thread.count ({thread})                               9
    dotnet.thread_pool.work_item.count ({work_item})                14,837,913
    dotnet.timer.count ({timer})                                            46

To shorten the screen you can pick only the metrics you want in square brackets: --counters 'System.Runtime[dotnet.gc.collections,dotnet.gc.pause.time]'. Multiple meters are separated by commas; the commas inside the brackets belong to the metric list.

Cumulative or Current?

The first thing to do when reading this screen is to know each row's type. Most metrics in System.Runtime are a Counter: a running total since the process started. The 36.7 GB in dotnet.gc.heap.total_allocated is the sum of all memory the process allocated over its life; it does not mean the process uses that much memory, the size of objects alive at the same time is around the 82 MB in gen2. dotnet.gc.pause.time is a total too: 4.1 seconds, the sum of all the times the GC paused threads over the process lifetime. The others are an UpDownCounter: the current value. Thread count, queue length, working set and heap size at the last GC belong to this group.

A single read of a cumulative number tells you almost nothing; you need its change. collect does that for you: it writes cumulative metrics to the file as the increase in each interval (Rate).

.NET 8 and Earlier: Falling Back to EventCounter Names

The System.Runtime Meter arrived with .NET 9. When you attach to a .NET 8 or older app, dotnet-counters shows the EventCounters of the old System.Runtime EventSource under the same name. I attached the same dotnet-counters 10.0.750501 to a small console app running on .NET 8.0.25 and took a CSV:

Counter Name,Counter Type,Mean/Increment
% Time in GC since last GC (%),Metric,0
Allocation Rate (B / 1 sec),Rate,829328
CPU Usage (%),Metric,5.515957612596816E-05
Gen 0 GC Count (Count / 1 sec),Rate,0
Monitor Lock Contention Count (Count / 1 sec),Rate,1
Number of Methods Jitted,Metric,43
ThreadPool Queue Length,Metric,0
ThreadPool Thread Count,Metric,0
Time paused by GC (ms / 1 sec),Rate,0
Working Set (MB),Metric,44.302336

The names, the units and even the meanings differ. The old "CPU Usage" is a percentage, the new dotnet.process.cpu.time is seconds; the old "% Time in GC since last GC" describes only the last GC, the new dotnet.gc.pause.time is total pause time. If your dashboard or runbook uses the old names, those rows go blank the day you move to .NET 9. Which name maps to which is covered in the last post of the series, among the version changes.

ASP.NET Core and Kestrel Metrics

Since .NET 8, ASP.NET Core publishes Meter-based metrics too. The most useful one is http.server.request.duration in the Microsoft.AspNetCore.Hosting meter: it records each request's duration into a histogram with method, status code and route tags. For histograms dotnet-counters shows the 50th, 95th and 99th percentiles:

docker exec -it profseries-api dotnet-counters monitor -p 1 \
    --counters 'Microsoft.AspNetCore.Hosting[http.server.request.duration],Bookstore.Ordering'

[Microsoft.AspNetCore.Hosting]
    http.server.request.duration (s)
        http.request.method http.response.status_code http.route        network.protocol.version url.scheme Percentile
        ------------------- ------------------------- ----------------- ------------------------ ---------- ----------
        GET                 200                       /api/books/search 1.1                      http       50         0.001
        GET                 200                       /api/books/search 1.1                      http       95         0.002
        GET                 200                       /api/books/search 1.1                      http       99         0.002
        POST                201                       /api/orders       1.1                      http       50         0.099
        POST                201                       /api/orders       1.1                      http       95         0.207
        POST                201                       /api/orders       1.1                      http       99         0.217

The route tag comes as a template such as /api/books/search, not as the real URL, so you do not get a separate series per query string. Kestrel's own meter, Microsoft.AspNetCore.Server.Kestrel, covers the connection side: kestrel.active_connections, kestrel.queued_connections and kestrel.connection.duration, the per-connection duration. This meter only records measurements when connections open and close; with a load tool that uses keep-alive, a monitor screen opened after the connections are established may never show it. I saw it by starting collect first and then running a curl loop that opens a new connection per request:

Microsoft.AspNetCore.Server.Kestrel,kestrel.queued_connections ({connection})[network.transport=tcp;network.type=ipv6;server.address=::ffff:172.17.0.3;server.port=8080],Metric,0
Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv6;server.address=::ffff:172.17.0.3;server.port=8080;Percentile=95],Metric,0.00321197509765625

kestrel.queued_connections moving away from zero shows that Kestrel cannot accept new connections fast enough. If you use rate limiting, Microsoft.AspNetCore.RateLimiting covers rejected and queued requests, and if you call out with HttpClient, System.Net.Http covers client-side duration and the connection pool.

Your Own Meter: Bookstore.Ordering

Runtime and framework metrics answer "how is the service doing"; you have to add "how is the business doing". I added two instruments to Bookstore's order flow: the number of orders placed (a Counter) and the payment provider's response time (a Histogram):

using System.Diagnostics.Metrics;

namespace Bookstore.Api.Ordering;

public sealed class OrderingMetrics
{
    public const string MeterName = "Bookstore.Ordering";

    private readonly Counter<long> _ordersPlaced;
    private readonly Histogram<double> _paymentDuration;

    public OrderingMetrics(IMeterFactory meterFactory)
    {
        Meter meter = meterFactory.Create(MeterName);

        _ordersPlaced = meter.CreateCounter<long>(
            "bookstore.orders.placed", unit: "{order}", description: "Number of orders placed");

        _paymentDuration = meter.CreateHistogram(
            "bookstore.payment.duration", unit: "s", description: "Duration of payment authorization",
            advice: new InstrumentAdvice<double> { HistogramBucketBoundaries = [0.025, 0.05, 0.1, 0.25, 0.5, 1] });
    }

    public void OrderPlaced(string paymentMethod) =>
        _ordersPlaced.Add(1, new KeyValuePair<string, object?>("bookstore.payment.method", paymentMethod));

    public void PaymentCompleted(TimeSpan elapsed, string paymentMethod) =>
        _paymentDuration.Record(elapsed.TotalSeconds, new KeyValuePair<string, object?>("bookstore.payment.method", paymentMethod));
}

using System.Diagnostics;

namespace Bookstore.Api.Ordering;

public sealed record PlaceOrderRequest(int CustomerId, int BookId, int Quantity, string PaymentMethod);

public static class OrderEndpoints
{
    public static void MapOrderEndpoints(this IEndpointRouteBuilder app)
    {
        app.MapPost("/api/orders", async (PlaceOrderRequest request, OrderingMetrics metrics, CancellationToken ct) =>
        {
            long started = Stopwatch.GetTimestamp();
            // simulated payment provider: card is slower than wallet
            int delayMs = request.PaymentMethod == "card" ? Random.Shared.Next(60, 220) : Random.Shared.Next(15, 60);
            await Task.Delay(delayMs, ct);
            metrics.PaymentCompleted(Stopwatch.GetElapsedTime(started), request.PaymentMethod);

            metrics.OrderPlaced(request.PaymentMethod);
            return Results.Created($"/api/orders/{Random.Shared.Next(1, 1_000_000)}", new { request.BookId, request.Quantity });
        });
    }
}

OrderingMetrics is registered in Program.cs with AddSingleton<OrderingMetrics>(). Using IMeterFactory instead of new Meter(...) has two benefits: the meter's lifetime is tied to the DI container (the meter closes when the container is disposed), and in tests each test can get a separate meter from its own factory, so tests do not see each other's measurements. For naming I used the lowercase, dotted style OpenTelemetry recommends, and UCUM notation such as s, By and {order} for units. HistogramBucketBoundaries (added in .NET 9) is a hint for exporters such as OpenTelemetry that export the histogram as buckets; dotnet-counters computes the percentiles itself:

[Bookstore.Ordering]
    bookstore.orders.placed ({order})
        bookstore.payment.method
        ------------------------
        card                                                             1,779
        wallet                                                           1,117
    bookstore.payment.duration (s)
        bookstore.payment.method Percentile
        ------------------------ ----------
        card                     50                                       0.14
        card                     95                                      0.212
        card                     99                                      0.218
        wallet                   50                                      0.039
        wallet                   95                                      0.058
        wallet                   99                                       0.06

The code waits 60-220 ms for card and 15-60 ms for wallet; the percentiles land exactly in those ranges. The p95 of POST /api/orders is 207 ms; that it is almost the same as the 212 ms card p95 tells you nearly all of the endpoint's time is the payment wait. Moving these metrics to a time-series database and a dashboard (an OpenTelemetry exporter, Prometheus, Grafana) is the topic of the Observability series in the DevOps category; here the goal is to set up the measurement itself and verify it.

Recording a Baseline: CSV with collect

A baseline recorded on a day without problems is the only thing that answers "is this value normal?" during an incident. I applied the same load (50 virtual users, 65 seconds) and took a 60-second recording:

docker exec profseries-api dotnet-counters collect -p 1 --refresh-interval 5 \
    --duration 00:00:01:00 --format csv -o /tmp/baseline.csv \
    --counters 'System.Runtime,Microsoft.AspNetCore.Hosting,Bookstore.Ordering'
docker cp profseries-api:/tmp/baseline.csv .

Timestamp,Provider,Counter Name,Counter Type,Mean/Increment
11/13/2026 23:58:30,System.Runtime,dotnet.jit.compiled_methods ({method} / 5 sec),Rate,4980
11/13/2026 23:58:30,System.Runtime,dotnet.gc.pause.time (s / 5 sec),Rate,0.020392999999999883
11/13/2026 23:58:30,Bookstore.Ordering,bookstore.orders.placed ({order} / 5 sec)[bookstore.payment.method=card],Rate,1489
11/13/2026 23:58:30,System.Runtime,dotnet.gc.heap.total_allocated (By / 5 sec),Rate,175837272

Cumulative metrics are written as Rate with the 5-second increase, current ones as Metric. The table from the median of the 5-second intervals:

Reading Healthy Values

The answer to the guessing question is in this table. 36.7 GB of allocation is not a problem: the GC cleans up 34 MB per second with 36 gen1 collections per minute, spending 0.44 percent of the time, and never needs a gen2; the gen2 heap stays at 82 MB. Nine threads are not a problem either: the queue is at zero, so work is picked up without waiting. The order endpoint spends nearly all its time in await Task.Delay and holds no thread while waiting; 9 threads being enough for 50 concurrent requests is what async code is supposed to look like. The picture where the thread count climbs slowly and steadily while the queue fills was the starvation case in the "Diagnosing Hangs and Slowness" post of the Debugging series.

The real problem is on the JIT row. In a warmed-up service the number of newly compiled methods should be close to zero; here 970 methods are compiled per second and the JIT takes 0.39 of the 1.17 CPU-seconds spent across 4 CPUs, a third of the CPU time. The 58 lock contentions per second probably come from the same place. Metrics do not answer "where", but they tell you where to look: in the next post we will find, with dotnet-trace, which code compiles something new on every request.

In general these are the thresholds I look at, but they are starting points for comparing against your own baseline, not rules: if gen2s exceed a few per minute or the LOH grows, get an allocation profile (04). If GC pauses exceed 5 percent of the time, request durations will feel it. If thread_pool.queue.length stays above zero for seconds, look at the thread pool. If lock contention reaches several times the baseline, look for a shared lock. If the working set grows while the GC heap stays flat, the problem is not in the managed heap (05).

In Production

dotnet-counters itself is cheap, tags are not. Every distinct tag combination is a separate time series; a tag with a handful of values such as bookstore.payment.method is harmless, an unbounded tag such as a customer ID or order number piles up thousands of series inside the process. dotnet-counters bounds this with --maxTimeSeries (default 1000) and --maxHistograms (default 10). The default of 10 histograms fills up quickly in an API with a few routes; once the limit is hit, some histograms never show up on screen and the error message is easy to miss. Raising these limits means more memory in the target process.

You cannot open two metric sessions on the same process at the same time. When I tried to open monitor while collect was running, the tool said "Another metrics collection session is already in progress for the target process." and stopped. In production, the right way to collect continuously is not leaving dotnet-counters on but exporting metrics with OpenTelemetry; dotnet-counters is for a quick look during an incident and for recording baselines. Think about the interval too: --refresh-interval 1 catches short events but makes the file five times bigger.

Common Pitfalls

• Taking a cumulative value for a rate: calling dotnet.gc.heap.total_allocated or dotnet.gc.pause.time "too high" from a single read.
• Trusting current-value metrics in the CSV: in this version dotnet.thread_pool.thread.count and dotnet.thread_pool.queue.length were written with the Rate type, as the 5-second change; for the absolute value use monitor or sum the changes.
• Moving .NET 8 dashboard queries to a .NET 9+ app: the old EventCounter names are no longer published.
• Bitness mismatch: watching a 32-bit app on Windows with the 64-bit dotnet-counters fails with "A 32 bit processes cannot access modules of a 64 bit process"; download the x86 tool.
• Creating the meter with new Meter(...) on every request: that means a new meter and new instruments every time; it should be a single instance.
• Looking for Kestrel or HttpClient metrics on a screen opened after keep-alive connections were established.

A Small Experiment

★☆☆ Open dotnet-counters monitor --counters System.Runtime on your own service and mark each row as cumulative or current. Take two reads 30 seconds apart, compute the allocation rate by hand, then compare it with the Rate row from collect.

★★☆ Add a counter named bookstore.orders.failed with an error.type tag to OrderingMetrics; mark orders whose payment wait exceeds 200 ms as "timeout" and see them split by tag in dotnet-counters.

★★★ Take a 10-minute baseline for your own service: collect --format csv under typical load, then a small script that extracts the median and worst-interval values as in the table. Put this table in your runbook; it is the first thing you will compare against in the next incident.

The metrics told us the JIT eats a third of the CPU, but not which code is responsible. In the next post we will take a CPU profile with dotnet-trace, read the flame graph, find and fix the bug in the search endpoint and compare before and after.