Prometheus and PromQL: Scrape or Push, rate, histogram_quantile, Recording Rules and Native Histograms

Version note: The examples in this post were run with .NET 10, OpenTelemetry .NET 1.21, OpenTelemetry.Exporter.Prometheus.AspNetCore 1.21.0-beta.1 and Prometheus 3.16.0 (the official prom/prometheus image).

Guess First

We sent traffic to Bookstore, the online bookstore used as the shared example across this site's series, for 111 seconds; exactly 398 orders were placed. We asked Prometheus "how many orders were placed in the last 3 minutes": sum(increase(bookstore_orders_placed_total[3m])). The answer was 410.1. In the same run the traffic tool measured checkout's p95 at 5.09 seconds; Prometheus's histogram_quantile said 5.93 seconds. Can't Prometheus count? Both answers lie in how PromQL does its math; they are in the middle of the post.

Pull or Push?

Prometheus's classic model is pull: it knows its targets and fetches metrics from their HTTP endpoint every scrape_interval. The application only serves the current values on a page; Prometheus builds the time series. The model has a practical benefit: if a scrape fails, Prometheus knows it by itself and keeps an up series for every target, so the difference between "the service stopped sending telemetry" and "there is simply no traffic" comes for free. On Kubernetes targets aren't written by hand; pods and services are discovered from the API server with kubernetes_sd_configs, or, with the Prometheus Operator, defined with a ServiceMonitor or PodMonitor object. We did not run that part in this post; the Kubernetes series and the production post of this series are the place for it.

OpenTelemetry's model is push: the SDK periodically sends data somewhere over OTLP. Prometheus 3 brought the two together: with the --web.enable-otlp-receiver flag it accepts OTLP metrics directly. A Collector sending to Prometheus with remote write or OTLP fits the same model. So there are two ways from .NET into Prometheus.

Two Ways from .NET into Prometheus

The first is scraping: the application exposes a /metrics endpoint. OpenTelemetry has a package for it, but mind its version:

<!-- Still prerelease: depends on the experimental Prometheus/OpenMetrics compatibility spec. -->
<PackageVersion Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.21.0-beta.1" />

The package has been in beta for years and its README states the reason plainly: OpenTelemetry's Prometheus and OpenMetrics compatibility specification is still experimental, and for production it recommends the OTLP exporter. Two lines are enough in the setup:

.WithMetrics(metrics => metrics
    .AddMeter("Bookstore.*", "Polly")
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddRuntimeInstrumentation()
    // GET /metrics for a Prometheus that scrapes (pull); OTLP below still pushes the same metrics.
    .AddPrometheusExporter())

// Prometheus scrapes this; in production keep it on an internal port or behind authorization.
app.MapPrometheusScrapingEndpoint();

The second is push: the application's OTLP exporter sends the metrics straight to Prometheus's OTLP endpoint or to a Collector. The code doesn't change; only the endpoint and the protocol are given for the metrics signal. We started Prometheus with both ways open:

docker run -d --name bookstore-prometheus -p 9090:9090 -v ${PWD}/deploy/observability/prometheus.yaml:/etc/prometheus/prometheus.yml:ro -v ${PWD}/deploy/observability/rules.yaml:/etc/prometheus/rules.yaml:ro prom/prometheus:v3.16.0 --config.file=/etc/prometheus/prometheus.yml --web.enable-otlp-receiver

# Prometheus for Bookstore (local). Two ways in:
#   pull: scrapes GET /metrics of Bookstore.Api every 15 s
#   push: OTLP on /api/v1/otlp (needs --web.enable-otlp-receiver)
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules.yaml

scrape_configs:
  - job_name: bookstore-api
    metrics_path: /metrics
    static_configs:
      - targets: ["host.docker.internal:5080"]

otlp:
  # Resource attributes that become labels on every series (the rest stay in target_info).
  promote_resource_attributes:
    - service.name
    - service.version
    - deployment.environment.name

Prometheus in the container reaches the API on the host through host.docker.internal. For push we started the application with these environment variables; we gave it a separate service name so the series from the two ways don't mix:

$env:OTEL_SERVICE_NAME = "bookstore-api-otlp"
$env:OTEL_EXPORTER_OTLP_METRICS_ENDPOINT = "http://localhost:9090/api/v1/otlp/v1/metrics"
$env:OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "http/protobuf"
$env:OTEL_METRIC_EXPORT_INTERVAL = "15000"

In the first attempt we hadn't changed the service name, and we learned something: on series that arrive over OTLP the job label is derived from service.name and instance from service.instance.id. Since the job name in the scrape config is also bookstore-api, the same metric arrived twice, as two series under the same job differing only in instance. Every query with sum(...) would have shown double. Don't collect the same metric both ways; pick one.

The Exposition Format and Name Translation

The scraped page is Prometheus's text format (Content-Type: text/plain; version=0.0.4). Part of the orders and the checkout duration (some bucket lines removed, labels shortened):

# TYPE bookstore_orders_placed_total counter
# HELP bookstore_orders_placed_total Orders that were placed and paid.
bookstore_orders_placed_total{otel_scope_name="Bookstore.Ordering",bookstore_order_type="retail"} 447
bookstore_orders_placed_total{otel_scope_name="Bookstore.Ordering",bookstore_order_type="institutional"} 50
# TYPE bookstore_checkout_duration_seconds histogram
bookstore_checkout_duration_seconds_bucket{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail",le="0.1"} 180
bookstore_checkout_duration_seconds_bucket{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail",le="0.25"} 336
bookstore_checkout_duration_seconds_bucket{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail",le="2.5"} 336
bookstore_checkout_duration_seconds_bucket{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail",le="5"} 423
bookstore_checkout_duration_seconds_bucket{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail",le="7.5"} 447
bookstore_checkout_duration_seconds_bucket{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail",le="+Inf"} 447
bookstore_checkout_duration_seconds_sum{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail"} 382.5907395000002
bookstore_checkout_duration_seconds_count{...,bookstore_checkout_outcome="placed",bookstore_order_type="retail"} 447

The OpenTelemetry name changed on its way into Prometheus. The dots in bookstore.orders.placed became underscores and, being a counter, it got the _total suffix; the {order} unit is a curly-brace annotation, so it wasn't added to the name. bookstore.checkout.duration has the unit s, so _seconds was appended and the histogram split into three families of series: _bucket (a cumulative counter for every le boundary), _sum and _count. Tags became labels and the meter name moved into the otel_scope_name label. bookstore.checkouts.active was an UpDownCounter; in Prometheus it shows up as a gauge. Looking at the buckets you can also read the shape of the histogram: all 336 orders are under 0.25 seconds; none between 0.25 and 2.5; then 87 orders fall into 2.5–5 and 24 into 5–7.5. The retry steps sit between the bucket boundaries.

Prometheus 3 supports UTF-8 in metric and label names, so names with dots are now possible. For data arriving over OTLP the translation is decided by otlp.translation_strategy in the config; the default in 3.16 is UnderscoreEscapingWithSuffixes, the behavior above. There are strategies that keep dots and units as they are (NoUTF8EscapingWithSuffixes, NoTranslation), but we didn't run them; if your dashboards and alerts are already written against underscored names, treat the move as a project. And target_info: all resource attributes live in this separate series; the promote_resource_attributes list in the config added the ones we chose as labels on every series (the OTLP series show service_name, service_version and deployment_environment_name).

PromQL: rate, increase and sum by

A raw counter is not useful: it is an ever-growing number that resets when the application restarts. The meaningful question is "how many per second", and the answer is rate. We ran every query pinned to the moment the traffic ended. In this first run the same metric came in both by scrape and over OTLP (as explained above), so we selected the scraped copy with the instance label:

sum by (bookstore_order_type) (rate(bookstore_orders_placed_total{job="bookstore-api",instance="host.docker.internal:5080"}[2m]))
{bookstore_order_type="retail"}         3.0804497507352013
{bookstore_order_type="institutional"}  0.33719619695252484

sum(increase(bookstore_orders_placed_total{job="bookstore-api",instance="host.docker.internal:5080"}[3m]))
{}  410.1175137225272

rate(x[2m]) computes the per-second increase from the samples inside the window; if the counter reset, it notices and corrects for it. sum by adds up the labels you don't want to keep, such as pods or status codes, and leaves only the ones you ask for. The first answer to Guess First is in increase: increase is rate multiplied by the window length, and when there are no samples at the edges of the window PromQL extrapolates the increase to the whole window. With a 15-second scrape interval, the first and last samples of a 3-minute window are not exactly at its edges; Prometheus extends the slope to the edges. The result is the non-integer 410.1. increase is an estimate, not an accounting tool; if you need an exact number (billing, reconciliation), use the database, not a metric.

histogram_quantile and the Error Rate

histogram_quantile(0.50, sum by (le) (rate(bookstore_checkout_duration_seconds_bucket{job="bookstore-api",instance="host.docker.internal:5080"}[3m])))
{}  0.16209084939551588
histogram_quantile(0.95, sum by (le) (rate(bookstore_checkout_duration_seconds_bucket{job="bookstore-api",instance="host.docker.internal:5080"}[3m])))
{}  5.933824811197164
histogram_quantile(0.99, sum by (le) (rate(bookstore_checkout_duration_seconds_bucket{job="bookstore-api",instance="host.docker.internal:5080"}[3m])))
{}  8.752432454185117

sum(rate(bookstore_checkout_duration_seconds_sum{job="bookstore-api",instance="host.docker.internal:5080"}[3m]))
  / sum(rate(bookstore_checkout_duration_seconds_count{job="bookstore-api",instance="host.docker.internal:5080"}[3m]))
{}  1.0270184636175672

histogram_quantile computes a percentile from buckets whose boundaries are in the le label. sum by (le) is required: if you don't add up the labels other than the bucket boundary, you get a separate percentile for every outcome and order type. The second answer to Guess First comes from here. Prometheus only knows "there are 87 orders between 2.5 and 5"; it assumes the values are evenly spread inside that range and interpolates linearly. The real values all cluster at 2.6 and 5.1 seconds, so the assumption fails: p95 came out at 5.93 and p99 at 8.75. The traffic tool measured about 5.09 for p95 and 7.64 for p99. The coarser the bucket boundaries, the rougher the percentile. The second query computes the average from _sum and _count: 1.03 seconds, the misleading average from the first post.

The error rate is a ratio of two rates. The payment provider's metrics are read the same way:

sum(rate(http_server_request_duration_seconds_count{job="bookstore-api",instance="host.docker.internal:5080",http_route="/api/orders/",http_response_status_code=~"5.."}[3m]))
  / sum(rate(http_server_request_duration_seconds_count{job="bookstore-api",instance="host.docker.internal:5080",http_route="/api/orders/"}[3m]))
{}  0.005972784178019447

sum by (event_name) (increase(resilience_polly_strategy_events_total{job="bookstore-api",instance="host.docker.internal:5080",event_name=~"OnRetry|OnTimeout"}[3m]))
{event_name="OnTimeout"}  152.0865574790551
{event_name="OnRetry"}    149.7651725593902

Six in a thousand checkout requests ended with 502; in the same period about 150 retries went to the payment provider. The error rate the user sees is low, but the retries put about 38 percent extra load on the provider (150 retries for 400 orders). Reading the two numbers together is the data you need to judge a retry policy.

Recording Rules

Every time they are opened, the queries above read hundreds of bucket series and compute. If a dashboard has 10 panels refreshing every 30 seconds and an alert rule uses the same query too, Prometheus does the same math over and over. A recording rule computes a query's result at regular intervals and stores it as a new series:

groups:
  - name: bookstore-checkout
    rules:
      # Orders per second, by order type
      - record: bookstore:orders_placed:rate5m
        expr: sum by (bookstore_order_type) (rate(bookstore_orders_placed_total{job="bookstore-api"}[5m]))

      # 95th percentile of the checkout duration over 5 minutes
      - record: bookstore:checkout_duration_seconds:p95_5m
        expr: histogram_quantile(0.95, sum by (le) (rate(bookstore_checkout_duration_seconds_bucket{job="bookstore-api"}[5m])))

      # Share of POST /api/orders requests that ended with 5xx
      - record: bookstore:orders_http_errors:ratio_rate5m
        expr: |
          sum(rate(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/", http_response_status_code=~"5.."}[5m]))
            /
          sum(rate(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/"}[5m]))

The naming convention is level:metric:operations: the labels it is aggregated to, the metric it comes from, and what was computed. Rules run every evaluation_interval (15 seconds for us); dashboards and alerts read a single precomputed series:

bookstore:checkout_duration_seconds:p95_5m                          5.120097534607394
bookstore:orders_placed:rate5m{bookstore_order_type="retail"}         1.390115888888889
bookstore:orders_placed:rate5m{bookstore_order_type="institutional"}  0.15389696296296296
bookstore:orders_http_errors:ratio_rate5m                           0.005874259178432363

The job="bookstore-api" filter in the rules is deliberate: we hadn't written it in the first attempt, and the rule was adding up the two copies that came in by scrape and over OTLP. Alert rules work with the same mechanism; we will write them in the alerting post of the series.

Native Histograms

We saw two problems with classic histograms: every bucket boundary is a separate series, and the percentile is only as accurate as the buckets are fine. A native histogram is a data type that keeps the whole distribution in a single series, in a single sample. It became stable in Prometheus 3.8 but is still opt-in: it is enabled with the scrape_native_histograms setting. There are two kinds. In an exponential native histogram nobody picks bucket boundaries; they are logarithmic and adjust to the distribution, with a bounded error. A native histogram with custom buckets (NHCB) keeps the classic buckets but stores them in a single series. .NET's Prometheus exporter only produces classic histograms; Prometheus can convert them to NHCB while scraping. Over OTLP, the OpenTelemetry SDK's exponential histogram arrives directly as a native histogram. We tried both at once. Three lines were added to the config:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  # Native histograms (stable since Prometheus 3.8, still opt-in). The .NET exporter only has classic
  # buckets, so they are stored as one native histogram with custom buckets (NHCB) per series.
  scrape_native_histograms: true
  convert_classic_histograms_to_nhcb: true
  # During the move keep the classic _bucket/_sum/_count series too, so existing queries keep working.
  always_scrape_classic_histograms: true

And a single environment variable on the application's OTLP side: OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=base2_exponential_bucket_histogram. After traffic of 500 orders we compared the three forms of the same metric:

count by (job) ({__name__=~"bookstore_checkout_duration_seconds.*"})
{job="bookstore-api-otlp"}  3
{job="bookstore-api"}       42

# classic buckets
histogram_quantile(0.95, sum by (le) (rate(bookstore_checkout_duration_seconds_bucket{job="bookstore-api"}[3m])))
{}  5.377143879619664
# NHCB: the same buckets, one series
histogram_quantile(0.95, sum(rate(bookstore_checkout_duration_seconds{job="bookstore-api"}[3m])))
{}  5.230765495094986
# exponential native histogram from OTLP
histogram_quantile(0.95, sum(rate(bookstore_checkout_duration_seconds{job="bookstore-api-otlp"}[3m])))
{}  4.987966421682704
histogram_quantile(0.99, sum(rate(bookstore_checkout_duration_seconds{job="bookstore-api-otlp"}[3m])))
{}  5.164380590563866
histogram_fraction(0, 1, sum(rate(bookstore_checkout_duration_seconds{job="bookstore-api-otlp"}[3m])))
{}  0.743496465027412

In this run the traffic tool measured 5.09 seconds for p95 and 5.16 for p99. The exponential histogram came closest with 4.99 and 5.16 seconds; and it needed just one series per outcome and order type combination, 3 in total, while classic and NHCB together kept 42 (both, because always_scrape_classic_histograms is on). The query syntax gets simpler too: no _bucket and no le, just the histogram itself. histogram_fraction is a function that comes with native histograms: 74 percent of checkouts finished in under a second. Keeping the classic series during the move is sensible so old queries don't break; once dashboards and recording rules are on the new syntax, always_scrape_classic_histograms can be turned off.

In Production

Make the rate window at least four times the scrape interval. With our 15-second interval [1m] is borderline, [2m] comfortable. When you change the scrape interval, review the windows in dashboards and rules too.

Label cardinality in Prometheus is memory, directly. Every series is held in memory in the head block; prometheus_tsdb_head_series from Prometheus's own metrics is the quickest way to watch the total series count. Since every bucket of a classic histogram is a series, this number grows fast with histograms.

Don't expose the /metrics endpoint to everyone. It contains routes, error counts and runtime information; the exporter's README also reminds you it is unprotected by default. Move it to a separate port or make it reachable only from the internal network.

Prometheus is a single server with its data on local disk. For long retention, high availability and querying several clusters from one place, remote write to Mimir, Thanos or a managed service is used; we look at managed options in the production post of the series.

Common Pitfalls

Applying rate to a gauge. rate(bookstore_checkouts_active[2m]) returned 0.022 for us; it means nothing, because rate assumes a counter and takes every drop for a reset. For gauges use avg_over_time, max_over_time or deriv.

Choosing a window shorter than the scrape interval. rate(bookstore_orders_placed_total[10s]) returned an empty result for us: a 10-second window doesn't contain two of the samples taken every 15 seconds. An empty chart gets mistaken for "no orders at all".

Leaving out sum by (le) before histogram_quantile, or summing away le. In the first case you get a separate percentile for every label combination, in the second the function can't find the buckets.

Averaging percentiles. The average of five pods' p95s is not the p95 of the five together. Add up the buckets first, then compute the percentile; recording rules should be written in that order too.

Collecting the same metric both by scrape and over OTLP. As we saw above, the series double under the same job and sum silently gives the wrong answer.

A Small Experiment

★☆☆ Add 2.75 and 5.25 to the bucket boundaries of bookstore.checkout.duration (in the advice). Run the traffic again and measure how close the classic histogram's p95 gets to the real value.

★★☆ Read the duration of the calls to the payment provider from http_client_request_duration_seconds, filtered by the server_address label, and write a recording rule to compare it with checkout's p95 on the same panel.

★★★ Set Prometheus's otlp.translation_strategy to NoUTF8EscapingWithSuffixes, look at how the name of a metric arriving over OTLP appears, and try querying it with the UTF-8 syntax, quoting the name inside the curly braces.

We can now ask the right questions in Prometheus, but writing PromQL every time is not a tool a team will use. In the next post we turn these queries into RED and USE dashboards in Grafana: variables, jumping from a spike in the p99 chart to that request's trace with exemplars, and keeping dashboards as code in Git.