Observability in Production: Azure Monitor and Application Insights, Sampling, Cost, PII and Auto-Instrumentation on Kubernetes

Version note: The examples in this post were run with .NET 10, OpenTelemetry .NET 1.21 and the grafana/otel-lgtm 0.41.0 image (Tempo 3.1.1, Loki 3.7.9, Prometheus 3.16.0, OpenTelemetry Collector contrib 0.165.0). The Azure Monitor code written with Microsoft.OpenTelemetry 1.2.0 was compiled but not sent to Azure; the Kubernetes and OpenTelemetry Operator part was not run.

Guess First

The Collector of Bookstore, the online bookstore used as the shared example across this site's series, does tail sampling: it keeps every error and every trace slower than 1 second, and 10 percent of the rest. A teammate who wants to cut trace cost further suggests adding 10 percent head sampling in the application as well: OTEL_TRACES_SAMPLER=parentbased_traceidratio, OTEL_TRACES_SAMPLER_ARG=0.1. We sent the same 300 orders with both settings; both had around 70 checkouts slower than 2 seconds. With head sampling on, how many of these slow checkouts keep their trace in Tempo? And does the checkout p95 in Prometheus change?

The Managed Option: Azure Monitor and Application Insights

The series' example stack is open source and runs in our own container. In production many teams pick a managed service instead of running this; on Azure that's Azure Monitor and its application side, Application Insights. Microsoft has moved this side to OpenTelemetry, and there are three paths today. For new applications Microsoft recommends the Microsoft OpenTelemetry Distro (the Microsoft.OpenTelemetry package). Those who already use the Azure Monitor OpenTelemetry Distro (Azure.Monitor.OpenTelemetry.AspNetCore) don't need to do anything. Application Insights' classic .NET SDK was rewritten on top of OpenTelemetry in its 3.x releases; for those on 2.x, moving to 3.x is the recommended way to carry existing code over with the fewest changes. The new distro can send telemetry to Azure Monitor and to your own Collector at the same time:

using Microsoft.OpenTelemetry;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

// Microsoft OpenTelemetry Distro: Azure Monitor and our own OTLP Collector at the same time.
builder.UseMicrosoftOpenTelemetry(options =>
{
    options.Exporters = ExportTarget.AzureMonitor | ExportTarget.Otlp;
    // Connection string from APPLICATIONINSIGHTS_CONNECTION_STRING (Key Vault / app settings), never from the repo.
    options.AzureMonitor.ConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
    // Fixed 10% instead of the default rate limit (5 traces per second).
    options.AzureMonitor.TracesPerSecond = null;
    options.AzureMonitor.SamplingRatio = 0.1F;
});

WebApplication app = builder.Build();
app.MapGet("/", () => "ok");
app.Run();

We compiled this code in a separate project with Microsoft.OpenTelemetry 1.2.0, but didn't send it to an Azure subscription. Two points from the docs: the distro uses rate-limited sampling by default (about 5 traces per second), so if you want percentage sampling you have to set TracesPerSecond to null; and the new distro and the old Azure Monitor distro must not be initialized together in the same application. We didn't add Azure Monitor to Bookstore in this post; the Collector's contrib distribution also has an Azure Monitor exporter (azuremonitorexporter), so telemetry can be sent from the Collector without changing the application. Which one: if the application team wants Azure's ready-made views (Application Map, Failures, Live Metrics), the distro; if you want to manage telemetry in one place and send it to several backends, the Collector.

Sampling: Head, Tail and Both

The head sampling decision is made at the start of the request, in the SDK: it looks at the trace id, and either no spans are created or all of them are. The parentbased_ prefix makes it follow a decision found in the incoming request's traceparent, so half of a trace isn't kept in one service and dropped in another. Tail sampling, as we saw in the Collector post, decides after the trace has finished, by looking at its outcome. The results of the two runs from Guess First, for the same minutes, from the Collector's own metrics, Tempo and Prometheus:

The answer to Guess First is in the table: with head sampling on, only 4 of the 78 slow checkouts kept their trace. Tail sampling's "keep everything slow" rule can't be applied to traces that never reach it; head sampling drops 90 percent at the very start, without telling slow and fast requests apart. The number of spans reaching the Collector also fell to about 9 percent, as expected. The answer to the second question is no: metrics aren't affected by sampling, because every request is recorded in the histogram; in both runs the order count is around 300 (the difference in the increase is increase's extrapolation). Logs aren't affected either: in both runs the Collector received about 2,100 log records. Exemplars did go down, because the TraceBased filter only accepts measurements inside sampled spans.

The practical rule: if you use tail sampling, don't do head sampling in the application, or leave it at 100 percent. Head sampling's only benefit is less network traffic from the application to the Collector and less memory in the Collector; the price is losing failed and slow traces. If traffic is very high and tail sampling runs out of memory, first consider scaling the Collector (spreading by trace id with the load-balancing exporter).

Cost: How Much Telemetry Do We Produce?

The telemetry bill comes from volume: data taken in (ingestion), retention, and the number of metric series. Every managed service charges by one of these three; in a stack you run yourself, the same three show up as disk, memory and CPU. The numbers per order from Bookstore's first run (no head sampling):

This table is a budgeting tool. At 100,000 orders a day the application produces 3.5 million spans; thanks to tail sampling 700,000 of them are kept. On the log side the filter cuts the volume by more than half. Metrics grow not with traffic but with label combinations: 150 of the 626 series are http_server_request_duration_seconds_bucket (route × method × status code × number of buckets). A new route or a new label increases the series count regardless of the number of orders; that's why the cardinality budget from the post on producing metrics matters. To watch the cost, put the Collector's own metrics (otelcol_receiver_accepted_spans_total, otelcol_exporter_sent_log_records_total) on a dashboard, with an alert that opens a ticket when the daily volume crosses a threshold, so the ingestion bill doesn't jump overnight. How to set log levels in production is in the Logging series, cost alerts on the cloud side in the Cloud Fundamentals series.

PII and KVKK

Telemetry can carry personal data: an e-mail address written to a log, a national ID number in a URL path, a customer name in an exception message, a user id put into baggage. From the point of view of KVKK, Turkey's personal data protection law, the telemetry backend is also a system that processes personal data: who can access it, how long it's kept and whether it leaves the country apply to it too. The legal assessment isn't this post's topic; on the technical side the principle is simple: don't write the data in the first place, mask it in the Collector if it was written, and measure the remaining risk regularly. In Bookstore the Collector hashes the e-mail address; we checked that this really works with three queries in Loki (for the 300 orders of the first run; results as Loki's API returned them):

sum(count_over_time({service_name="bookstore-api"} |~ "@example\\.com" [105s]))
[]
sum(count_over_time({service_name="bookstore-api"} | CustomerEmail=~".*@.*" [105s]))
[]
sum(count_over_time({service_name="bookstore-api"} | CustomerEmail!="" [105s]))
[{"metric":{},"value":[1794689449,"300"]}]

The first two queries came back empty. 300 order confirmation logs carry the CustomerEmail field, but none of them contains an @; they're all hashed. Turning queries like these into an alerting rule (Loki's ruler can produce alerts with LogQL) catches a new log line that bypasses the masking from day one. On the access side, limit who can look at production telemetry with roles, and match retention to the purpose: there's no reason to keep a trace stored for debugging for 90 days.

Semantic Convention Migrations

OpenTelemetry's attribute names changed over time: for HTTP http.method → http.request.method, http.status_code → http.response.status_code; for databases db.statement → db.query.text, and so on. During the migration periods, instrumentations in some languages used the OTEL_SEMCONV_STABILITY_OPT_IN environment variable; the http/dup value wrote the old and new names together, giving time to move dashboards. In .NET this setting is gone: the AspNetCore instrumentation removed it for HTTP in 1.6.0 (2023), the SqlClient instrumentation for databases in 1.14.0 (2026), and they only write the stable names. In practice this means that when you upgrade instrumentation packages you have to check the attribute and metric names your dashboards and alerts use. The SQL spans we saw in the trace backend post come with these new names, db.query.summary and db.system.name; an old dashboard looking for db.statement silently stays empty.

Automatic Instrumentation on Kubernetes

We didn't run this section. On Kubernetes the OpenTelemetry Operator is a way to add instrumentation without changing code: an Instrumentation object defines the exporter, the propagators and the sampler; an annotation on the pod injects .NET's automatic instrumentation (through a startup hook) into the container. The example from the Operator docs, adapted to Bookstore:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: bookstore-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
---
# In the Deployment's pod template:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-dotnet: "true"
    # For Alpine (musl) based images:
    instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"

.NET's automatic instrumentation sends OTLP with http/protobuf by default, so to port 4318. This path is a good start for applications whose code you can't change; for services whose code you write, such as Bookstore, setting up the SDK in code gives more control: your own spans, your own metrics, your own filters. On Kubernetes the Collector usually runs in two layers: a DaemonSet on every node (taking OTLP from the pods on the node, collecting node and pod metrics) and a central gateway (tail sampling, PII and sending to the backends). For the state of the cluster itself, kube-state-metrics (the state of deployment, pod and node objects) and the node exporter or the Collector's kubeletstats receiver are common choices. Kubernetes itself is the Kubernetes series' topic.

Before Going to Production: A Checklist

Before a new service goes to production we check that its telemetry is ready with this list; it's also in the repo as docs/telemetry-checklist.md:

• Resource: service.name, service.version and deployment.environment.name are correct and the same on every signal.
• Traces: incoming and outgoing HTTP, database and message queue calls produce spans; context propagation doesn't break between services.
• Metrics: RED metrics exist, business metrics are defined, the cardinality of the labels has been estimated.
• Logs: levels are set for production, log records carry trace_id, there's no personal data or it's masked.
• Sampling: tail sampling rules keep errors and slow requests; no head sampling in the application.
• Dashboards: the service's RED dashboard is in Git and loaded by provisioning.
• Alerts: symptom alerts, a "no telemetry" alert and a runbook for each are ready; the notification channel has been tested.
• Cost: daily spans, log records and series are estimated, retention periods are decided.

In Production

Telemetry needs an SLO too. If the data the Collector drops (such as otelcol_exporter_send_failed_spans_total), the data waiting in queues and the backend's delay aren't watched, you find out during an incident that the dashboards show incomplete data.

Keep environments separate but alike. Staging and production should use the same Collector config and the same dashboards; the difference should be just a few values such as deployment.environment.name and the sampling rate. That way a telemetry problem you didn't see in staging doesn't surprise you in production.

Common Pitfalls

Using head and tail sampling together. As we measured, most slow traces are lost and exemplars go down; tail sampling's purpose is defeated.

Producing cost or count reports from the trace backend. Sampled traces are a sample; for counts and rates look at metrics.

Putting the connection string or the backend token into the repo. The Application Insights connection string and the OTLP endpoint's headers are secrets; they should come from Key Vault or the environment's secret management.

Not knowing the managed service's default sampling. The Microsoft OpenTelemetry Distro's default of 5 traces per second keeps a very small share of traces for a high-traffic service; the numbers in Application Map are computed with a sampling correction, but you feel it when searching for individual traces.

A Small Experiment

★☆☆ Build a "telemetry volume" panel from the Collector's own metrics: spans, log records and metric points received per minute. Run the traffic tool and compare the per-order values with the table above.

★★☆ Write an alerting rule in Loki (the Loki ruler) that fires when a log line or attribute contains an e-mail address. Add an unmasked field to OrderConfirmationSender and show that the alert fires.

★★★ Deploy Bookstore to a local Kubernetes cluster with kind, install the OpenTelemetry Operator and, with the Instrumentation object above, turn the application's SDK off and automatic instrumentation on. Compare the span tree in Tempo in both cases: which spans disappeared?

The production side is done too. The series' last post looks along a different axis: what came to .NET's observability APIs from .NET 8 to .NET 11; IMeterFactory, Gauge, new built-in metrics, and ASP.NET Core in .NET 11 producing the HTTP span by itself, without an instrumentation package.