Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
What Is Observability? How It Differs from Monitoring and the Question Each of Logs, Metrics and Traces Answers
Observability is being able to ask, from the logs, metrics and traces a system emits, a question you had not thought of beforehand; monitoring watches for known failures. We narrow down Bookstore's checkout that sometimes takes 8 seconds with logs, metrics from dotnet-counters and traces, and look at the telemetry pipeline and the cost of cardinality.
How OpenTelemetry Is Structured and a First Setup in .NET: SDK, Instrumentation, OTLP, Resource and the Aspire Dashboard
To add OpenTelemetry to .NET you configure the resource, instrumentation and UseOtlpExporter through AddOpenTelemetry; since ILogger, Meter and ActivitySource are already in the platform, the SDK just collects them and sends them over OTLP. We set up Bookstore.Api with the 1.21 packages, open the slow checkout's trace in the standalone Aspire Dashboard, and see AddService in code overriding OTEL_SERVICE_NAME and silent export errors.
Producing Metrics in .NET: Counters, Histograms, Gauges, Built-in Metrics and a Cardinality Budget
In .NET, metrics are produced with the instruments of a Meter taken from IMeterFactory: Counter for counts, UpDownCounter for totals that go up and down, Histogram for durations, Gauge for a current value set from outside. We write Bookstore's checkout metrics, look at built-in metrics and the RED/USE methods, test with MetricCollector and measure what the SDK does at its 2000-series cardinality limit.
Distributed Tracing: Spans, Context Propagation, W3C traceparent and Custom Spans with ActivitySource
Distributed tracing records every step of a request as spans carrying the same trace id; in .NET a span is an Activity, and it travels between services in the W3C traceparent header. We add spans to Bookstore's checkout with ActivitySource, record exceptions and status, and look at the traceparent of retries, baggage, propagation over a queue, the null Activity and where starting traces in the browser stands today.
The OpenTelemetry Collector: Receivers, Processors, Exporters, Tail Sampling and PII Filtering
The OpenTelemetry Collector is a separate process that decouples the application from backends and receives and processes telemetry in pipelines of receivers, processors and exporters. We set it up for Bookstore in a one-container LGTM stack, filter unneeded logs, hash the e-mail address, keep failed and slow traces with tail sampling and measure how head sampling throws away 90 percent of the orders people complain about.
Prometheus and PromQL: Scrape or Push, rate, histogram_quantile, Recording Rules and Native Histograms
.NET metrics get into Prometheus two ways: scraping the /metrics endpoint of the still-beta exporter, or pushing over OTLP to Prometheus 3's OTLP receiver. With Bookstore's metrics we work through rate, increase, histogram_quantile and error-rate queries, name translation and recording rules, and measure how accurate percentiles are with classic buckets versus native histograms.
Dashboards with Grafana: RED and USE Dashboards, Variables, Exemplars from Metrics to Traces and Dashboards as Code
A good Grafana dashboard answers one question: RED shows a service's health, USE shows resource saturation. We keep two dashboards for Bookstore as JSON in Git and load them with provisioning, feed variables from target_info and use exemplars to jump from a p99 spike to the trace of a slow request, and we measure why tail sampling leaves most exemplars without a trace.
Finding the Slow Request with a Trace Backend: Jaeger v2, Grafana Tempo and the N+1 Query in Checkout
Finding the slow request in a trace backend depends on asking the right question: Jaeger v2 searches indexed tags, Tempo uses TraceQL with conditions across spans. We open Bookstore's 7.6-second checkout in the waterfall, find the retry storm and the N+1 query, measure two fixes and hedging, and show why percentiles computed from tail-sampled traces mislead.
Log Backends: OpenTelemetry Logs, Loki and ELK — Labels, LogQL and Jumping from Logs to Traces
The way to keep logs cheap in Loki is to keep labels few: some resource attributes of an OpenTelemetry log become labels, the rest and the trace_id become structured metadata. With Bookstore we look at templated log bodies, service.instance.id opening a new stream on every restart, and LogQL queries for errors, levels, durations and jumping from logs to traces, then compare ELK, OpenSearch and Seq.
Alerting: Symptom-Based Alerts, Alertmanager, Grafana Alerting, Runbooks and Alert Fatigue
Alerts go on symptoms the user feels, and each one has a runbook; causes are tickets, not pages. For Bookstore we write burn-rate, latency, retry, missing-telemetry and Watchdog alerts in Prometheus, route them with Alertmanager, watch grouping, silences and inhibition during a real failure, and measure why the error-rate alert goes quiet when the API is down.
Observability in Production: Azure Monitor and Application Insights, Sampling, Cost, PII and Auto-Instrumentation on Kubernetes
In production the cost and risk of telemetry are managed through sampling, volume and personal data: head sampling in the application on top of tail sampling loses most slow traces. With Bookstore we measure both settings, build a per-order budget of spans, logs and series, audit PII in Loki, and look at the new Microsoft OpenTelemetry Distro for Azure Monitor, semantic convention migrations and the Kubernetes Operator.
What's New in Observability from .NET 8 to .NET 11: IMeterFactory, Gauge, New Built-in Metrics and Built-in Tracing in ASP.NET Core
In .NET 11 ASP.NET Core writes the OpenTelemetry attributes of the HTTP server span itself and MemoryCache publishes dotnet.cache.* metrics, with no instrumentation package. We map the releases from .NET 8's IMeterFactory and .NET 9's Gauge to .NET 10's Identity metrics and schema URLs, run the .NET 11 items on the GA SDK and give tips for reading older code.