Alerting: Symptom-Based Alerts, Alertmanager, Grafana Alerting, Runbooks and Alert Fatigue

Version note: The examples in this post were run with .NET 10, OpenTelemetry .NET 1.21, the grafana/otel-lgtm 0.41.0 image (Prometheus 3.16.0, Grafana 13.3.0) and Alertmanager 0.35.0. Grafana Alerting and blackbox probes were not run for this post.

Guess First

We put the payment provider of Bookstore, the online bookstore used as the shared example across this site's series, into a bad day: 60 percent of attempts take 5 seconds. Prometheus has five alerting rules: error rate, checkout duration, payment retries, telemetry not arriving at all, and an always-firing "Watchdog". After the problem starts, how many notifications reach the on-call engineer's phone, and in which minute does the first one arrive? Then we stopped the API entirely; every checkout will fail. What does the error-rate alert do this time?

Symptom or Cause?

An alert has one job: to say that a human must do something right now. The Monitoring Distributed Systems chapter of Google's SRE book explains this with the difference between symptoms and causes: alert on what the user feels (checkout fails, checkout is slow); each cause (high CPU, the payment provider needs retries) is input for the investigation, not an alert. If CPU is at 90 percent and users aren't affected, there's no point waking someone at 3 a.m.; if users can't place orders, someone has to get up even when CPU is normal.

The second distinction is the type of notification. A page needs immediate action and wakes the on-call engineer. A ticket is a work item to look at during working hours. Bookstore's rule is simple: symptoms that affect users are severity: page, causes and early warnings are severity: ticket. This label decides below where Alertmanager sends the notification.

An Alerting Rule in Prometheus

An alerting rule is written in the same file format as recording rules; the difference is that an alert is created when the query returns a result. Bookstore's rules are in deploy/observability/alerts.yaml. The checkout duration alert:

# Customers wait: p95 of checkout above 3 seconds for 10 minutes.
- alert: CheckoutLatencyHigh
  expr: bookstore:checkout_duration_seconds:p95_5m > 3
  for: 10m
  labels:
    severity: page
    service: bookstore-api
  annotations:
    summary: "Checkout p95 is {{ $value | humanizeDuration }}"
    runbook_url: "docs/runbooks/checkout-latency.md"

expr reads the recording rule we wrote in the Prometheus post; the alert doesn't redo the expensive percentile calculation at every evaluation. for: 10m wants the condition to stay true without a break for 10 minutes; during that time the alert waits in the "pending" state. That keeps a spike of a few minutes from waking anyone. labels are for routing, annotations are for humans: the summary shown in the notification and the runbook link. The rules are wired into the Prometheus inside otel-lgtm with a copy of the image's own Prometheus config (lgtm-prometheus.yaml); we appended the rule files and the Alertmanager address to it and lowered the evaluation interval to 15 seconds. Checking rules with promtool (it's in the otel-lgtm image too) before giving them to Prometheus keeps a typo from quietly disabling an alert:

promtool check rules alerts.yaml rules.yaml
Checking alerts.yaml
  SUCCESS: 5 rules found

Checking rules.yaml
  SUCCESS: 4 rules found

The Burn-Rate Alert

A fixed threshold for the error rate, such as "alert above 1 percent", goes wrong in two ways: at night, with little traffic, a few errors make the ratio jump, while during the day a steady 0.8 percent error rate never produces an alert. Instead we set a target: an SLO. Bookstore's target is that over 30 days 99.5 percent of checkouts don't end with a 5xx. That means an error budget of 0.5 percent. The burn rate is how fast the budget is consumed: at 1 the budget lasts exactly 30 days, at 14.4 two percent of the budget goes every hour (14.4 / 720 hours) and the budget is gone in two days. How to choose the SLO and what to do when the budget runs out is the topic of the SRE and Production Incidents series; here we look at the alerting side.

# SLO: 99.5% of checkouts (POST /api/orders) do not end with 5xx, over 30 days (error budget 0.5%).
# Burning at 14.4x for an hour uses 2% of the 30-day budget: page. The 5-minute window makes it stop quickly.
- alert: CheckoutErrorBudgetBurn
  expr: |
    bookstore:orders_http_errors:ratio_rate1h > (14.4 * 0.005)
      and
    bookstore:orders_http_errors:ratio_rate5m > (14.4 * 0.005)
  for: 2m
  labels:
    severity: page
    service: bookstore-api

The two windows work together. The one-hour window makes sure the alert fires only when the budget consumption really matters; the 5-minute window keeps the alert from firing for another hour after the problem is over. This multi-window approach comes from the chapter on alerting on SLOs in Google's SRE Workbook; the same chapter recommends a second page alert for slower burns with a 6-hour window and a 6x burn rate, and a ticket alert with a 3-day window. We added one more recording rule to rules.yaml for the 1-hour ratio.

Alertmanager: Routing, Grouping and Inhibition

Prometheus decides when an alert exists; Alertmanager decides who gets it, how often and through which channel. We ran Alertmanager's official binary (0.35.0) on the host next to the container; Prometheus reaches it through host.docker.internal:9093. Instead of a real on-call tool or chat channel, a small application called tools/Bookstore.AlertSink receives the notifications and writes each one as a single line:

# Alertmanager for Bookstore (local). Notifications go to tools/Bookstore.AlertSink as webhooks;
# in production these receivers would be the on-call tool, the ticket queue and a heartbeat service.
route:
  receiver: ticket
  # One notification per alert and service, not one per series.
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: on-call
    - matchers: [severity="heartbeat"]
      receiver: heartbeat
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m

receivers:
  - name: on-call
    webhook_configs:
      - url: http://localhost:5095/alerts/on-call
  - name: ticket
    webhook_configs:
      - url: http://localhost:5095/alerts/ticket
  - name: heartbeat
    webhook_configs:
      - url: http://localhost:5095/alerts/heartbeat

inhibit_rules:
  # When the API sends no telemetry at all, its other alerts are noise: they can't be evaluated anyway.
  - source_matchers: [alertname="BookstoreApiTelemetryMissing"]
    target_matchers: [severity=~"page|ticket"]
    equal: [service]

alertmanager --config.file=deploy/observability/alertmanager.yml --storage.path=./alertmanager-data --cluster.listen-address=
amtool check-config deploy/observability/alertmanager.yml
amtool config routes test --config.file=deploy/observability/alertmanager.yml severity=page alertname=CheckoutLatencyHigh service=bookstore-api
on-call

group_by collects the different series of the same alert (for example the same alert from each of ten pods) into one notification. group_wait waits 30 seconds before the first notification to see whether other alerts join the group; repeat_interval is how often an unresolved alert is sent again as a reminder. The answer to Guess First's first question is what AlertSink wrote after we started the traffic at 20:22:30:

20:25:28 info: Program[0] [on-call] firing CheckoutErrorBudgetBurn severity=page Checkout is failing: 14.79% of orders end with 5xx runbook=docs/runbooks/checkout-errors.md
20:33:13 info: Program[0] [on-call] firing CheckoutLatencyHigh severity=page Checkout p95 is 10s runbook=docs/runbooks/checkout-latency.md
20:38:13 info: Program[0] [ticket] firing PaymentProviderRetriesHigh severity=ticket ...

Two notifications reached the on-call phone: the error budget alert in the third minute, the duration alert in the eleventh. The retry alert went to the work queue as a ticket, not a page, in the sixteenth minute (we shortened the summary on the last line; we fixed the summary text after this run). Three alerts, three faces of the same incident: all of them lead to one runbook, the payment provider. After contacting the payment provider, the on-call engineer added a silence for the retry alert:

amtool --alertmanager.url=http://localhost:9093 silence add alertname=PaymentProviderRetriesHigh service=bookstore-api --duration=2h --author=emre --comment="Provider incident open, ticket filed"
389fa9ca-4f62-49b0-9471-391b5e61a07f

amtool --alertmanager.url=http://localhost:9093 alert query
Alertname                Starts At                Summary                                             State
Watchdog                 2026-11-14 20:21:43 UTC  Alerting pipeline is alive                          active
CheckoutErrorBudgetBurn  2026-11-14 20:24:58 UTC  Checkout is failing: 19.12% of orders end with 5xx  active
CheckoutLatencyHigh      2026-11-14 20:32:43 UTC  Checkout p95 is 10s                                 active

A silence stops notifications for matching alerts for a given time; who silenced it, why and until when is recorded. That's the right way to stop the same notification from arriving again and again while a known problem is being worked on, not changing the rule.

For Guess First's second question we stopped the API at 20:39:02. The expectation was that the error-rate alert would climb even higher; the opposite happened:

20:45:28 info: Program[0] [on-call] resolved CheckoutErrorBudgetBurn severity=page Checkout is failing: 19.56% of orders end with 5xx runbook=docs/runbooks/checkout-errors.md
20:46:43 info: Program[0] [on-call] firing BookstoreApiTelemetryMissing severity=page No telemetry from bookstore-api for 7 minutes runbook=docs/runbooks/api-down.md

The error budget alert became "resolved". With the API down, no requests, no errors and no metrics arrive; with nothing left to compute, the 5-minute ratio returns an empty result, and an empty result isn't an alert. The only rule that caught the situation was absent_over_time(target_info{job="bookstore-api"}[5m]): it fires if no target_info has come from the service in the last 5 minutes, and together with for: 2m it reached the on-call engineer after about 7 minutes. For metrics pushed over OTLP there's no up series like the one Prometheus keeps for scrapes; a rule of this kind answers the question "is the service saying anything at all".

We saw inhibition at this point too. The retry alert was still firing in Prometheus, but Alertmanager suppressed it both because of the silence and because of the telemetry alert. To make the behavior clear, we sent a CheckoutLatencyHigh alert by hand with amtool alert add; Alertmanager's API showed both as "suppressed":

curl -s localhost:9093/api/v2/alerts | jq -c '.[] | {a: .labels.alertname, state: .status.state, inhibitedBy: (.status.inhibitedBy|length), silencedBy: (.status.silencedBy|length)}'
{"a":"BookstoreApiTelemetryMissing","state":"active","inhibitedBy":0,"silencedBy":0}
{"a":"Watchdog","state":"active","inhibitedBy":0,"silencedBy":0}
{"a":"PaymentProviderRetriesHigh","state":"suppressed","inhibitedBy":1,"silencedBy":1}
{"a":"CheckoutLatencyHigh","state":"suppressed","inhibitedBy":1,"silencedBy":0}

When the API is down, the on-call engineer gets a single notification, and it leads to the right runbook: check first whether the API is up. The rest is a reflection of the same incident.

Dead Man's Switch and Blackbox Probes

The alerting system itself can break: Prometheus stops, the webhook Alertmanager sends to changes, someone edits a route wrongly. Then no notification arrives at all, and the silence looks like "everything is fine". The Watchdog alert exists for that: with expr: vector(1) it always fires and goes to a heartbeat receiver every minute through its own route. In production that receiver is an external service that raises an alert itself when it hasn't received a heartbeat for a while. That's the "dead man's switch" approach. In our run AlertSink got a [heartbeat] firing Watchdog line every minute.

Another blind spot is the outside view. All our alerts come from the application's own telemetry; if the DNS record is broken or the TLS certificate has expired, requests never reach the application and its metrics say nothing. Prometheus's blackbox exporter tries a URL from the outside at regular intervals and produces metrics such as probe_success, probe_duration_seconds and probe_ssl_earliest_cert_expiry; a rule that opens a ticket when fewer than 14 days are left on the certificate is the simplest use. We didn't run it for this post; certificate management itself is the SSL/TLS series' topic.

Grafana Alerting

Grafana's own alerting does the same job from a different place; we didn't run it for this post. Rules are defined in Grafana and can use any data source Grafana can query besides Prometheus, such as Loki, Tempo or a SQL database. Alertmanager's routes correspond to notification policies, receivers to contact points; silences and grouping exist too. Grafana runs an Alertmanager internally and can also send to an external one. The choice is practical: if you want rules to live in Git next to the metrics and be evaluated where Prometheus is, use Prometheus and Alertmanager; if you need alerts based on logs or on several data sources and the team already works in Grafana, use Grafana Alerting. Don't use both for the same alert; the same incident produces two notifications through two paths.

Runbooks and Alert Fatigue

Every alert has a runbook_url, and the file it points to is in the repo under docs/runbooks/. A runbook is written for the person who wakes up at 3 a.m.: what the alert means, who is affected, what to do in the first 5 minutes. The error budget alert's, shortened:

# CheckoutErrorBudgetBurn

**What it means:** more than 7.2% of `POST /api/orders` requests end with 5xx, both over the last hour and the last
5 minutes. At this rate the 30-day error budget (0.5%) is gone in about two days.

**First 5 minutes**
1. Open the "Bookstore API (RED)" dashboard: which status codes, since when, one instance or all?
2. Look at "Payment provider: retries and timeouts". 502s with many timeouts point at the payment provider.
3. In Grafana Explore (Loki): `{service_name="bookstore-api"} | detected_level="error"`; open a line and follow its `trace_id` to Tempo.
4. Was there a deploy in the last hour? If yes and the errors started with it: roll back first, investigate later.

Runbooks live in the repo, in the team's shared language, which is why they're in English. An alert without a runbook is one of two things: either nobody knows what to do, and then the alert itself should be discussed, or the action is always the same, and then it should be automated.

Alert fatigue is a team starting to ignore notifications because there are too many useless ones. It can be measured: Prometheus keeps the state of every alert in the ALERTS series, and a query such as count by (alertname) (changes(ALERTS{alertstate="firing"}[7d])) shows how many times each alert opened and closed in a week. An alert that keeps opening and closing (flapping) usually points to a for that's too short, or to a threshold that looks at an instant value instead of a trend. A short weekly alert review is a good habit: for every page of the last week ask "did this notification need an action?"; if not, demote the alert to a ticket, change its threshold or delete it.

In Production

Manage alerting rules like code. alerts.yaml is in the repo, changes through pull requests, and CI runs promtool check rules and amtool check-config. You can write tests for the rule itself too: promtool test rules checks with sample series whether an alert fires at the expected moment.

Monitor Prometheus and Alertmanager themselves. Prometheus's prometheus_notifications_dropped_total and Alertmanager's alertmanager_notifications_failed_total metrics are the first place that tells you notifications are getting lost. In production Alertmanager runs as several instances with gossip between them for high availability; locally we turned that off with --cluster.listen-address=.

Actually test the notification channel. The phone number in the on-call tool, the webhook's token or the channel name changes and nobody notices. The Watchdog tests this all the time; on top of that, sending a test alert at the start of each on-call rotation is a cheap check.

Common Pitfalls

Alerting on every exception. A single exception is usually one request of one user; alerts go on rates and budgets, not on individual events. Look for exceptions in logs and traces, and put the alert on the outcome (checkout fails).

Forgetting that the alert goes quiet when there's no data. As we saw above, when the API stopped the error-rate alert didn't fire, it resolved; there were no requests left to compute a ratio from. A "no telemetry" alert is a must for every service.

Not noticing a notification channel that has gone silent. No notifications for a whole month can be good news or a broken webhook. Without the Watchdog and an external heartbeat service you can't tell them apart.

Paging on a cause. If the "payment provider needs retries" alert had woken the on-call engineer, it would also wake them for situations that don't affect users. Causes are tickets; symptoms are pages.

Giving for a very short duration, or none. A single bad evaluation fires the alert and the next one resolves it; that's the most common reason for flapping.

A Small Experiment

★☆☆ Use amtool config routes test to check which receiver alerts labelled severity=ticket and severity=heartbeat go to; then see where an alert without a severity label ends up.

★★☆ Add the SRE Workbook's second page alert: a 6x burn rate over 6-hour and 30-minute windows. Write the recording rules it needs, and use promtool test rules with a test file to verify that the alert fires at the expected moment.

★★★ Add a certificate expiry alert to Bookstore: run the blackbox exporter, probe the API's HTTPS address and write a ticket alert with the condition probe_ssl_earliest_cert_expiry - time() < 14 * 86400. Show that the alert fires with a short-lived test certificate.

The alerts now fire on the right symptoms and go to the right person. But all of this ran in a local container; in production telemetry comes with a bill, sampling decisions, responsibility for personal data and hundreds of pods on Kubernetes. In the next post we go to production: Azure Monitor and Application Insights, sampling and cost, PII and KVKK, automatic instrumentation on Kubernetes, and the telemetry checklist for a new service before it goes to production.