On-Call and Runbooks: Rotations, Escalation and Writing Actionable Runbooks

Version note: The examples in this post were run with .NET 10, Prometheus 3.16.0 (including promtool), the amtool of Alertmanager 0.35.0 and dotnet-counters 10.0. PagerDuty and similar paging tools were not run; they are described from their documentation.

Guess First

Three people run production for Bookstore, the online bookstore used as the shared example across this site's series: Selin, Emre and Ayşe. They are setting up a 24/7 on-call rotation. According to Google's SRE book, at most 25 percent of an engineer's time should go to on-call, and at any moment there should be a primary and a secondary on call. How many people does a single-site team need at minimum to follow this rule? What should the three-person Bookstore team do in light of it? Second question: the checkout alert fired at 02:10 at night and the primary did not hear the phone. When, and to whom, should the alert go next? The answers are halfway down.

Designing the Rotation

The basic parts of on-call are simple: a rotation, the order and length of shifts among people; at any moment a primary, the first person called; and a secondary, called when the primary does not answer or asks for help. Companies with teams in different time zones use follow-the-sun: everyone is on call during their own working hours and nobody is woken up at night. For small teams in a single office, a weekly rotation is the most common setup.

The "Being On-Call" chapter of the SRE book builds the numbers like this: at least 50 percent of the time goes to engineering work, at most 25 percent to on-call. If two people (primary and secondary) are on call at any moment and shifts are weekly, the 25 percent rule means at least eight people for a single-site team: every engineer is primary or secondary one week a month. For dual-site teams it suggests at least six people per site. The answer to the first question in Guess First is eight.

By that rule, Bookstore's three people cannot hold a 24/7 rotation; if they did, each would spend two thirds of their time on call. Realistic options for a small team:

• At night and at weekends, only pages that are really urgent: checkout's burn rate alert yes; search latency, disk usage or a certificate warning become tickets for working hours.
• A weekly rotation with consecutive primary and secondary: this week Emre is primary and Ayşe secondary; next week Ayşe is primary and Selin secondary.
• No other planned work in the on-call week: no sprint items that week, only toil and small reliability work.
• Compensation in writing: whoever got paged at night starts late the next day or takes a day off.

A Page Budget and Sustainable On-Call

The number of pages decides how heavy a shift is. The SRE book assumes an incident takes 6 hours on average (detection, response, postmortem and follow-up work included) and concludes that a 12-hour shift can take at most 2 incidents. The target distribution has a median of zero: most shifts should pass without a page. If one component pages every day, the problem is not the person on call but the system or the alert.

Counting pages is one query in Prometheus. For the live outage of the previous post (the payment provider rejected 30 percent of requests for 10 minutes), which alert fired for how many minutes (we ran the query at the end of the run, at 21:44:22):

sum by (long_window, severity) (count_over_time(ALERTS{alertname="CheckoutErrorBudgetBurn", alertstate="firing"}[1h])) * 15 / 60

{long_window="3d", severity="ticket"} 41.5
{long_window="6h", severity="page"} 34.75
{long_window="1h", severity="page"} 7

Rules are evaluated every 15 seconds, so the number of samples times 15 seconds gives the minutes. Two pages in an hour, about 42 minutes in total: more than production would see because of the lab's short windows (explained in the previous post), but this is the number that tells how the on-call person's night went. For a weekly total the same query runs with [7d].

Writing this number into the weekly handoff note makes the on-call load visible. In Bookstore the handoff note is a template in the repo (docs/oncall/handoff.md); the person handing over the shift goes through it with the next one in a 15-minute call on Monday:

# On-call handoff: week <NN>

From: <name> · To: <name> · Handoff call: Monday 10:00 (15 minutes)

## Pages this week
| When (UTC) | Alert | What it was | Time spent | Runbook OK? |
|---|---|---|---|---|

## Still open
- Incidents or tickets the next person inherits, with links.

## Watch out for
- Deploys, campaigns, provider maintenance windows planned for next week.
- Flapping alerts or known noise (and the ticket that will fix it).

## Error budget (30 days)
- Checkout availability: <remaining> · Checkout latency: <remaining>

## Toil
- Manual work done this week that should be automated (one line each, with a ticket).

Escalation and Routing

Two layers decide who an alert goes to. The first is Alertmanager: it sends the alert to the right receiver by severity, groups repeats of the same alert, and holds back the ticket for a problem while its page is firing. The second is the paging tool (PagerDuty, incident.io, Splunk On-Call and the like): phone calls, SMS, the calendar of who is on call, and the escalation policy, which says who the alert goes to and after how many minutes if nobody answers.

# Alertmanager routing for Bookstore (check: amtool check-config deploy/alertmanager/alertmanager.yml).
# page   → the on-call person's phone via the paging tool (escalation to the secondary happens there)
# ticket → the team's work tracker, read during working hours
route:
  receiver: team-ticket
  group_by: [alertname, long_window]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: oncall-pager
      repeat_interval: 1h
    - matchers: [severity="ticket"]
      receiver: team-ticket
      group_wait: 10m
      repeat_interval: 24h

# While the page of an alert is firing, its ticket is held back.
inhibit_rules:
  - source_matchers: [severity="page"]
    target_matchers: [severity="ticket"]
    equal: [alertname]

receivers:
  - name: oncall-pager
    pagerduty_configs:
      # The key is never in the repo; it is mounted from the secret store at deploy time.
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-routing-key
        severity: critical
  - name: team-ticket
    webhook_configs:
      - url: https://tickets.example.com/hooks/alertmanager

We checked the routing tree and where the two alerts go with amtool:

amtool config routes --config.file=deploy/alertmanager/alertmanager.yml
Routing tree:
.
└── default-route  receiver: team-ticket
    ├── {severity="page"}  receiver: oncall-pager
    └── {severity="ticket"}  receiver: team-ticket

amtool config routes test --config.file=deploy/alertmanager/alertmanager.yml alertname=CheckoutErrorBudgetBurn severity=page long_window=1h
oncall-pager

amtool config routes test --config.file=deploy/alertmanager/alertmanager.yml alertname=CheckoutErrorBudgetBurn severity=ticket long_window=3d
team-ticket

amtool config routes test --verify.receivers=oncall-pager --config.file=deploy/alertmanager/alertmanager.yml alertname=CheckoutErrorBudgetBurn severity=ticket
team-ticket
WARNING: Expected receivers did not match resolved receivers.

The output of the last command matters too: a ticket does not go to a page receiver, and --verify.receivers stops with a non-zero exit code on a wrong expectation. As a step in the CI pipeline, this command keeps routing changes from accidentally ringing phones at night.

The escalation policy itself lives in the paging tool. PagerDuty's open incident response documentation suggests a policy that moves to the secondary after 5 minutes if the primary does not answer, and treats the whole team as the last resort at the third level. That answers the second question in Guess First: a page that fired at 02:10 should ring the secondary's phone at 02:15. We did not run this part against an account; the settings come from the tool's documentation. A note when choosing a tool: Atlassian stopped selling Opsgenie in June 2025 and shuts the product down on 5 April 2027; it should not be chosen for a new setup.

A Runbook for Every Alert

A runbook is the document that says step by step what to do when a specific alert fires. The person woken up at 02:10 may not be the one who knows the system best; the runbook tells them where to look in the first five minutes and what to do in which case. The parts of a good runbook: what the alert means and what users experience; the checks for the first five minutes (each a command or query you can copy); a mitigation for each likely cause (stop the impact first, find the cause later); escalation; and the aftermath (timeline, postmortem criteria).

We keep the runbook next to the code, as Markdown in the repo: docs/runbooks/checkout-error-budget-burn.md. A wiki page goes stale independently of the code; a runbook in the repo is updated in the same pull request that changes the alert rule. The alert points to the runbook too:

      - alert: CheckoutErrorBudgetBurn
        expr: |
          slo:checkout_availability:error_ratio_rate1h > (14.4 * 0.005)
          and
          slo:checkout_availability:error_ratio_rate5m > (14.4 * 0.005)
        labels:
          severity: page
          long_window: 1h
        annotations:
          summary: "Checkout is burning its 30-day error budget 14.4 times too fast (1h and 5m windows)"
          description: >-
            Checkout errors (5m): {{ with query "slo:checkout_availability:error_ratio_rate5m" }}{{ . | first | value | humanizePercentage }}{{ end }}.
            {{ with query "topk(1, sum by (server_address) (rate(http_client_request_duration_seconds_count{job='bookstore-api', error_type!=''}[5m])))" }}Most failing outgoing calls: {{ (. | first).Labels.server_address }} ({{ . | first | value | printf "%.2f" }}/s).{{ else }}No failing outgoing calls.{{ end }}
          runbook_url: https://git.example.com/bookstore/blob/main/docs/runbooks/checkout-error-budget-burn.md

The first five minutes of the runbook, as fixed after the trial below:

## First 5 minutes
1. Acknowledge the page and write one line in the incident channel: "Looking at CheckoutErrorBudgetBurn".
2. How bad, and which answer? In Prometheus:
   ```promql
   sum by (http_response_status_code) (rate(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/"}[5m]))
   ```
3. Is it the payment provider? Bookstore.Api's calls to it, by host and result:
   ```promql
   sum by (server_address, http_response_status_code, error_type) (rate(http_client_request_duration_seconds_count{job="bookstore-api"}[5m]))
   ```
   503, or `error_type="System.Threading.Tasks.TaskCanceledException"` (timeout; such series have no status code)
   from the provider's host: go to "Payment provider".
4. Was there a deploy? Versions that sent metrics in the last hour:
   ```promql
   count by (service_version) (count_over_time(target_info{job="bookstore-api"}[1h]))
   ```
   Two versions, or a version younger than the alert: go to "Bad deploy".
5. Is the process itself healthy? On the host or in the pod, find the PID first (never attach by name: another process
   may have the same name), then watch it:
   ```
   dotnet-counters ps
   dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting
   ```
   Thread pool queue length growing or CPU near 100 %: go to "Neither".

Trying the Runbook on a Real Outage

We did not leave the runbook on paper: during the live outage of the previous post, while the payment provider rejected 30 percent of charges, we followed it step by step in the fourth minute of the outage (21:06). The 6-hour page was firing at that point. Step 2:

{http_response_status_code="201"} 2.8841801665245628
{http_response_status_code="502"} 0.9085599098956854

2.88 successful and 0.91 failed checkouts a second: an error rate of 24 percent over the 5-minute window (the first minute of the window was before the outage). Step 3 showed the cause:

{error_type="404", http_response_status_code="404", server_address="localhost"} 0.1999978947590025
{http_response_status_code="200", server_address="127.0.0.1"} 2.8841801665245623
{http_response_status_code="200", server_address="localhost"} 0.0666659649196675
{error_type="503", http_response_status_code="503", server_address="127.0.0.1"} 0.9086392774812895

Calls to the payment provider (at 127.0.0.1 in the lab) get 503 0.91 times a second; the same rate as the checkouts' 502s. The same output held something we did not expect: some calls to localhost were getting 404. That is the API's own OTLP exporter. The UseOtlpExporter() we used in the second post of the series sends logs to the same address together with the metrics, and Prometheus answers with 404 because it does not accept logs. Following the runbook, we found an unrelated bug as well. Step 4:

{service_version="1.0.0+623d076ccae5222ab9e0e03dcf7caebe482e97cd"} 1

Only one version sent metrics in the last hour, so there was no deploy (the part after + is the build's commit). For step 5 we attached dotnet-counters to the API process; the relevant lines of the output (CSV, time column removed):

Provider,Counter Name,Counter Type,Mean/Increment
System.Runtime,dotnet.thread_pool.queue.length ({work_item}),Metric,0
System.Runtime,dotnet.thread_pool.thread.count ({thread}),Metric,7
System.Runtime,dotnet.process.cpu.time (s / 1 sec)[cpu.mode=system],Rate,0.010300000000000864
System.Runtime,dotnet.process.cpu.time (s / 1 sec)[cpu.mode=user],Rate,0.024915000000007126
Microsoft.AspNetCore.Hosting,http.server.request.duration (s)[http.request.method=POST;http.response.status_code=201;http.route=/api/orders/;network.protocol.version=1.1;url.scheme=http;Percentile=50],Metric,0.069091796875
Microsoft.AspNetCore.Hosting,http.server.request.duration (s)[error.type=502;http.request.method=POST;http.response.status_code=502;http.route=/api/orders/;network.protocol.version=1.1;url.scheme=http;Percentile=50],Metric,0.0110931396484375

The process is healthy: the thread pool queue is empty, there are 7 threads, CPU is 3.5 percent of one core. The 502s come back in 11 ms; the API is not waiting and timing out, it passes on the provider's fast rejection. The runbook sends us to the "Payment provider" section; in the lab we fixed the provider at 21:12:20.

The trial tested two more things in the runbook. First, the error_type value we wrote to recognise the provider's timeout was a guess. We made the provider wait 4 seconds on every charge for 90 seconds (the API's payment timeout is 3 seconds) and ran the same query:

sum by (server_address, http_response_status_code, error_type) (increase(http_client_request_duration_seconds_count{job="bookstore-api"}[90s]))

{http_response_status_code="200", server_address="127.0.0.1"} 0
{http_response_status_code="200", server_address="localhost"} 6
{error_type="System.Threading.Tasks.TaskCanceledException", server_address="127.0.0.1"} 329.27580000000006

The guess was right, but it should not have stayed a guess; and we saw a detail we did not know: on a timeout there is no http_response_status_code label at all, because no answer came. A query filtering by status code would never show timeouts. The 404s on localhost are gone in this measurement, because we fixed the exporter before it (below). Second, dotnet-counters monitor -n Bookstore.Api in step 5 finds the process by name. In the lab the API ran as dotnet Bookstore.Api.dll, so the process name was dotnet; on top of that, another copy of Bookstore.Api with the same name was running on the same machine. Attaching by name means watching the wrong process. We fixed the runbook to find the PID first with dotnet-counters ps. We also changed the exporter to send metrics only:

.WithMetrics(metrics => metrics
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddView("http.server.request.duration", new ExplicitBucketHistogramConfiguration
    {
        Boundaries = [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 0.8, 1, 2.5, 5, 7.5, 10]
    })
    // Metrics only. OTEL_EXPORTER_OTLP_ENDPOINT / _PROTOCOL decide where they go (here: Prometheus' OTLP
    // receiver). UseOtlpExporter() would also send logs there, and Prometheus answers those with 404.
    .AddOtlpExporter());

Turning a Runbook Step into Automation

Steps 2 and 3 of the runbook are the same two queries in every outage. Instead of the on-call person typing them at midnight, the alert itself can carry them: in Prometheus, alert annotations are processed as templates, and a template can run a query with query. The description above does this; we added a series standing for the provider's 503s and the expected text to the promtool test:

  # Five minutes in which every checkout fails, after three days without errors: burn rate 200, a page after about 4.3 minutes,
  # and quiet again a few minutes after it ends.
  - interval: 1m
    input_series:
      - series: 'http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/", http_request_method="POST", http_response_status_code="201"}'
        values: '0+600x4320 2592000+0x5 2592000+600x30'
      - series: 'http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/", http_request_method="POST", http_response_status_code="502"}'
        values: '0+0x4320 600+600x4 3000+0x30'
      # The cause: the payment provider answers 503 to every charge.
      - series: 'http_client_request_duration_seconds_count{job="bookstore-api", server_address="payments.example.com", http_response_status_code="503", error_type="503"}'
        values: '0+0x4320 600+600x4 3000+0x30'
    alert_rule_test:
      - eval_time: 72h4m
        alertname: CheckoutErrorBudgetBurn
        exp_alerts: []
      - eval_time: 72h5m
        alertname: CheckoutErrorBudgetBurn
        exp_alerts:
          - exp_labels:
              severity: page
              long_window: 1h
            exp_annotations:
              summary: "Checkout is burning its 30-day error budget 14.4 times too fast (1h and 5m windows)"
              description: "Checkout errors (5m): 100%. Most failing outgoing calls: payments.example.com (10.00/s)."
              runbook_url: https://git.example.com/bookstore/blob/main/docs/runbooks/checkout-error-budget-burn.md
  ...

The test passed. In the live trial (with the provider rejecting half of the charges), the text in Prometheus' /api/v1/alerts answer:

curl -s http://localhost:9090/api/v1/alerts
...
{
  "labels": {
    "alertname": "CheckoutErrorBudgetBurn",
    "long_window": "1h",
    "severity": "page"
  },
  "annotations": {
    "description": "Checkout errors (5m): 48.85%. Most failing outgoing calls: 127.0.0.1 (1.84/s).",
    "runbook_url": "https://git.example.com/bookstore/blob/main/docs/runbooks/checkout-error-budget-burn.md",
    "summary": "Checkout is burning its 30-day error budget 14.4 times too fast (1h and 5m windows)"
  },
  "state": "firing",
  "activeAt": "2026-11-14T21:48:15.11230018Z"
}
...

When the on-call person picks up the phone, the answers to the first two steps are in the notification, with the runbook link next to them. The next step could be to turn this into automation: for example an automatic rollback if there was a deploy in the last hour and the error rate started after it. The rollback and kill switch post of the Deployment Strategies series covers the criteria for such decisions.

Self-healing, the system fixing a problem by itself without a person, has a risk of its own. A rule like "restart the API when the error rate goes up" would have fixed nothing in our outage; it would have dropped the orders in flight and piled the load on the provider at every restart. Automate a step that has been applied by hand from the runbook at least a few times with a known result, and add an alert for every automation itself ("more than three automatic rollbacks in the last hour"). To measure toil, the line in the handoff note is enough: if the same manual intervention shows up two weeks in a row, its automation has earned priority.

In Production

Test the runbook. A runbook nobody tried turns out wrong the first time it is really used: even ours had a wrong error type name and a process name that does not work on this machine. Game days, and a newcomer following the runbook through an outage scenario, are the cheapest way to fix it.

Shadow shifts. For the first few weeks a new team member goes on call next to an experienced person, taking the pages together. PagerDuty's documentation suggests this as well.

Tickets can wait during a shift. Opening a work item for a non-urgent warning that arrives outside working hours and going back to sleep is the right behaviour; responding to every warning at night makes on-call unsustainable.

Every page asks a question: should this alert really have been a page? Asking "did the runbook help, could this alert have been a ticket" in the weekly handoff keeps alert fatigue from building up.

Common Pitfalls

An alert without a runbook. An "alert I don't know what to do about" makes the on-call person spend their time guessing. An alert you cannot write a runbook for is also a suspicious page.

"Decide depending on the situation" in a runbook. Every step should be a concrete command or query; "look at the logs" is not a step, "run this query, if you see 503 go there" is.

A one-person rotation. Without a secondary, there is nobody for a primary who does not hear the phone, is ill or on the road. Even in a small team a secondary is a must.

Making restart the default fix. In our scenario a restart does not fix the payment provider; it only drops the orders being processed. Do not put a restart in a runbook without knowing what it fixes.

A Small Experiment

★☆☆ Add a route to the Alertmanager configuration that sends alerts with severity="page" and a team="search" label to a separate receiver, and verify it with amtool config routes test --verify.receivers.

★★☆ Set the fake payment provider's slowRatio to 0.3 and slowDelayMs to 4000 (Bookstore.Api's payment timeout is 3 seconds). Follow the runbook from start to finish: which step shows the cause, which one misleads? Fix the runbook accordingly.

★★★ Set a monthly page budget for your own team (for example at most 2 night pages a week). Write a recording rule that computes the weekly page count from Prometheus' ALERTS series, and add an alert that opens a ticket for the team when the budget is exceeded.

The runbook covers the first five minutes of an alert. But some outages go beyond one person's response: several teams, customer communication and a mitigation that takes hours. In the next post we cover declaring an incident, severity levels, the incident commander role and the "mitigation first" principle, with a tabletop exercise on the payment provider's timeouts.