SLOs and Error Budgets: Setting Targets, the Math of Nines and an Error Budget Policy

Version note: The examples in this post were run with .NET 10 (dotnet run on a file-based app), OpenTelemetry .NET 1.21, Prometheus 3.16.0 and OpenSLO's oslo CLI 0.13.0.

Guess First

For the checkout of Bookstore, the online bookstore used as the shared example across this site's series, we set a 30-day availability SLO: 99.5 percent. In the lab, a four-minute payment provider outage plus the background error rate of one in a thousand made 53 checkouts fail in total. We asked Prometheus for the remaining error budget and got this answer: -1.599727387380033, meaning 160 percent of the budget spent on top. Does a four-minute outage really use up a month's budget? Second question: at the same time, the remaining budget for the latency SLO came out at -92.35, even though 99.7 percent of checkouts were under 1 second. Which is right? Both answers are halfway down, and both come from a wrong question, not from wrong numbers.

SLA, SLO and SLI

The three terms are often mixed up. The SLI is the ratio we defined in the previous post: what share of valid checkout requests succeeded. The SLO is the target for that ratio: "99.5 percent over a rolling 30-day window". The SLA is a commercial contract: missing the target has a consequence such as a refund, a discount or the right to terminate.

Bookstore gives no SLA to individual customers; but on the Institutional Orders side, the contract with Nevin Hanım, who runs purchasing for a school, might contain a clause like "the order portal is available 99 percent of each month". The rule is simple: the SLO must be stricter than the SLA. With a 99 percent SLA and an internal target of 99.5 percent, there is still time to protect the SLA when the SLO is missed. Legal and sales write the SLA; the team that runs the service sets the SLO and changes it when needed.

Where Does the Target Come From?

There are two sources: how the service performs today and what users expect. Measure first. In a 7-minute window without faults, Bookstore's checkout showed 100 percent availability and 99.7 percent of checkouts stayed under 800 ms. Because the fake payment provider rejects one request in a thousand, the long-run expected availability is 99.9 percent. A target just below today's value is one the service can meet today. Then look at the user: a customer who waits more than a second on the payment page reloads it or gives up; 800 ms is a threshold that leaves room for the browser and the network.

Sometimes the target that would make users happy cannot be met today. For this case the SRE Workbook suggests an aspirational SLO: the stricter target is measured and tracked, but the error budget policy explicitly marks it as "no action required". That way the team tracks its progress towards the target without living in a permanent state of alarm. For Bookstore, keeping 99.9 percent of checkouts under 800 ms could be such a target.

The Math of Nines

Turning a target into minutes and requests makes the discussion concrete. For that we added a small file-based app to the repo; in .NET 10 a single .cs file runs without a project file:

// Error budget arithmetic for an SLO: how much downtime and how many failed requests a target allows.
// Usage: dotnet run tools/slo-math/slo-math.cs -- <windowDays> <requestsPerDay> <target%>...
// Example: dotnet run tools/slo-math/slo-math.cs -- 30 5500 99 99.5 99.9 99.95 99.99
using System.Globalization;

CultureInfo inv = CultureInfo.InvariantCulture;
int windowDays = int.Parse(args[0], inv);
long requestsPerDay = long.Parse(args[1], inv);
decimal[] targets = [.. args.Skip(2).Select(a => decimal.Parse(a, inv))];

double windowMinutes = windowDays * 24 * 60;
long requestsInWindow = requestsPerDay * windowDays;
Console.WriteLine($"window: {windowDays} days = {windowMinutes:N0} minutes, {requestsInWindow:N0} requests");
Console.WriteLine($"{"target",-8} {"budget",8} {"downtime",12} {"per day",10} {"failed requests",16} {"per day",8}");
foreach (decimal target in targets)
{
    double budget = (double)(100m - target) / 100;
    double downMinutes = windowMinutes * budget;
    double failed = requestsInWindow * budget;
    Console.WriteLine(
        $"{target.ToString(inv) + "%",-8} {budget,8:P2} {FormatMinutes(downMinutes),12} {FormatMinutes(downMinutes / windowDays),10} " +
        $"{failed,16:N0} {failed / windowDays,8:N1}");
}

static string FormatMinutes(double minutes) =>
    minutes >= 60 ? $"{minutes / 60:0.0} h" : minutes >= 1 ? $"{minutes:0.0} min" : $"{minutes * 60:0} s";

Bookstore takes about 1,290 orders a day (the number from the last post of the Database Performance and Scaling series). For a 30-day window:

dotnet run tools/slo-math/slo-math.cs -- 30 1290 99 99.5 99.9 99.95 99.99

window: 30 days = 43,200 minutes, 38,700 requests
target     budget     downtime    per day  failed requests  per day
99%         1.00%        7.2 h   14.4 min              387     12.9
99.5%       0.50%        3.6 h    7.2 min              194      6.5
99.9%       0.10%     43.2 min    1.4 min               39      1.3
99.95%      0.05%     21.6 min       43 s               19      0.6
99.99%      0.01%      4.3 min        9 s                4      0.1

99.5 percent means 194 failed checkouts or 3.6 hours of full outage a month. Every nine shrinks the budget tenfold: at 99.99 percent it is 4 failed orders a month, 4.3 minutes, less than it takes to roll back a single bad deploy. Seeing these rows side by side quickly pulls a "let's make it 99.99" request back to something realistic.

The window matters too. A rolling window, which counts the last 30 days at every moment, is closer to what users experience: a big outage on the last day of the month is not forgotten the next day. A calendar window, which resets at the start of the month, is easier for planning and reporting. The examples in the Workbook use a four-week window; because 28 days always contain the same number of weekends, comparisons stay consistent for services whose weekend traffic differs. The same target gives a slightly smaller budget over 28 days:

dotnet run tools/slo-math/slo-math.cs -- 28 1290 99.5

window: 28 days = 40,320 minutes, 36,120 requests
target     budget     downtime    per day  failed requests  per day
99.5%       0.50%        3.4 h    7.2 min              181      6.5

Then there is the dependency chain: if checkout calls the payment provider and the database synchronously, their availabilities multiply and set the ceiling checkout can reach. The calculation itself is in the System Design post "Back-of-the-Envelope Estimation: QPS, Storage, Bandwidth and Availability Math"; its conclusion here: an SLO cannot be more ambitious than what its hard dependencies promise.

Adding a Histogram Bucket for 800 ms

The latency SLO is "99 percent of valid checkouts under 800 ms". In the previous post we saw that a histogram can only count exactly up to its bucket boundaries, and ASP.NET Core's default boundaries do not include 0.8. There are two ways out: move the threshold to 0.75 or 1 second, or add a 0.8 boundary to the histogram. Since the threshold comes from users, we chose the second. In OpenTelemetry a View does that:

.WithMetrics(metrics => metrics
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    // ASP.NET Core's default boundaries plus 0.8: the checkout latency SLO counts requests under 800 ms,
    // and a histogram can only count exactly up to one of its bucket boundaries.
    .AddView("http.server.request.duration", new ExplicitBucketHistogramConfiguration
    {
        Boundaries = [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 0.8, 1, 2.5, 5, 7.5, 10]
    }))

The View applies by metric name, so every route's histogram gains a bucket; one more series per route and status combination. Keeping a separate histogram just for checkout is possible too, but reading the SLI from the framework's own metric means less code. The threshold in the SLI rule became le="0.8" as well. After restarting the API, the checkout buckets in Prometheus:

sum by (le) (http_server_request_duration_seconds_bucket{http_route="/api/orders/", http_response_status_code="201"})

{le="0.5"} 1030
{le="0.75"} 1030
{le="1"} 1030
...
{le="+Inf"} 1032
{le="0.8"} 123

le="0.8" is much smaller than the others because it only exists in the new process's series; the old process's series never carried that bucket. We will see the consequence in a moment.

Computing the Error Budget in Prometheus

We wrote the SLO as recording rules over a 30-day window. A remaining budget of 1 means untouched, 0 means used up, negative means the SLO was missed:

# Checkout SLOs (docs/slo/checkout.md): availability 99.5 %, latency 99 % under 800 ms, rolling 30 days.
# Remaining error budget: 1 = untouched, 0 = used up, below 0 = SLO missed.
groups:
  - name: bookstore-slo
    interval: 1m
    rules:
      - record: slo:checkout_valid:increase30d
        expr: |
          sum(increase(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/",
            http_request_method="POST", http_response_status_code=~"[235]..|429"}[30d]))

      - record: slo:checkout_availability:ratio30d
        expr: |
          sum(increase(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/",
            http_request_method="POST", http_response_status_code=~"[23].."}[30d]))
          / slo:checkout_valid:increase30d
      - record: slo:checkout_availability:budget_remaining
        expr: 1 - (1 - slo:checkout_availability:ratio30d) / (1 - 0.995)

      - record: slo:checkout_latency:ratio30d
        expr: |
          sum(increase(http_server_request_duration_seconds_bucket{job="bookstore-api", http_route="/api/orders/",
            http_request_method="POST", http_response_status_code=~"[235]..|429", le="0.8"}[30d]))
          / slo:checkout_valid:increase30d
      - record: slo:checkout_latency:budget_remaining
        expr: 1 - (1 - slo:checkout_latency:ratio30d) / (1 - 0.99)

A 30-day increase is an expensive query; interval: 1m evaluates this group once a minute while the other rules run every 15 seconds. The result in the lab:

{__name__="slo:checkout_valid:increase30d"} 4100.429934210527
{__name__="slo:checkout_availability:ratio30d"} 0.9870013630630998
{__name__="slo:checkout_availability:budget_remaining"} -1.599727387380033
{__name__="slo:checkout_latency:ratio30d"} 0.06644434382685678
{__name__="slo:checkout_latency:budget_remaining"} -92.35556561731424

Here is the answer to Guess First. The "30-day" window holds only about 30 minutes of data: 4,100 valid checkouts. In that window 53 errors are 1.3 percent and exceed the 0.5 percent budget 2.6 times. With real traffic, 1,290 checkouts a day and a window of 38,700 checkouts over 30 days, the same 53 errors would be about 27 percent of the 194-error budget: an annoying incident, but not one that misses the SLO. A budget computed before the window is full misleads; during the first 30 days of a new SLO, label the budget as "data so far".

The latency side is simply wrong, and it is a trap: the numerator only counts the new process's le="0.8" series, while the denominator also counts the old processes' requests. With the le="1" bucket, which exists for the whole period, the result is 99.7 percent. On the day you add a bucket to a histogram, that bucket has no history; until 30 days have passed, compute the SLO with an old boundary (0.75 or 1) or start the window at the moment of the change.

Writing the SLO Down

A recording rule is a calculation; an SLO is a decision the team agreed on. That decision belongs in a document in the repo: who owns it, which SLI, why this target, what it does not measure. In Bookstore, docs/slo/checkout.md starts like this:

# SLO: Checkout

Status: Agreed (2026-11) · Owner: Emre (Bookstore.Api) · Product: Selin · Next review: 2027-02

## Objectives (rolling 30 days)
| SLO | Target | Error budget | At 1,290 checkouts/day |
|---|---|---|---|
| Availability | 99.5 % | 0.5 % | about 194 failed checkouts, or 3.6 hours of full outage |
| Latency | 99 % under 800 ms | 1 % | about 387 checkouts slower than 800 ms |

## Not covered (known blind spots)
- Requests that never reach the API (DNS, load balancer, API process down). Planned: a synthetic checkout probe.
- Correctness of the order total and payment amount.
- The institutional bulk order import (a separate SLO is planned).

The "Not covered" section matters: in the previous post we measured that the SLI sees nothing while the API is down. Writing that into the document prevents the "the SLO was green but customers were complaining" surprise during an incident.

The error budget policy says in advance what happens when the budget runs out. The whole point is that the decision is made on a calm day, not under stress during an incident. The heart of Bookstore's policy is a table:

| Remaining budget (30 days) | Releases | Work |
|---|---|---|
| above 50 % | normal release cadence | normal |
| 0–50 % | normal, but risky changes (schema, payment flow) need a rollback plan in the PR | look at what spent the budget |
| below 0 (SLO missed) | only fixes for the SLO, security fixes and data fixes until back above 0 | the next sprint starts with reliability items |

Two more decisions are written in the policy. Errors caused by the payment provider count against the budget too, because customers do not care whose fault it was; but the work in that case is not "wait for the provider" but the timeout, the fallback and the message shown to the customer on our side. If a single incident spends more than 20 percent of the 30-day budget, a postmortem is mandatory; the example policy in the Workbook uses a similar threshold. Who decides in a disagreement is written down as well: Selin and Emre together, and the engineering manager if they cannot agree.

If you want to keep the SLO in a tool-independent format, there is OpenSLO, an open specification for defining SLOs in YAML. The same availability SLO in deploy/slo/checkout.openslo.yaml:

apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-availability
  displayName: Checkout availability
spec:
  description: Valid checkout requests (POST /api/orders, 4xx except 429 excluded) that did not fail on our side.
  service: bookstore-api
  indicator:
    metadata:
      name: checkout-availability-sli
    spec:
      ratioMetric:
        counter: true
        good:
          metricSource:
            type: Prometheus
            spec:
              query: sum(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/", http_request_method="POST", http_response_status_code=~"[23].."})
        total:
          metricSource:
            type: Prometheus
            spec:
              query: sum(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/", http_request_method="POST", http_response_status_code=~"[235]..|429"})
  timeWindow:
    - duration: 30d
      isRolling: true
  budgetingMethod: Occurrences
  objectives:
    - displayName: 99.5 percent of checkouts succeed
      target: 0.995

oslo validate -f deploy/slo/checkout.openslo.yaml
Valid!

Writing the target as a percentage (target: 99.5) is a common mistake, and oslo catches it:

Errors in bad.openslo.yaml:
  Validation for v1.SLO 'checkout-availability' has failed for the following properties:
    - 'spec.objectives[0].target' with value '99.5':
      - should be less than '1'
Error: Configuration is not valid!

Tool support is limited: Sloth 0.16, which we use in the next post, can read OpenSLO's older v1alpha version, and it rejected our openslo/v1 file with "could not load with any of the supported spec types". For a small team a Markdown document and recording rules are enough; OpenSLO pays off when the number of SLOs grows and several tools have to read the same definition.

Percentage SLOs at Low Traffic

The institutional order portal takes about 40 orders a day. What does the same target mean there?

dotnet run tools/slo-math/slo-math.cs -- 30 40 99.5 99

window: 30 days = 43,200 minutes, 1,200 requests
target     budget     downtime    per day  failed requests  per day
99.5%       0.50%        3.6 h    7.2 min                6      0.2
99%         1.00%        7.2 h   14.4 min               12      0.4

Six failed orders a month. A two-minute payment provider problem during a single lunch break can use up the budget; on the other hand, if the portal is completely down overnight while no requests arrive, the SLI does not see it. Computing a percentage from so little data is statistically meaningless. The options: generate synthetic traffic (a fake institutional order every minute), combine similar small services into one SLO, lengthen the window (90 days), or write the target as a count ("at most 3 failed orders a month, a postmortem for each").

In Production

Review the SLO regularly. Bookstore's document has the next review date written in it. Traffic, architecture and user expectations change; a target set a year ago may be too loose or too strict today.

Do not set the target above the measured value. Putting a 99.95 percent SLO on a service measured at 99.9 percent today puts the policy into "no releases" mode from day one. If a stricter target is wanted, start with an aspirational SLO.

Learn what your hard dependencies promise. If the payment provider's own SLA is 99.9 percent, targeting 99.95 percent for checkout only makes sense if you have a fallback that absorbs provider errors.

Talk about the budget every week. The remaining budget can be a one-line item in the on-call handoff meeting; it is the cheapest way to avoid a surprise at the end of the month.

Common Pitfalls

Computing the SLO with the new boundary on the day you add a bucket. As we saw above, the remaining budget comes out at -92, and if an alert is attached it fires for no reason.

A 100 percent or "as high as possible" target. A target without a budget removes the whole idea of the error budget; the SLO is missed at the first incident and nobody takes it seriously again.

An SLO without a policy. If what happens when the budget runs out is not written down, the decision goes to whoever talks loudest that day. The value of an SLO comes not from a chart but from the decision it is tied to.

Using the SLA as the SLO. When an SLO equal to the SLA is missed, you are already refunding customers; there is no room left for an early warning.

A Small Experiment

★☆☆ Use slo-math.cs to find how many failed orders a 99.9 percent target allows over a 90-day window at 1,290 checkouts a day. Compare it with the same target over 30 days; how does the longer window change the effect of a single incident?

★★☆ Write an SLO document for the search journey (GET /api/books): SLI, target, rationale and a "Not covered" section. Take a 10-minute measurement with the traffic tool before setting the target.

★★★ Change the latency rules in slo.yaml so that they count only from the moment the bucket was added (hint: leave the period without an le="0.8" series out of the denominator too). Compare the result with the value computed with le="1".

The budget is measured now, but someone has to watch it. In the next post we turn burn rate, the speed at which the budget is spent, and the Workbook's multiwindow, multi-burn-rate alerts into Prometheus rules; we break the payment provider and watch when the alert fires and when it goes quiet.