Game Days and Chaos Engineering: Rehearsing Incidents Before They Happen

Version note: The examples in this post were tried in the lab with .NET 10, Microsoft.Extensions.Http.Resilience 11.0.0 (with Polly.Core 8.4.2) and Prometheus 3.16.0. The game day is a lab rehearsal; nothing was run in production or against a real payment provider.

Guess First

What happens if the payment provider of Bookstore, the online bookstore used as the shared example across this site's series, starts answering 10 percent of requests with 503 one day? The team has three opinions. Emre: "Our checkout availability SLO is 99.5 percent; 10 percent errors burn the budget 20 times too fast, the 1-hour page fires within a few minutes." Ayşe: "It fires, but step 3 of the runbook shows the provider's 503s, and we find the cause in five minutes." Selin: "Are we going to learn this in a real outage?" In this post we test Emre's math, Ayşe's claim and Selin's question. A hint: where you inject the fault decides what becomes visible. Ayşe's claim did not hold in this post's experiment; guess why.

Chaos Engineering: An Experiment That Starts with a Hypothesis

Chaos engineering is gaining confidence, through experiments, that a system can withstand unexpected conditions in production. The Principles of Chaos Engineering text published at principlesofchaos.org sets it up like a scientific experiment: first define the system's normal behavior with a measurable output (the steady state; for example checkout success rate and latency), form a hypothesis that this behavior will continue during a failure, inject an event that can happen in real life (losing a server, a dependency slowing down, a network split) in a controlled way, and try to disprove the hypothesis. There are advanced principles too: build the hypothesis on the output users see rather than on internal metrics, run experiments on real traffic and automatically, and keep the blast radius, the users and systems an experiment can affect, small.

This shows that chaos engineering is not "breaking things at random": an experiment has a hypothesis, a measure and a stop condition. If the hypothesis holds, you now know the system withstands that failure; if it is disproved, you found a weakness, not a customer. A game day is doing these experiments as a team, in a session on the calendar: a scenario, people in their roles (the IC, ops lead, comms lead and scribe from the sixth post of this series) and the real tools. A game day tests not only the system but also people and process: does the alert fire, does the runbook lead to the right place, do the levers work, does communication flow?

Planning a Game Day

The most important part of a game day is one page written before it starts. Bookstore's first game day rehearses the payment provider returning errors, and its plan is in the repository as docs/gamedays/2026-11-payment-provider-errors.md:

## Steady state (measured before the start)
- Checkout availability (5m error ratio, `slo:checkout_availability:error_ratio_rate5m`) below 0.5 %.
- Checkout latency: at least 99 % under 800 ms.

## Hypotheses
1. With 10 % of payment calls answered 503 and 5 % delayed by 1 s, checkout fails about 10 % of the time and the
   1h CheckoutErrorBudgetBurn page fires within 5 minutes.
2. The on-call finds the cause with runbook step 3 within 5 minutes.
3. Turning chaos off (Chaos:Enabled=false in switches.json) stops the failures within a minute.

## Injection
`switches.json`: `"Chaos": { "Enabled": true, "FaultRate": 0.1, "LatencyRate": 0.05, "LatencyMs": 1000 }` for 6 minutes,
4 user journeys per second of synthetic traffic. Blast radius: the lab only; real customers never.

## Abort conditions (anyone may call them; the script also checks the first one every 15 s)
- 5m checkout error ratio above 20 %.
- Any page other than CheckoutErrorBudgetBurn.
- Anyone in the room is unsure what is happening.
Abort = `"Chaos": { "Enabled": false }` in switches.json; no restart needed.

The plan has four parts, and each answers a question. Steady state: what does "normal" look like, in numbers? Hypotheses: what do we think will happen, again in numbers and time? Injection: what exactly will we do, and whom can it affect? Abort conditions: what makes us stop, and how long does stopping take? Writing the abort condition in advance and letting anyone call it is critical; "let's wait a bit longer" during an experiment is the shortest way to turn a rehearsal into a real incident. The roles are the same as in the sixth post of the series: a game day is also a rehearsal of the incident process.

Fault Injection in .NET: Polly's Chaos Strategies

Where you inject the fault decides what you can test. The fake payment provider's fault settings (the /admin/faults we used in the earlier posts) imitate the provider side; the real provider in production has no such switch. Polly v8's chaos strategies, formerly known as Simmy, produce the fault in the application's own HTTP client: AddChaosLatency adds latency, AddChaosFault an exception, AddChaosOutcome a fake result (e.g. a 503) and AddChaosBehavior an arbitrary behavior. They are part of Polly.Core and have been there since 8.3.0; we added the Microsoft.Extensions.Http.Resilience 11.0.0 package to Bookstore (it brings Polly.Core 8.4.2). The detailed mechanics of the strategies and their use in tests are covered in the resilience testing post of the External Service Integration and Resilience series; here we focus on the three properties a game day needs:

// Fault injection for game days. Not even registered in Production; elsewhere it is off until
// Chaos:Enabled is set (in switches.json, so a game day starts and aborts without a restart).
bool chaosAllowed = !environment.IsProduction();
if (chaosAllowed)
{
    services.AddOptions<ChaosOptions>()
        .BindConfiguration(ChaosOptions.SectionName)
        .ValidateDataAnnotations()
        .ValidateOnStart();
    services.AddSingleton<LastKnownGood<ChaosOptions>>();
}

client.AddResilienceHandler("payments", (pipeline, context) =>
{
    pipeline.AddRetry(...);   // see below

    if (!chaosAllowed)
    {
        return;
    }
    // Chaos goes last (innermost), so the retry above sees injected faults like real ones.
    LastKnownGood<ChaosOptions> chaos = context.ServiceProvider.GetRequiredService<LastKnownGood<ChaosOptions>>();
    pipeline
        .AddChaosLatency(new ChaosLatencyStrategyOptions
        {
            EnabledGenerator = _ => ValueTask.FromResult(chaos.Value.Enabled),
            InjectionRateGenerator = _ => ValueTask.FromResult(chaos.Value.LatencyRate),
            LatencyGenerator = _ => ValueTask.FromResult(TimeSpan.FromMilliseconds(chaos.Value.LatencyMs))
        })
        .AddChaosOutcome(new ChaosOutcomeStrategyOptions<HttpResponseMessage>
        {
            EnabledGenerator = _ => ValueTask.FromResult(chaos.Value.Enabled),
            InjectionRateGenerator = _ => ValueTask.FromResult(chaos.Value.FaultRate),
            OutcomeGenerator = _ => ValueTask.FromResult<Outcome<HttpResponseMessage>?>(
                Outcome.FromResult(new HttpResponseMessage(HttpStatusCode.ServiceUnavailable)))
        });
});

First, chaos is never registered in Production; even if Enabled: true is left in the configuration by mistake, the production client's pipeline has no chaos strategy. An integration test pins this: with the same settings, every payment fails in Development and every payment goes through in Production. Second, switching on and off is read on every call through EnabledGenerator and InjectionRateGenerator; the values come from the switches.json of the sixth post through the LastKnownGood of the eighth, so a game day starts without a restart and an abort takes effect within seconds. Polly's defaults were changed on purpose here: according to the documentation, chaos strategies are on by default (Enabled true) with an InjectionRate of 0.001. Third, the rate: 10 percent errors and 5 percent latency is a level that resembles a real provider outage without stopping the system completely.

Game Day 1: Two Hypotheses Disproved

In the lab we ran the traffic tool that produces about 228 checkouts a minute, measured the steady state for three minutes and changed switches.json at 23:02:52 UTC. A small loop checked the plan's abort condition every 15 seconds at the same time (turn chaos off if the 5-minute error ratio goes above 20 percent). In this first run the pipeline consisted only of the two chaos strategies and was named payments-chaos; the retry below was added as a result of the game day. We ended the experiment at 23:08:23:

# 23:02:52 the game day starts
echo '{ "Payments": { "TimeoutSeconds": 3, "CheckoutEnabled": true }, "Chaos": { "Enabled": true, "FaultRate": 0.1, "LatencyRate": 0.05, "LatencyMs": 1000 } }' > switches.json
# 23:08:23 the game day ends (an abort is the same line)
echo '{ "Payments": { "TimeoutSeconds": 3, "CheckoutEnabled": true }, "Chaos": { "Enabled": false, "FaultRate": 0.1, "LatencyRate": 0.05, "LatencyMs": 1000 } }' > switches.json

The traffic tool's per-minute summary (search, cart and health columns removed):

23:02 UTC checkout=228 5xx=0 slowest-checkout=1632ms
23:03 UTC checkout=228 5xx=21 slowest-checkout=1138ms
23:04 UTC checkout=228 5xx=21 slowest-checkout=1516ms
23:07 UTC checkout=229 5xx=21 slowest-checkout=1139ms
23:08 UTC checkout=228 5xx=16 slowest-checkout=1124ms
23:09 UTC checkout=228 5xx=0 slowest-checkout=1514ms
23:10 UTC checkout=228 5xx=0 slowest-checkout=170ms

About 9 percent of checkouts got a 502; the 5-minute ratio read by the abort loop settled at 9.2 percent and the abort condition never triggered. The latency injection produced no errors (1 second is below the 3-second timeout) but hit the latency SLO: the share of checkouts under 800 ms dropped to 95.9 percent (target 99 percent). In the first full minute after chaos was turned off there were no 5xx; the abort took effect in under 30 seconds without a restart. The third hypothesis held. The first two did not.

First hypothesis: "the 1-hour page fires within 5 minutes." It did not; it never fired during the experiment. In the lab only the 6-hour page and the 3-day ticket fired, and only because we started Prometheus from scratch right before the experiment (the window held just a few minutes of data; explained in the fourth post of the series). What would happen in production? 9.3 percent errors is a burn rate of 18.6 for a 30-day 99.5 percent SLO. The threshold of the 1-hour window is 14.4; after a clean hour, the window's average reaches the threshold after 60 × 14.4 / 18.6 ≈ 46 minutes. So Emre's "a few minutes" is 46 minutes for a burn rate just above the threshold. This is not a bug but a deliberate consequence of the multiwindow design: a 9 percent error rate spends only 2.6 percent of the budget in an hour. But the team learned it in a game day, not on a real night; the answer to "how long until 10 percent errors page?" is now in the runbook.

Second hypothesis: "step 3 of the runbook shows the cause." It did not. Steps 2 and 3 of the runbook and Polly's own metric at the end of the experiment:

# step 2: checkout answers (per second)
sum by (http_response_status_code) (rate(http_server_request_duration_seconds_count{job="bookstore-api", http_route="/api/orders/"}[5m]))
{http_response_status_code="201"} 3.4455898555097857
{http_response_status_code="502"} 0.35438347801068065

# step 3: outgoing calls
sum by (server_address, http_response_status_code, error_type) (rate(http_client_request_duration_seconds_count{job="bookstore-api"}[5m]))
{http_response_status_code="200", server_address="127.0.0.1"} 3.4455898555097857
{http_response_status_code="200", server_address="localhost"} 0.06666619883369239

# Polly's events (last 5 minutes)
sum by (event_name, pipeline_name) (increase(resilience_polly_strategy_events_total[5m]))
{event_name="PipelineExecuting", pipeline_name="IPaymentGateway-payments-chaos"} 1139.99200005614
{event_name="Chaos.OnOutcome", pipeline_name="IPaymentGateway-payments-chaos"} 106.31504340320419
{event_name="Chaos.OnLatency", pipeline_name="IPaymentGateway-payments-chaos"} 43.15759187654823

# share of checkouts under 800 ms
{} 0.9593721144967683

0.35 checkouts per second get a 502, yet every call to the payment provider (127.0.0.1) is a 200. The alert's description said the same: "Checkout errors (5m): 9.326%. No failing outgoing calls." The runbook was sending the on-call to the "Neither" section, the "cause unknown, call the secondary" path. The reason is where the metric is measured: http.client.request.duration is recorded in the innermost (primary) handler, where the request really goes out to the network, and the observation shows exactly that; the chaos strategy produces the 503 in the resilience pipeline before the request ever leaves the process. Even more misleading, the HttpClient's own log (the outer logical handler) wrote "End processing HTTP request after 0.0825ms - 503"; anyone reading the log would have thought the provider returned 503. The only true signal was Polly's Chaos.OnOutcome event.

This finding is not specific to chaos. The same blind spot applies to every resilience mechanism that produces an answer before the request leaves the process: an open circuit breaker, a rate limiter, a retry that gives up. When Bookstore adds one of these later, step 3 of the runbook would mislead in the same way. The game day's first feedback went into the runbook:

3. Is it the payment provider? Bookstore.Api's calls to it, by host and result:
   ```promql
   sum by (server_address, http_response_status_code, error_type) (rate(http_client_request_duration_seconds_count{job="bookstore-api"}[5m]))
   ```
   503, or `error_type="System.Threading.Tasks.TaskCanceledException"` (timeout; such series have no status code)
   from the provider's host: go to "Payment provider".
   No failing outgoing calls, but checkouts fail? Our own resilience pipeline may be answering before the request
   leaves the process (a chaos experiment left on, a retry giving up, later a circuit breaker). Polly's events:
   ```promql
   sum by (event_name, pipeline_name) (rate(resilience_polly_strategy_events_total{job="bookstore-api"}[5m]))
   ```
   `Chaos.*` events outside a game day: set `Chaos:Enabled` to false in switches.json, then go on.

Feeding the Results Back

A game day is worth what it feeds back into the system. The first feedback was the runbook. The second was code: a payment provider returning an occasional 503 is common in real life, and since the second post of the series the Payments module deliberately had no retry (so that every provider failure shows up in the SLI). The game day measured the price of that decision: 10 percent provider errors mean more than 9 percent checkout errors. Since the Idempotency-Key header is already sent, a second attempt is safe; we added a retry to the pipeline, outside the chaos strategies:

// Game day 2026-11-14: one retry after a 503 or a broken connection (the Idempotency-Key header makes the
// second attempt safe). Timeouts are not retried: the per-call timeout already covers both attempts.
pipeline.AddRetry(new HttpRetryStrategyOptions
{
    MaxRetryAttempts = 1,
    Delay = TimeSpan.FromMilliseconds(200),
    BackoffType = DelayBackoffType.Constant,
    UseJitter = true,
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
        .HandleResult(response => response.StatusCode == HttpStatusCode.ServiceUnavailable)
});

A new integration test verifies that the payment goes through when the provider answers the first call with 503 and the second with 200, and that both calls carry the same Idempotency-Key; 19 of 19 tests pass. We played the same game day again with the same settings (23:16–23:21):

# step 2: checkout answers (per second)
{http_response_status_code="201"} 3.7683946077571386
{http_response_status_code="502"} 0.03508747307036442

# Polly's events (last 5 minutes)
{event_name="PipelineExecuting", pipeline_name="IPaymentGateway-payments"} 1139.99200005614
{event_name="Chaos.OnOutcome", pipeline_name="IPaymentGateway-payments"} 133.68327239808843
{event_name="OnRetry", pipeline_name="IPaymentGateway-payments"} 123.1570304769791
{event_name="Chaos.OnLatency", pipeline_name="IPaymentGateway-payments"} 70.52582087143247

# share of checkouts under 800 ms
{} 0.9345018450184501

Chaos produced 134 503s in 5 minutes; the retry kicked in 123 times and the checkout error rate fell from 9.3 percent to 0.9 percent (0.035 / 3.8 per second). The roughly 11 checkouts the retry could not save are calls that got a 503 on both attempts; 10 percent squared, the expected 1 percent. The burn rate fell from 18.6 to 1.8; no new page fired in the second run (the 6-hour page left over from the first run stopped at 23:18). But the retry has a price too: the share of checkouts under 800 ms fell from 95.9 to 93.5 percent, because every attempt is exposed to the latency injection separately and a 200 ms wait sits between them. The retry traded latency for availability; the latency SLO now burns faster. This shows why choosing the right number and timing of retries (the reliability patterns post of the System Design series) must be done by measuring.

The game day's plan file closed with the results and the action items; the four classes from the seventh post of the series are used here too:

## Action items
| ID | Action | Class | Owner | Due | Ticket | Status |
|---|---|---|---|---|---|---|
| GD-1 | Runbook step 3: look at Polly's resilience events when outgoing calls look healthy | detect | Emre | 2026-11-15 | BOOK-421 | done |
| GD-2 | One retry for a 503 from the provider (Idempotency-Key makes it safe) | mitigate | Ayse | 2026-11-15 | BOOK-422 | done |
| GD-3 | Error budget policy: write down how long a 10 % error rate takes to page (about 46 minutes) and decide if that is acceptable | process | Selin | 2026-11-28 | BOOK-423 | open |
| GD-4 | Repeat this game day in three months with a circuit breaker in the pipeline | process | Ayse | 2027-02-15 | BOOK-424 | open |

The third action item, GD-3, is not a code task but a decision: is it acceptable for Bookstore that a 10 percent error rate pages nobody for 46 minutes? If not, the answer is not a more sensitive alert (the fourth post of the series showed its noise cost) but a faster mitigation, such as the retry itself. The game day turned this question from an assumption into a measured number.

Rehearsals at the Infrastructure Level

Polly imitates a dependency failing from inside the application; some failures are outside it. Deleting a pod or draining a node in Kubernetes shows whether readiness probes, graceful shutdown and the PodDisruptionBudget really work (the health probe and graceful shutdown posts of the Kubernetes series). Lowering a pod's CPU or memory limit shows how the thread pool and the GC behave under pressure. Cutting a dependency's DNS record or network access in the lab shows whether the timeouts and error messages really are what you expect. There are ready-made tools for these, but the tool itself does not matter: each must be done with the same plan page (steady state, hypothesis, blast radius, abort).

The restore drill in the ninth post of the series is a kind of game day too: the hypothesis is "we can come back from a backup in 30 minutes", the injection "we lost the database", the measurement the restore time. Break-glass access and the dependency map must be rehearsed the same way. The "Wheel of Misfortune" described in Google's SRE book is the cheapest kind: an old postmortem becomes a tabletop game with one person in the on-call role.

When Not to Do It

Chaos engineering needs maturity. Injecting faults without SLOs and alerts is an experiment whose result you cannot measure: if you cannot define the steady state, you cannot verify the hypothesis. That is why this post is at the end of the series; the earlier posts built the SLI, SLO, alerts, runbook and incident process. Testing an alert that has no runbook with chaos only produces "yes, the alert fired and nobody knew what to do".

Chaos in production without approval and without abort conditions is never done. The Principles of Chaos Engineering text recommends that experiments eventually run in production on real traffic; but you get there with a small blast radius, with experiments proven in staging first, with the business side's approval (Selin, at Bookstore) and in periods the error budget allows. A service whose error budget is exhausted (according to the policy in the third post of the series) is already doing only reliability work; a chaos experiment there is only done to verify that reliability work. Polly's documentation also recommends narrowing the rate and the target (specific users, specific tenants) in production with InjectionRateGenerator and EnabledGenerator.

In Production

Chaos code ships with production code too. At Bookstore it is not registered in Production and a test pins that; a stricter option is to move the chaos package into a separate build configuration. Whichever you choose, prove the claim "chaos is off in production" with a test.

Put the game day on the calendar and follow up on its results. A one-off game day is a show; a game day repeated every three months that verifies the previous one's action items each time is a process.

Make the injection visible in telemetry. Polly's chaos events are counted in their own metric; if you do not add it to the dashboard and the runbook, the next game day, or a forgotten Enabled: true, will send you looking for the cause in the wrong place again.

Common Pitfalls

Chaos without a hypothesis. "Let's break something and see what happens" is not an experiment; you cannot learn from what you do not measure.

Not thinking about the abort. A chaos setup that needs a deploy to stop can turn a rehearsal into a real outage. Aborting must be a configuration change, and its duration must be measured.

Only asking "did the system survive?". In our experiment the real finding was not about the system but about what the runbook and the metrics did not show. Watch the people and the tools too.

Putting the retry inside the chaos. If the chaos strategy sits before (outside) the retry in the pipeline, the retry never sees the injected fault and the experiment does not represent a real failure. Polly's documentation also recommends making chaos the last, innermost strategy of the pipeline.

A Small Experiment

★☆☆ Write a game day plan page for your own service: define the steady state with an SLI, the hypothesis with a number and a time, and the abort condition with one sentence everyone understands.

★★☆ In Bookstore's lab, set LatencyRate to 0.2 and LatencyMs to 4000. Because the payment timeout is 3 seconds, the delays turn into timeouts; does the retry catch them? Why? What does step 3 of the runbook show this time?

★★★ Change EnabledGenerator so that chaos is only on for orders placed with a specific customer email (e.g. gameday@example.com; hint: put a property into the ResilienceContext). That limits the blast radius to one test customer; show in Prometheus that the SLO of the rest of the traffic is not affected.

This series started with what SRE is and came all the way here: from the user journey to the SLI, the SLO and error budget, burn rate alerts, on-call and runbooks, incident management, postmortems, other companies' major outages and finally a payment client broken on purpose. What they all have in common is that reliability is engineering work that can be measured, planned and rehearsed. The next step for the Bookstore team is to run this game day again in three months and see that the action items are really closed.