Real Incidents 1: Globally Propagated Config — Cloudflare 2025, Google Cloud 2025 and CrowdStrike 2024

Version note: The examples in this post were tried with .NET 10 and Prometheus 3.16.0 (including promtool). The summaries of the real incidents come only from the companies' official postmortems; nothing was run on those systems.

Guess First

In the sixth post of the series we added two levers to Bookstore.Api (Bookstore, the online bookstore used as the shared example across this site's series) for use during an incident: the payment timeout and a kill switch that turns checkout off. Both are read without a restart from a small switches.json file next to every instance. At midnight someone ships this file to all instances, but the file has a mistake. Guess three cases: (1) the file is valid JSON but TimeoutSeconds is 0; (2) the value is a string that is not a number, such as "6s"; (3) the file is cut in half and the JSON is broken. What do the running instances do in each case? What does an instance do that starts at that very moment (autoscaling, a deploy, a restart after a crash)? We measured the answers in the middle of the post.

The Common Pattern: Configuration, Not Code

This post and the next one learn from postmortems that large companies published themselves. What the three cases in this post have in common is that the outage was triggered not by a code deploy but by configuration or metadata: a file produced after a permission change, a policy record added to a database, a content update. In all three the change spread around the world within seconds or minutes, and in all three the staged rollout, testing and validation that existed for the code did not apply to this kind of data.

We roll out code deploys percentage by percentage with a canary, but configuration, feature files and policy records are often treated as "just data" and go everywhere at once. Yet for a running binary this data is an input that runs the code down a different path. The summaries below come only from the companies' official postmortems, in our own words.

Cloudflare, November 18, 2025

At 11:05 UTC Cloudflare made an access permission change on its ClickHouse database cluster: tables in the r0 database, which users could already reach implicitly, became explicitly visible. The change was rolled out to the cluster gradually. The query that produced the Bot Management feature file did not filter system.columns by database name; after the change the same columns came a second time, from r0, and the file doubled in size.

This file was regenerated every few minutes and distributed to the whole network. For performance, the Bot Management module in the proxy preallocates memory and set a limit of 200 features; normal use was around 60. The new file went over 200. In FL2, the new proxy written in Rust, that meant a panic, because the error returned when the limit was exceeded was unwrapped with unwrap(), and HTTP 5xx. In the old proxy, FL, every request got a bot score of 0 instead, and customers blocking bots blocked real users too.

What made the outage longer was that it fluctuated: the file was produced every five minutes on the gradually updated cluster and came out good or bad depending on which node the query hit. The network went down and came back, which first made the team think of a large DDoS attack. The status page, hosted entirely outside Cloudflare, going down at the same moment by coincidence strengthened that suspicion. At 13:05 a path that bypassed the proxy was opened for Workers KV and Access. At 14:24 generating and distributing new files was stopped, a known good file was put into the distribution queue and the proxy was restarted; at 14:30 core traffic was largely normal. All impact ended at 17:06. In its write-up Cloudflare calls it the worst outage it has had since 2019.

The fixes the company announced: validating its own configuration files like user input, more global kill switches for features, keeping error reports from exhausting system resources, and reviewing the failure modes of the proxy modules.

Google Cloud, June 12, 2025

On May 29 a new quota policy check was added to Service Control, the Google Cloud binary that checks authorization, policy and quota for API requests. The code was rolled out region by region, but the new code path only ran when a particular policy change arrived and never ran during the rollout. The code was not protected by a feature flag and its error handling was incomplete.

On June 12 around 17:45 UTC (the Google report gives Pacific times, 10:45 PDT) a new policy record was inserted into the Spanner tables. The record had fields that were left blank unintentionally. By design, policy data was copied to every region within seconds; Service Control in every region took the new path that read the blank field and crashed on a null pointer; the binaries went into a crash loop and external API requests got 503. Google's report sums it up in one sentence: "If this had been flag protected, the issue would have been caught in staging." (Google Cloud incident report, 2025).

Detection was fast: the SRE team found the cause within 10 minutes, prepared the "red button" that turned the new code path off, and rolled it out to every region within 40 minutes. But for the large region us-central1, recovery took up to 2 hours 40 minutes: Service Control tasks restarting at the same time hit the Spanner table as a herd and overloaded it, because there was no randomized exponential backoff. The impact lasted about 3 hours in total. The first customer notice was also about an hour late, because Cloud Service Health, where the notices are published, was affected too.

The fixes Google announced: splitting Service Control into modules that fail open on errors instead of rejecting requests, propagating globally replicated data incrementally and with validation, putting every change to critical binaries behind a feature flag that is off by default, auditing backoff, and keeping the communication infrastructure up when Google Cloud is down.

CrowdStrike, July 19, 2024

CrowdStrike's Falcon sensor receives content updates called "Rapid Response Content" independently of the sensor version. This content is interpreted by a Content Interpreter in the sensor according to predefined Template Types. The IPC Template Type that came with sensor 7.11 in February 2024 defined 21 input fields, but the sensor code that used this type supplied only 20 values. The mismatch went unnoticed for months, because the tests and the content until then had used a wildcard, which matches anything, in the 21st field.

Of the two new Template Instances published on July 19 at 04:09 UTC, one used a criterion other than a wildcard in the 21st field. A logic error in the Content Validator let this content through; when the sensor tried to read the 21st value it read past the end of the input array, an out-of-bounds read, and Windows crashed. CrowdStrike's root cause analysis puts it as "This parameter count mismatch evaded multiple layers of build validation and testing" (CrowdStrike, Channel File 291 RCA, August 2024). The content was withdrawn at 05:27 UTC, 78 minutes later; Windows machines that were online in that window crashed. Microsoft estimated the number of affected devices at about 8.5 million. Recovery took days: according to the RCA, about 99 percent of Windows sensors were online again only on July 29.

The fixes CrowdStrike announced: checking the field count at compile time and array bounds at runtime, testing every field with non-wildcard values, new Content Validator checks and, most importantly, staged deployment for Rapid Response Content: a canary first, then rings with bake time; customers also get to choose where and when content is deployed.

Side by side, the three cases show the common pattern:

The Same Mistake in Bookstore: switches.json

A small copy of the three cases exists in our own lab. The switches.json we added to Bookstore.Api in the sixth post of the series is a configuration file that reaches all instances at once and is read again at runtime. We tried the three cases of Guess First on a separate API instance: we sent the checkout endpoint a request with a cart ID that does not exist (with a valid configuration the answer is 404; with the kill switch on, 503), changed the file at every step and waited 3 seconds:

# switches.json changed at every step; a checkout request 3 seconds later (a cart that does not exist: 404 with a valid config)
{ "Payments": { "TimeoutSeconds": 3, "CheckoutEnabled": true } }          → 404
{ "Payments": { "TimeoutSeconds": 0, "CheckoutEnabled": true } }          → 500, 500
{ "Payments": { "TimeoutSeconds": 3, "CheckoutEnabled": true } }          → 404
{ "Payments": { "TimeoutSeconds": "6s", "CheckoutEnabled": true } }       → 500
{ "Payments": { "TimeoutSeconds": 6, "CheckoutEnabled": false } }         → 503
{ "Payments": { "TimeoutSeconds": 6, "CheckoutEnabled": fal               → 503
# at the last step the API wrote no log line at all

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Microsoft.Extensions.Options.OptionsValidationException: DataAnnotation validation failed for 'PaymentsOptions' members: 'TimeoutSeconds' with the error: 'The field TimeoutSeconds must be between 1 and 30.'.
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.InvalidOperationException: Failed to convert configuration value '6s' at 'Payments:TimeoutSeconds' to type 'System.Int32'.

The results show three different behaviors. When a value is invalid (0 or "6s"), every checkout got a 500 the moment the file changed. IOptionsMonitor.CurrentValue throws on every call when rebinding or validation fails; since the file reaches all instances at once, a single bad value breaks checkout on all of them at the same time. Cloudflare's file that went over the limit of 200 did exactly that. When the file was cut in half, the running instance did nothing: no log, no metric, the kill switch stayed on. That is not good either: you can no longer tell from the file which values the instances run with. And when we started a new instance with the same broken file, the process crashed at startup:

# A new instance with the same broken switches.json
dotnet Bookstore.Api.dll
Unhandled exception. System.IO.InvalidDataException: Failed to load configuration from file '/app/switches.json'.
 ---> System.FormatException: Could not parse the JSON file.
 ---> System.Text.Json.JsonReaderException: 'fal' is an invalid JSON literal. Expected the literal 'false'. LineNumber: 0 | BytePositionInLine: 59.
...
echo $?
134

Exit code 134. In Kubernetes that means a crash loop: every new pod started by a deploy, autoscaling or a node replacement reads the same file and crashes. The running instances quietly carry on with the old value while new ones cannot start; a system that cannot scale out when load rises and slowly shrinks.

Validating Configuration Like Input

Cloudflare's first fix was to validate its own configuration files like user input. In Bookstore we do it in two places: before the file is shipped and inside the instance. The first is a small check that runs in the pipeline: tools/switches-check/switches-check.cs reads the file with a size limit, parses it with a setting that rejects duplicate keys (AllowDuplicateProperties in .NET 10), treats unknown keys as errors and checks the type and range of every value:

const int MaxBytes = 4096;
string path = args.Length > 0 ? args[0] : "switches.json";
List<string> problems = [];

byte[] bytes = File.ReadAllBytes(path);
if (bytes.Length > MaxBytes)
{
    problems.Add($"file is {bytes.Length} bytes, the limit is {MaxBytes}");
}

try
{
    using JsonDocument doc = JsonDocument.Parse(bytes, new JsonDocumentOptions { AllowDuplicateProperties = false });
    CheckObject(doc.RootElement, "", new()
    {
        ["Payments"] = payments => CheckObject(payments, "Payments", new()
        {
            ["TimeoutSeconds"] = v => CheckInt(v, "Payments:TimeoutSeconds", 1, 30),
            ["CheckoutEnabled"] = v => CheckBool(v, "Payments:CheckoutEnabled")
        })
    });
}
catch (JsonException ex)
{
    problems.Add($"not valid JSON: {ex.Message}");
}
...
void CheckObject(JsonElement element, string name, Dictionary<string, Action<JsonElement>> known)
{
    if (element.ValueKind != JsonValueKind.Object)
    {
        problems.Add($"'{name}' must be an object");
        return;
    }
    foreach (JsonProperty property in element.EnumerateObject())
    {
        // Unknown keys are errors: a typo ("CheckoutEnable") would otherwise be ignored silently.
        if (known.TryGetValue(property.Name, out Action<JsonElement>? check))
        {
            check(property.Value);
        }
        else
        {
            problems.Add($"unknown key '{(name.Length > 0 ? name + ":" : "")}{property.Name}'");
        }
    }
}

Treating unknown keys as errors matters: a typo such as CheckoutEnable is silently ignored by .NET configuration and the kill switch never turns on. Run on four files:

dotnet run tools/switches-check/switches-check.cs -- switches.json
OK switches.json

dotnet run tools/switches-check/switches-check.cs -- switches-typo.json
ERROR switches-typo.json: 'Payments:TimeoutSeconds' must be a whole number, got "6s"
ERROR switches-typo.json: unknown key 'Payments:CheckoutEnable'
2 problem(s)

dotnet run tools/switches-check/switches-check.cs -- switches-duplicate.json
ERROR switches-duplicate.json: not valid JSON: Duplicate property 'TimeoutSeconds' encountered during deserialization.
1 problem(s)

dotnet run tools/switches-check/switches-check.cs -- switches-truncated.json
ERROR switches-truncated.json: not valid JSON: 'fal' is an invalid JSON literal. Expected the literal 'false'. LineNumber: 0 | BytePositionInLine: 59.
1 problem(s)

For the bad files the exit code is 1; the pipeline that ships the file to the instances stops there. Rejecting duplicate keys is deliberate too: Cloudflare's file grew because the same columns came a second time.

The pipeline check alone is not enough; the file can also be changed by hand. The second defense is inside the instance. LastKnownGood<T> reads from IOptionsMonitor, and when binding or validation fails it returns the last valid value, logs the error once and increments a metric:

// Reads options that can change at runtime and keeps the last valid value when a change does not bind or validate.
// Without it, IOptionsMonitor.CurrentValue throws on every call until someone fixes the file.
public sealed class LastKnownGood<T>(IOptionsMonitor<T> monitor, ILogger<LastKnownGood<T>> logger) where T : class
{
    private T current = monitor.CurrentValue;   // validated at startup (ValidateOnStart)
    private string? lastError;

    public T Value
    {
        get
        {
            try
            {
                current = monitor.CurrentValue;
                lastError = null;
            }
            catch (Exception ex) when (ex is OptionsValidationException or InvalidOperationException)
            {
                // Log and count once per distinct problem, not on every request.
                if (ex.Message != lastError)
                {
                    lastError = ex.Message;
                    SwitchesFile.Rejected.Add(1, new KeyValuePair<string, object?>("reason", "invalid"));
                    logger.LogError(ex, "Rejected new {Options} values; keeping the last valid ones", typeof(T).Name);
                }
            }
            return current;
        }
    }
}

Broken JSON needs a decision that depends on the phase the application is in. While running, a broken file was already ignored (the old values stayed); we keep that and add a metric. At startup we choose to ignore the file and start with the appsettings values instead of a crash loop:

public static WebApplicationBuilder AddSwitchesFile(this WebApplicationBuilder builder)
{
    string path = builder.Configuration["SwitchesFile"] ?? "switches.json";
    builder.Configuration.AddJsonFile(source =>
    {
        source.Path = path;
        source.Optional = true;
        source.ReloadOnChange = true;
        source.OnLoadException = context =>
        {
            Rejected.Add(1, new KeyValuePair<string, object?>("reason", "parse"));
            // While starting: go on with the appsettings values (fail open) instead of a crash loop.
            // While running: do not ignore it; the reload is dropped and the last values stay in effect.
            if (!started)
            {
                context.Ignore = true;
                Console.Error.WriteLine($"switches.json ignored at startup: {context.Exception.GetBaseException().Message}");
            }
        };
        source.ResolveFileProvider();
    });
    return builder;
}

Program.cs calls builder.AddSwitchesFile(), and once the application has started, app.Lifetime.ApplicationStarted.Register(SwitchesFile.MarkStarted) sets the "we are running now" flag. The payment client and OrderService now take LastKnownGood<PaymentsOptions> instead of IOptionsMonitor<PaymentsOptions>. We repeated the experiment in the same order:

{ "Payments": { "TimeoutSeconds": 3, "CheckoutEnabled": true } }          → 404
{ "Payments": { "TimeoutSeconds": 0, "CheckoutEnabled": true } }          → 404
{ "Payments": { "TimeoutSeconds": "6s", "CheckoutEnabled": true } }       → 404
{ "Payments": { "TimeoutSeconds": 6, "CheckoutEnabled": false } }         → 503
{ "Payments": { "TimeoutSeconds": 6, "CheckoutEnabled": fal               → 503
# we started a new instance with the same broken file; it started and answered the checkout:
new instance                                                              → 404

Invalid values no longer break checkout; the instance runs with the last valid value and logs the error once:

fail: Bookstore.Configuration.LastKnownGood[0]
      Rejected new PaymentsOptions values; keeping the last valid ones
      Microsoft.Extensions.Options.OptionsValidationException: DataAnnotation validation failed for 'PaymentsOptions' members: 'TimeoutSeconds' with the error: 'The field TimeoutSeconds must be between 1 and 30.'.
fail: Bookstore.Configuration.LastKnownGood[0]
      Rejected new PaymentsOptions values; keeping the last valid ones
      System.InvalidOperationException: Failed to convert configuration value '6s' at 'Payments:TimeoutSeconds' to type 'System.Int32'.

The new instance started with the broken file no longer crashes either. The counter shows up in Prometheus, and the SwitchesFileRejected alert in deploy/prometheus/rules/config-alerts.yaml watches it and opens a ticket (in the repository together with its promtool test):

bookstore_switches_rejected_total

{__name__="bookstore_switches_rejected_total", job="bookstore-api", reason="invalid", ...} 2
{__name__="bookstore_switches_rejected_total", job="bookstore-api", reason="parse", ...} 1

# switches.json changes that an instance rejected (bad JSON, wrong type, value out of range).
# The instance keeps its last valid values, so nothing breaks; but the lever someone just pulled did not happen.
groups:
  - name: bookstore-config
    rules:
      - alert: SwitchesFileRejected
        expr: sum by (reason) (increase(bookstore_switches_rejected_total{job="bookstore-api"}[10m])) > 0
        labels:
          severity: ticket
        annotations:
          summary: "An instance rejected a switches.json change ({{ $labels.reason }}); it still runs with the previous values"
          runbook_url: https://git.example.com/bookstore/blob/main/docs/incidents/levers.md

The rejection at startup is missing from the counter: when OnLoadException runs, OpenTelemetry's metric pipeline does not exist yet. That is why the startup rejection is also written to stderr (switches.json ignored at startup: 'fal' is an invalid JSON literal…); you have to look at the log to catch it.

Fail Open or Fail Closed?

Starting the new instance with a broken file is a decision: we chose to fail open, carrying on with the default, permissive behavior when something fails. The price is visible in the experiment: while the old instance keeps the kill switch on (503), the new instance accepts the checkout (404, meaning it looks for the cart). Two different behaviors at the same time. That is acceptable for this file, because the defaults are the normal operating values and the worst outcome is the kill switch being off on new instances; the alert reports that too.

But it is not right for every configuration. For security-related configuration such as authorization rules, rate limits or payment amount checks, answering a broken file with "allow everything" turns an outage into a vulnerability; failing closed, rejecting when something fails, is right there. Make the decision for every configuration file in advance and in writing, not during an incident. Bookstore wrote it into docs/incidents/levers.md.

The second lesson is staged deployment. Google announced propagating globally replicated data region by region with validation, CrowdStrike a canary and rings for content updates. At Bookstore's scale the equivalent is simple: switches.json goes to one instance first, checkout status codes are watched for a minute, then it goes to the rest. The setup you built for code canaries (the canary release post of the Deployment Strategies series) can be used for configuration too; how ConfigMap changes reach pods is covered in the "ConfigMap and Secret with .NET Configuration" post of the Kubernetes series.

The third lesson is how limits behave: Cloudflare's limit of 200 features was not wrong, the problem was the proxy stopping with a panic when the limit was exceeded. When a limit is exceeded, reject the new input, carry on with the old one and report it loudly; that is what LastKnownGood does. The fourth is the communication infrastructure: at Cloudflare the failing status page fed the attack suspicion, at Google the first notice was an hour late. Bookstore's status page and incident channel must not depend on Bookstore's infrastructure.

In Production

Make visible which configuration the instances run with. When a bad change is rejected, instances can run with different values; publishing a summary of the effective values (a hash, for example) through a metric or a health endpoint shows which instance is behind.

Kill switches are configuration themselves. That does not contradict Cloudflare's "more global kill switches" fix: a kill switch must be validated too, and what happens when it is broken must be decided in advance.

Common Pitfalls

Treating configuration as "data" and not testing it. Every input that changes the code path needs as much testing as code. In the CrowdStrike case the tests always used a wildcard for the 21st field; when a real value arrived, a path that had never been tried ran.

Not knowing the defaults. That IOptionsMonitor throws on every call for an invalid value, while broken JSON is silently ignored at runtime and crashes the process at startup, is something we saw by trying it, not in the documentation. Learn by measuring what your configuration library does in these three cases.

Retries without backoff. The late recovery of us-central1 at Google came from tasks restarting at the same time and overloading the shared database. Restarts and retries need a randomized, exponentially growing wait.

A Small Experiment

★☆☆ Add a rule to switches-check.cs: when CheckoutEnabled is false, a Reason field is required in the file. Note that the application does not know this field; .NET configuration ignores unknown keys, but the check tool must now know it.

★★☆ Add a feature to LastKnownGood<T> that publishes a hash of the effective values as a metric (e.g. bookstore_switches_info{hash="…"}), run two instances with different files and show the difference in Prometheus.

★★★ Read the postmortem of one of the three cases in this post from start to finish and sort the fixes the company announced into the four classes from the seventh post of the series (prevent, detect, mitigate, process). Which class is missing?

In the three outages of this post the trigger was a data change. In the next post we look at another family: automation's own failures, hidden dependencies and recovery paths that depend on the very thing that broke, in the AWS October 2025, Facebook October 2021, GitLab January 2017 and Knight Capital August 2012 cases; and a restore drill for Bookstore.