Trade-off Analysis: Comparing Options Against Scenarios, Sensitivity and Trade-off Points

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

In the planning meeting for campaign week, Grace brings a proposal: "Let's put Redis in front of catalog search." Linus asks "who is going to run Redis?", Alan says "HybridCache is already in the framework, why a new container?". The debate has been going around for half an hour and everyone seems right. Ada stops it and proposes a spike: try the three options on the same traffic, with two application instances. We simulate the database (every search query takes 40 ms), while Redis is real, in a local Docker container. The spike's main loop:

using System.Diagnostics;
using System.Globalization;
using System.Text.Json;
using Bookstore.Catalog;
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.Hybrid;
using Microsoft.Extensions.DependencyInjection;

CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture;
string mode = args.FirstOrDefault() ?? "spike";

if (mode == "spike")
{
    // Fixed-seed traffic that looks like a campaign: 2,000 searches to warm up, then 2,000 measured searches
    string[] warmUp = Requests(seed: 42), measured = Requests(seed: 43);
    foreach (string option in new[] { "no-cache", "hybrid", "redis", "hybrid+l2" })
    {
        CatalogDb db = new();
        string run = Guid.NewGuid().ToString("N");
        ICatalogSearch[] instances = [Build(option, db, "localhost:6390", run), Build(option, db, "localhost:6390", run)];
        for (int i = 0; i < warmUp.Length; i++)
            await instances[i % 2].SearchAsync(warmUp[i]); // requests alternate between the two instances

        int before = db.Queries;
        List<double> ms = [];
        for (int i = 0; i < measured.Length; i++)
        {
            Stopwatch watch = Stopwatch.StartNew();
            await instances[i % 2].SearchAsync(measured[i]);
            ms.Add(watch.Elapsed.TotalMilliseconds);
        }
        int queries = db.Queries - before;

        // The price changed: instance 0 clears its cache, what does instance 1 see?
        string hot = measured[0];
        await instances[1].SearchAsync(hot);
        db.ChangePrice(hot, 21.99m);
        await instances[0].InvalidateAsync(hot);
        SearchResult seenByOther = await instances[1].SearchAsync(hot);

        Console.WriteLine($"{option,-9} p50={P(ms, 50),5:F2} ms  p95={P(ms, 95),5:F2} ms  " +
            $"db queries={queries,4}  other instance price={seenByOther.Price}");
    }
}

The helper methods that build the options, and the fixed-seed Zipf distribution:

static string[] Requests(int seed)
{
    Random random = new(seed);
    return [.. Enumerable.Range(0, 2_000).Select(_ => $"term-{Zipf(random, 1_000)}")];
}

static ICatalogSearch Build(string option, CatalogDb db, string redis, string run)
{
    ServiceCollection services = new();
    if (option is "redis" or "hybrid+l2")
        services.AddStackExchangeRedisCache(o => { o.Configuration = redis; o.InstanceName = run; });
    if (option is "hybrid" or "hybrid+l2")
        services.AddHybridCache(o => o.DefaultEntryOptions = new() { Expiration = TimeSpan.FromMinutes(5) });
    ServiceProvider sp = services.BuildServiceProvider();

    return option switch
    {
        "no-cache" => new DbSearch(db),
        "redis" => new RedisSearch(db, sp.GetRequiredService<IDistributedCache>()),
        _ => new HybridSearch(db, sp.GetRequiredService<HybridCache>()),
    };
}

static int Zipf(Random random, int n)
{
    // Simple Zipf (s = 1): popular terms are searched much more often
    double h = 0;
    for (int k = 1; k <= n; k++) h += 1.0 / k;
    double u = random.NextDouble() * h, sum = 0;
    for (int k = 1; k <= n; k++)
    {
        sum += 1.0 / k;
        if (sum >= u) return k;
    }
    return n;
}

static double P(List<double> values, int percentile)
{
    List<double> sorted = [.. values.Order()];
    return sorted[(int)Math.Ceiling(percentile / 100.0 * sorted.Count) - 1];
}

And the three search implementations:

namespace Bookstore.Catalog
{
    public sealed record SearchResult(string Term, int Hits, decimal Price);

    public sealed class CatalogDb
    {
        private readonly Dictionary<string, decimal> _prices = [];
        private int _queries;

        public int Queries => _queries;

        public async Task<SearchResult> SearchAsync(string term, CancellationToken ct = default)
        {
            Interlocked.Increment(ref _queries);
            await Task.Delay(40, ct); // simulation standing in for the full-text search query
            return new SearchResult(term, term.Length * 7, _prices.GetValueOrDefault(term, 18.49m));
        }

        public void ChangePrice(string term, decimal price) => _prices[term] = price;
    }

    public interface ICatalogSearch
    {
        Task<SearchResult> SearchAsync(string term, CancellationToken ct = default);
        Task InvalidateAsync(string term, CancellationToken ct = default);
    }

    public sealed class DbSearch(CatalogDb db) : ICatalogSearch
    {
        public Task<SearchResult> SearchAsync(string term, CancellationToken ct = default) => db.SearchAsync(term, ct);
        public Task InvalidateAsync(string term, CancellationToken ct = default) => Task.CompletedTask;
    }

    public sealed class HybridSearch(CatalogDb db, HybridCache cache) : ICatalogSearch
    {
        public async Task<SearchResult> SearchAsync(string term, CancellationToken ct = default) =>
            await cache.GetOrCreateAsync($"search:{term}", async token => await db.SearchAsync(term, token), cancellationToken: ct);

        public async Task InvalidateAsync(string term, CancellationToken ct = default) =>
            await cache.RemoveAsync($"search:{term}", ct);
    }

    public sealed class RedisSearch(CatalogDb db, IDistributedCache cache) : ICatalogSearch
    {
        public async Task<SearchResult> SearchAsync(string term, CancellationToken ct = default)
        {
            byte[]? cached = await cache.GetAsync($"search:{term}", ct);
            if (cached is not null)
                return JsonSerializer.Deserialize<SearchResult>(cached)!;

            SearchResult result = await db.SearchAsync(term, ct);
            await cache.SetAsync($"search:{term}", JsonSerializer.SerializeToUtf8Bytes(result),
                new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5) }, ct);
            return result;
        }

        public async Task InvalidateAsync(string term, CancellationToken ct = default) =>
            await cache.RemoveAsync($"search:{term}", ct);
    }
}

Four questions. How much does p95 drop with the cached options? Which option sends the fewest queries to the database? After the price changes and instance 0 clears its cache, which price does instance 1 show? Finally, does hybrid+l2, putting Redis behind HybridCache as the second-level cache, solve the price problem?

The Results of the Spike

no-cache  p50=42.06 ms  p95=42.10 ms  db queries=2000  other instance price=21.99
hybrid    p50= 0.00 ms  p95=42.09 ms  db queries= 324  other instance price=18.49
redis     p50= 0.31 ms  p95=42.79 ms  db queries= 192  other instance price=21.99
hybrid+l2 p50= 0.00 ms  p95=42.47 ms  db queries= 193  other instance price=18.49

First, p95 did not drop for any option. It is around 42 ms for all three, which is the database time. The reason is simple: even after the warm-up, 16% of the searches with HybridCache and 10% with Redis still go to the database. When the miss ratio is above 5%, the slowest 5% of requests consists entirely of misses and p95 equals the database time. The cache brought p50 down to almost zero, but what saves the "p95 < 300 ms at campaign peak" scenario is not latency but database load: 324 or 192 queries for 2,000 searches instead of 2,000. In the third Introduction to Architecture post we saw why the average misleads; here the same lesson applies to p50.

Second, the price. Instance 0 cleared its own cache, but instance 1 keeps showing the old price (18.49); HybridCache's L1 (first level, the in-process cache) lives in each instance's own memory. The real surprise is the last line: adding Redis behind HybridCache as L2 (second level, the shared cache) brought the database load down to the Redis level (193), but the stale price problem stayed exactly the same. RemoveAsync can clear the L1 of the instance it is called on and the L2, but it does not reach the memory of the other instance. We learned that from the spike, not from the documentation.

Third, what if Redis itself goes down? The program's third mode tries the same search with the container stopped:

else
{
    // With the Redis container stopped: three searches in a row, with defaults and with short timeouts
    foreach (string redis in new[] { "localhost:6390", "localhost:6390,abortConnect=false,connectTimeout=250,asyncTimeout=250" })
    foreach (string option in new[] { "redis", "hybrid+l2" })
    {
        ICatalogSearch search = Build(option, new CatalogDb(), redis, "down");
        List<string> calls = [];
        for (int i = 1; i <= 3; i++)
        {
            Stopwatch watch = Stopwatch.StartNew();
            try
            {
                await search.SearchAsync($"term-{i}");
                calls.Add($"ok {watch.ElapsedMilliseconds} ms");
            }
            catch (Exception ex)
            {
                calls.Add($"{ex.GetType().Name} {watch.ElapsedMilliseconds} ms");
            }
        }
        Console.WriteLine($"{option,-9} [{redis}] {string.Join(" | ", calls)}");
    }
}

redis     [localhost:6390] RedisConnectionException 6037 ms | RedisConnectionException 5992 ms | RedisConnectionException 6002 ms
hybrid+l2 [localhost:6390] ok 6075 ms | ok 5967 ms | ok 6000 ms
redis     [localhost:6390,abortConnect=false,connectTimeout=250,asyncTimeout=250] RedisConnectionException 5007 ms | RedisConnectionException 5998 ms | RedisConnectionException 5000 ms
hybrid+l2 [localhost:6390,abortConnect=false,connectTimeout=250,asyncTimeout=250] ok 5044 ms | ok 5999 ms | ok 6001 ms

With the Redis-only option every search waits about 5–6 seconds and fails with RedisConnectionException. HybridCache swallows the L2 error and answers from the database, but it also waits the same amount of time on every request. Shortening the connection settings did not change the duration on this machine; we did not investigate why within the scope of this post, but the outcome does not change: the Redis option turns a Redis outage on campaign day into a search outage. The last mode of the same spike measures the effect of the instance count:

else if (mode == "scale")
{
    // The load an in-process cache leaves on the database as the instance count grows
    string[] warmUp = Requests(seed: 42), measured = Requests(seed: 43);
    foreach (int count in new[] { 1, 2, 4, 8 })
    {
        CatalogDb db = new();
        ICatalogSearch[] instances = [.. Enumerable.Range(0, count).Select(_ => Build("hybrid", db, "", ""))];
        for (int i = 0; i < warmUp.Length; i++)
            await instances[i % count].SearchAsync(warmUp[i]);
        int before = db.Queries;
        for (int i = 0; i < measured.Length; i++)
            await instances[i % count].SearchAsync(measured[i]);
        Console.WriteLine($"hybrid x{count}  db queries={db.Queries - before,4}");
    }
}

hybrid x1  db queries= 192
hybrid x2  db queries= 324
hybrid x4  db queries= 474
hybrid x8  db queries= 647

Weighing Options Against Scenarios

In the third Introduction to Architecture post we wrote quality attribute scenarios and prioritized them with a utility tree. In this post we use them as a decision tool: evaluate each option, one by one, against the prioritized scenarios it affects. "Which one is better?" has no answer; "which one meets the search scenario at campaign peak, which one breaks the price scenario?" has a measurable one. Bookstore has four scenarios for this decision: database load at campaign peak (from the search scenario in the utility tree), a changed price being visible everywhere within 60 seconds (a new scenario written with the product team for this decision), search continuing to work when a cache component fails, and an operational load that one part-time ops engineer can carry.

The matrix does not name a "winner"; it shows which scenario each option meets and at what cost. No cache is consistent and simple, but it does not carry the campaign load. Redis carries the load best and is consistent, but it brings a new failure mode and new operational work. HybridCache is free in terms of failures and operations, but it breaks the price scenario, and the load it leaves on the database grows with the instance count. The letters S, T and R in the cells are the topic of the next section.

Four Concepts from ATAM

ATAM (Architecture Tradeoff Analysis Method) is a nine-step architecture evaluation method that runs for several days with stakeholders, defined by Rick Kazman, Mark Klein and Paul Clements in a 2000 SEI (Software Engineering Institute) technical report (CMU/SEI-2000-TR-004). Applying all of it in a small team is not realistic, but four concepts from the report carry over to any trade-off conversation.

Sensitivity point: an architectural property that a quality attribute response depends on critically. The report's example is the confidentiality of a VPN (Virtual Private Network) being sensitive to the number of encryption bits. For us, with the HybridCache option the database load is sensitive to the instance count: 192, 324, 474, 647 queries for 1, 2, 4, 8 instances. Sensitivity points are yellow flags: "be careful when you change this parameter."

Trade-off point: a property that is a sensitivity point for more than one quality attribute at once and moves them in opposite directions. The report's example is encryption again: raising its level increases security and processing time. For us, it is the expiration time of a cache entry (TTL, time to live): making it longer reduces database load and lengthens the time an old price can be shown. The report calls trade-off points the most critical decisions in an architecture; that is why the TTL value should be written down in the ADR (Architecture Decision Record) with its rationale, not left in the code as a magic number.

Risk and non-risk. A risk is a potentially problematic architecture decision, or one that has not been made yet. A non-risk is a good decision that usually rests on implicit assumptions. Our risk: "how long the client waits and what it does when Redis is unreachable is not defined." Our non-risk: "a 60-second-old price is acceptable, because checkout always reads the price from the database." The report gives an important warning: a non-risk stays a non-risk only as long as its assumptions do not change. If someone starts using the cached price in checkout too, the same decision becomes a risk. That is why we write the non-risk down together with its assumption.

Risk theme. In ATAM's last step the risks are grouped under common themes, and each theme shows which business goal it threatens. The SEI's 2006 report "Risk Themes Discovered Through Architecture Evaluations" (CMU/SEI-2006-TR-012) looked at 18 evaluations and reached an interesting conclusion: roughly twice as many of the themes found were of the "omission" kind, coming from decisions that had not been made. Our Redis risk is exactly that, and it is not alone: there is no written timeout and fallback policy for the payment provider either. Both fall under the same theme: "the timeout behaviour of external dependencies is undefined." Individual risks give you a list of decisions; a theme shows you an architectural gap.

A Two-Hour Lightweight ATAM

The Bookstore team weighed this decision in one afternoon, in a two-hour session. The agenda was derived from ATAM's steps, but instead of a stakeholder workshop and scenario voting, the team's already written utility tree was used:

• 0–15 min: business goal and prioritized scenarios. Only the (H, *) scenarios from the utility tree that this decision touches are read; a missing scenario (for us, price freshness) is written here with the product team.
• 15–35 min: options. Each option is presented in five minutes by the person proposing it; the "do nothing" option is always on the list.
• 35–85 min: option × scenario walkthrough. For each cell the question is "does this option meet this scenario, and based on what?"; if the answer is a measurement it comes from the spike, if it is a guess it is marked "guess". Sensitivity points, trade-off points, risks and non-risks are noted along the way.
• 85–105 min: risks are grouped and themes are drawn out. For each theme an owner and a next step (a spike, a separate ADR, a test) are written down.
• 105–120 min: the decision, or why it cannot be made yet. If it is made, someone is assigned to draft the ADR; if not, the missing information and the spike that will bring it are defined.

The output of the session is not a report; it is notes that go directly into the Decision Drivers, Pros and Cons and More Information sections of the ADR. For anyone curious about the full ATAM, the order of its steps and its stakeholder roles, the SEI report is a good place to start.

The False Precision of a Weighted Decision Matrix

In the middle of the session Alan made the classic proposal: let's weight the criteria, score the options from 1 to 5, and the highest score wins. The team gave the scores while looking at the spike results. To see how robust the result was, we wrote a small program that moves every weight and every score by one step:

using System.Globalization;

CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture;

string[] criteria = ["db load", "p50 latency", "price freshness", "cache outage", "operations"];
int[] weights = [5, 3, 4, 4, 3];

// Scores from 1 to 5, given in a team meeting after looking at the spike results
Dictionary<string, int[]> scores = new()
{
    ["no-cache"] = [1, 1, 5, 5, 5],
    ["hybrid"] = [4, 5, 2, 5, 4],
    ["redis"] = [5, 4, 5, 2, 2],
};
string baseline = Winner(scores, weights);
Print("baseline", scores, weights);

// Move each weight by one step: does the winner change?
int weightFlips = 0, weightVariants = 0;
foreach ((int i, int[] w) in OneStep(weights))
{
    weightVariants++;
    if (Winner(scores, w) != baseline) weightFlips++;
}
Console.WriteLine($"weights: winner changed in {weightFlips} of {weightVariants} one-step changes");

// Move each score by one step: does the winner change?
int scoreVariants = 0;
foreach (string option in scores.Keys)
{
    foreach ((int i, int[] s) in OneStep(scores[option]))
    {
        scoreVariants++;
        Dictionary<string, int[]> changed = new(scores) { [option] = s };
        if (Winner(changed, weights) != baseline)
            Print($"{option}.{criteria[i]} {scores[option][i]}->{s[i]}", changed, weights);
    }
}
Console.WriteLine($"scores: {scoreVariants} one-step changes tried");

static IEnumerable<(int Index, int[] Values)> OneStep(int[] values)
{
    for (int i = 0; i < values.Length; i++)
        foreach (int delta in new[] { -1, 1 })
        {
            int v = values[i] + delta;
            if (v is < 1 or > 5) continue;
            int[] copy = [.. values];
            copy[i] = v;
            yield return (i, copy);
        }
}

static string Winner(Dictionary<string, int[]> s, int[] w) => s.MaxBy(x => Score(x.Value, w)).Key;

static double Score(int[] s, int[] w) => (double)s.Zip(w, (a, b) => a * b).Sum() / w.Sum();

static void Print(string label, Dictionary<string, int[]> s, int[] w) =>
    Console.WriteLine($"{label,-28} " + string.Join("  ", s.Select(x => $"{x.Key}={Score(x.Value, w):F2}")) + $"  -> {Winner(s, w)}");

baseline                     no-cache=3.32  hybrid=3.95  redis=3.74  -> hybrid
weights: winner changed in 0 of 9 one-step changes
hybrid.db load 4->3          no-cache=3.32  hybrid=3.68  redis=3.74  -> redis
scores: 21 one-step changes tried

HybridCache wins with 3.95, Redis has 3.74. Two decimal places give an impression of precision, but the scores are between 1 and 5, half of them are based on guesses, and the gap is smaller than a single score step. With respect to the weights the result is robust: none of the nine changes changes the winner. For the scores, one cell is enough: if HybridCache's "db load" score were 3 instead of 4, Redis would win. That cell is exactly what the spike measured: is an 84% reduction a 4 or a 3? With two instances it is easy to say 4; with eight instances and 647 queries it is just as easy to say 3.

We do not throw the matrix away, but what we take from it is a location, not a number: the cell that changes the winner is where the measuring effort should go. The score itself does not go into the ADR; what goes in is the assumption the decision is sensitive to ("revisit if the instance count goes above 4").

Carrying the Result into the ADR

At the end of the session the decision was made, and Grace took on writing the ADR even though her proposal was not the one chosen. The spike and the matrix entered the relevant sections of the ADR like this (we show only the Redis part of Pros and Cons):

---
status: accepted
date: 2026-06-10
decision-makers: Grace, Ada
consulted: Linus (ops), Catalog & Search team
informed: customer support
---

# Cache catalog search in-process with HybridCache

## Context and Problem Statement

At campaign peak, catalog search sends about 200 queries per second to the
database and the search scenario (p95 < 300 ms at 200 requests/s) is at risk.
Should we put a cache in front of catalog search, and if so, which one?

## Decision Drivers

* Database load at campaign peak (search scenario, H/M in the utility tree)
* A changed price must not be shown for longer than 60 seconds
* Search keeps working when any cache component fails
* One part-time ops engineer

## Considered Options

* No cache
* HybridCache, in-process only
* Redis through IDistributedCache

## Decision Outcome

Chosen option: "HybridCache, in-process only", because it removed 84% of the
search queries in the spike with no new infrastructure and no new failure mode.
The entry expiration is 60 seconds, because the price freshness driver and the
database load pull it in opposite directions (trade-off point).

### Consequences

* Good, because search no longer depends on the database for popular terms.
* Bad, because each instance has its own cache; database load grows with the
  instance count (192, 324, 474, 647 queries for 1, 2, 4, 8 instances).
* Bad, because a changed price can be shown for up to 60 seconds; checkout
  always reads the price from the database.

### Confirmation

A load test in the nightly pipeline fails when database queries per 1,000
searches exceed 250 with two instances.

## Pros and Cons of the Options

### Redis through IDistributedCache

* Good, because one shared cache: fewest queries (192) and no stale prices.
* Bad, because with Redis down every search failed after about 6 seconds in
  the spike; a timeout and fallback policy has to be designed first.
* Bad, because a new container to monitor, back up and secure.

## More Information

Reopen when we run more than 4 instances, or when price changes must be
visible in less than 60 seconds. Risk recorded: timeout behaviour of external
dependencies is not defined anywhere (same theme as the payment provider).

Three things are worth noticing. Decision Outcome names the trade-off point; 60 seconds is not an arbitrary number but a compromise between two drivers. Consequences gives the sensitivity point with its measured values; when the team moves to eight instances, nobody can say "we didn't know that". More Information records the two conditions that would reopen the decision and the risk theme. Redis was not rejected, it was postponed: first the timeout policy for external dependencies has to be written. We covered HybridCache itself in the "What's New in .NET 9" post of the ASP.NET Core Web API (Application Programming Interface) series, and the caching options in the caching post of the same series.

Common Pitfalls

Comparing options by features instead of scenarios. Neither "Redis is faster" nor "HybridCache is simpler" is tied to a scenario. Tie every claim to the question "which scenario, measured how?".

Deciding without a spike. The two important findings in this post, p95 not moving and L2 not solving the stale price, did not occur to anyone in the debate. The cheapest way to reduce uncertainty is to measure the point the decision turns on with a prototype of a few hours.

Taking spike results for production truth. The database was simulated, Redis was on the same machine, the traffic was synthetic. A spike compares the options with each other; absolute numbers need a load test close to production. Say so explicitly in the ADR.

Taking the weighted matrix's score for the decision. A score with two decimals is a number produced from guesses between 1 and 5. Report the cell that changes the winner, not the score.

Writing a non-risk without its assumption. "The old price is not a problem", without "because checkout reads the price from the database", turns into a bug six months later.

Looking only at the risks of decisions already made. The SEI report's finding is clear: most risks come from decisions not made. In the session, ask separately "what has nobody decided yet?".

A Small Experiment

★☆☆ In the spike, raise the number of terms in Zipf from 1,000 to 5,000. How does the query gap between HybridCache and Redis change? For p95 the miss ratio has to drop below 5%; at which number of terms does that happen?

★★☆ Add a LocalCacheExpiration setting to HybridSearch and repeat the price test after waiting longer than the expiration. Then measure the trade-off point: with expiration at 5, 30 and 60 seconds, how many database queries does a time-stamped request sequence produce?

★★★ Run a two-hour lightweight ATAM session for a pending decision in your own system. Fill in the option × scenario matrix, and find at least one sensitivity point, one trade-off point, one risk and one non-risk with its assumption. Can you group the risks and draw out a theme?

This decision was made inside one team, in two hours. Some decisions, though, involve several modules and teams, and their solution is not clear from the start. In the next post we move to the tools for debating such decisions in writing: the sections and lifecycle of a design doc, the RFC (Request for Comments) process, and a two-page design doc written to add an outbox to Bookstore's order service.