Quality Attribute Scenarios: Turning "Make It Fast" into a Measurable Requirement

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

Bookstore's requirements document has a single sentence about catalog search: "Search must be fast." In the previous post we called that a wish. In this post the team decides to measure it. The program below starts the search endpoint, sends 400 requests with 20 concurrent users and reads the durations from the metric ASP.NET Core produces itself. We simulate the endpoint's latency with a fixed seed: most requests are fast, a small share hits a slow query that isn't cached.

using System.Collections.Concurrent;
using System.Diagnostics.Metrics;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.WebHost.UseUrls("http://127.0.0.1:5203");
WebApplication app = builder.Build();

// Fixed seed for the test: about 8 in every 100 searches hit a slow query
Random random = new(42);
int[] delays = [.. Enumerable.Range(0, 400).Select(_ =>
    random.NextDouble() < 0.08 ? random.Next(450, 900) : random.Next(30, 90))];

app.MapGet("/books/search", async (int i) =>
{
    await Task.Delay(delays[i]);
    return Results.Ok(new[] { "Dune", "Emma" });
});

ConcurrentBag<double> durationsMs = [];
using MeterListener listener = new();
listener.InstrumentPublished = (instrument, l) =>
{
    if (instrument.Meter.Name == "Microsoft.AspNetCore.Hosting" &&
        instrument.Name == "http.server.request.duration")
    {
        Console.WriteLine($"listening: {instrument.Meter.Name} / {instrument.Name} ({instrument.Unit})");
        l.EnableMeasurementEvents(instrument);
    }
};
listener.SetMeasurementEventCallback<double>((_, seconds, _, _) => durationsMs.Add(seconds * 1000));
listener.Start();

await app.StartAsync();

using HttpClient http = new() { BaseAddress = new Uri("http://127.0.0.1:5203") };
await Parallel.ForEachAsync(Enumerable.Range(0, 400), new ParallelOptions { MaxDegreeOfParallelism = 20 },
    async (i, ct) => await http.GetStringAsync($"/books/search?i={i}", ct));

await app.StopAsync();

double[] sorted = [.. durationsMs.Order()];
Console.WriteLine($"requests: {sorted.Length}");
Console.WriteLine($"average:  {sorted.Average():F0} ms");
Console.WriteLine($"p50:      {Percentile(sorted, 50):F0} ms");
Console.WriteLine($"p95:      {Percentile(sorted, 95):F0} ms");
Console.WriteLine($"p99:      {Percentile(sorted, 99):F0} ms");
Console.WriteLine($"max:      {sorted[^1]:F0} ms");

QualityScenario search = new(
    Source: "Customer on the web site",
    Stimulus: "Searches the catalog by title",
    Artifact: "Catalog search endpoint",
    Environment: "Normal operation, 20 concurrent users",
    Response: "Matching books are returned",
    Percentile: 95,
    ThresholdMs: 300);

double measured = Percentile(sorted, search.Percentile);
Console.WriteLine($"scenario p{search.Percentile} < {search.ThresholdMs} ms: " +
    $"{(measured < search.ThresholdMs ? "PASS" : "FAIL")} ({measured:F0} ms)");

// Nearest-rank method: the ceil(p/100 * n)-th item of the sorted list
static double Percentile(double[] sorted, double p) =>
    sorted[(int)Math.Ceiling(p / 100 * sorted.Length) - 1];

public sealed record QualityScenario(
    string Source,
    string Stimulus,
    string Artifact,
    string Environment,
    string Response,
    double Percentile,
    double ThresholdMs);

The target is 300 milliseconds. What does the average come out to, and is it under the target? Below which duration do 95 percent of the requests stay? Does the scenario pass?

Why Does the Average Mislead?

listening: Microsoft.AspNetCore.Hosting / http.server.request.duration (s)
requests: 400
average:  121 ms
p50:      66 ms
p95:      659 ms
p99:      828 ms
max:      880 ms
scenario p95 < 300 ms: FAIL (659 ms)

The average is 121 milliseconds. If your dashboard shows only the average, you are under half the target and everyone is happy. Yet 5 percent of the requests take longer than 659 milliseconds and 1 percent longer than 828. The median (p50) is 66 milliseconds: a typical request is very fast, but the tail of the distribution is long. The average mixes these two worlds into a single number and describes both of them wrongly.

A quick calculation shows why the tail matters. Here about 8 percent of the requests are slow. If a customer runs ten searches in a session, the chance that at least one is slow is 1 − 0.92¹⁰, about 57 percent. More than half of the users get stuck at least once per session, and the average shows none of it. That is why we talk about latency in percentiles: p50 describes the typical experience, p95 and p99 describe the tail.

The first line of the output says something important too: we did not measure the durations ourselves. Since .NET 8, ASP.NET Core writes the duration of every HTTP (Hypertext Transfer Protocol) request, in seconds, to the http.server.request.duration histogram on the Microsoft.AspNetCore.Hosting meter. MeterListener is the BCL (Base Class Library) way to listen to that instrument in process; in production you send the same metric to a monitoring system with OpenTelemetry. So the data needed to measure the "make it fast" requirement is already produced by the framework; what is missing is the sentence that says what to measure and where the threshold is.

The Quality Attribute Scenario: Six Parts

There is an established template for that sentence. Software Architecture in Practice (4th edition, 2021) by SEI (Software Engineering Institute) researchers Bass, Clements and Kazman suggests writing a quality attribute requirement as a six-part scenario:

Source of stimulus: who or what generates the stimulus. A customer, an attacker, the payment provider, a developer. The source matters, because the system may react differently to a trusted source than to an unknown one. Stimulus: the condition that arrives at the system and must be handled; a request, a fault, a change request. Environment: the conditions at the moment the stimulus arrives; normal operation, campaign peak, a dependency being down, a deployment in progress. Artifact: the part the stimulus targets; the whole system or an endpoint, a module. Response: the activity the system undertakes in reaction. Response measure: the measurable target for the response; this is the part that makes the scenario testable.

The book talks about two kinds of scenario. General scenarios are not specific to a system; they are templates such as "if a dependency doesn't respond, the system keeps operating" that apply to any system. A concrete scenario is the version tailored to a particular system, and it plays the role for quality attributes that use cases play for functional requirements. The QualityScenario record in the opening is a concrete scenario expressed in code: five of the six parts are text, while the response measure is a percentile and a threshold. The last line of the program turns the scenario into a test, and the result is clear: FAIL.

Three Scenarios from Bookstore

In the previous post we chose Bookstore's top three characteristics: availability, performance and modifiability. One concrete scenario for each:

Every cell in the table closes a debate. "Fast" is now "p95 below 300 milliseconds at campaign peak". "Highly available" is now "no lost orders even when the payment provider is down". "Easy to maintain" is "a new payment method in three days, touching a single module". The third is especially valuable, because modifiability is usually assumed to be unmeasurable. It is measurable; the unit just isn't milliseconds, it is developer-days and the number of modules changed.

The response in the availability scenario also implies an architecture decision: accepting the order as "PaymentPending" instead of rejecting it when the payment provider doesn't respond. That is a preview of one of the trade-offs in the next post; we give up a little consistency to gain availability.

Prioritizing with a Utility Tree

As scenarios multiply, the question of which one to tackle first comes up. ATAM (Architecture Tradeoff Analysis Method, the SEI method by Kazman, Klein and Clements described in a 2000 technical report) uses a utility tree for this. The root of the tree is "utility", the overall goodness of the system; below it are quality attributes, below those more concrete refinements, and the leaves are concrete scenarios. Each scenario gets two ratings: its importance to the business and the architectural difficulty of achieving it, each H (high), M (medium) or L (low). A small tree for Bookstore:

• Performance › search › p95 < 300 ms under 200 requests per second during campaigns (H, M)
• Performance › checkout › checkout p95 < 800 ms (H, M)
• Availability › external dependency › no lost orders when the payment provider doesn't respond for 5 seconds (H, H)
• Availability › deployment › checkout is not interrupted for more than 1 minute during the weekly deployment (M, M)
• Modifiability › a new payment method in at most 3 developer-days, only Payment changes (H, M)
• Modifiability › adding English to the interface without changing existing pages (M, L)

Scenarios marked (H, H) are where the architecture will need the most effort; the first analysis and prototypes go there. (H, L) ones are important but easy, probably already met by the current structure. (L, H) ones should be questioned: is it worth complicating the architecture for a hard but unimportant goal? This is consistent with FoSA's (Fundamentals of Software Architecture, Richards and Ford) "top three characteristics" practice: one or two (H, *) scenarios for each of three characteristics is about as many targets as a team can really track in a quarter. The full ATAM, with its stakeholder workshops and concepts such as sensitivity points and trade-off points, belongs to the upcoming Architecture Decisions series; here we use only the tree, as a prioritization tool.

Measurable Targets: Percentiles, Error Rate, Time

A few details make a difference when writing a response measure. The first is how you calculate the percentile. The code in the opening uses the nearest-rank method: the ⌈p/100 × n⌉-th item of the sorted list. With 400 measurements, p95 is the 380th. Monitoring systems interpolate over histogram buckets, so they can give a result a few milliseconds off on the same data; what matters is that the team uses the same definition.

The second is the window and the load. "p95 < 300 ms" alone is incomplete; over which time window and under which load? A p95 measured at three in the morning with two requests per second says nothing about campaign day. The environment part of the scenario exists exactly for this. Our measurement used 20 concurrent users; validating the campaign scenario needs a real load test that generates 200 requests per second.

The third is correctness, not just speed. An endpoint that is fast but returns errors for 5 percent of requests easily meets a p95 target, because errors usually come back fast. That is why we added an error rate threshold to the performance scenario. The http.server.request.duration histogram records every measurement together with the http.response.status_code tag, so the error rate can be derived from the same data.

The fourth is turning an unmeasurable sentence into a measurable one. Instead of "the code must be maintainable", "a new payment method in 3 developer-days"; instead of "the system must be secure", "if a signed-in customer requests another customer's order number, the API (Application Programming Interface) returns 404 and the event reaches the security log within 1 minute". The second sentence is both testable and a direct counterpart of the IDOR (Insecure Direct Object Reference) post in the OWASP (Open Worldwide Application Security Project) Top 10 series.

Connecting Scenarios to Tests and Monitoring

A scenario that stays in a document is forgotten within six months. There are two ways to keep it alive, and you need both.

The first is connecting it to a test. The last line of the program in the opening is a small example: it compares the measurement with the scenario's threshold and produces PASS or FAIL. The same comparison can run in a CI (Continuous Integration) pipeline every night, after a load test close to campaign load. For the modifiability scenario the test looks different: an architecture test that checks the Payment module does not reference Ordering. Such tests are called fitness functions and are the subject of the last post in this series.

The second is connecting it to monitoring. In production, a dashboard that calculates p95 from the same http.server.request.duration metric and an alert that fires when the threshold is crossed revalidate the scenario every day. There is an important boundary here: turning the scenario's target into an SLO (Service Level Objective), defining the measurement as an SLI (Service Level Indicator) and operating it with an error budget is a separate discipline that we cover in the upcoming SRE (Site Reliability Engineering) and Production Incidents series. From an architecture point of view, our job is to make sure the target is written measurably from the start and that the system produces the data to measure it.

Common Pitfalls

Making average latency the target. In the opening measurement the average was under half the target while p95 was more than twice it. Write latency targets as percentiles and look at p95 or p99 alongside p50.

Not writing down the environment. A latency target with no load or conditions gets measured in the easiest condition and declared "passed". Write into the scenario under which load and in which window the target holds.

Keeping unmeasurable sentences on the list. "Should be maintainable", "should be user-friendly", "should be secure" are not scenarios. Rewrite each with at least one concrete stimulus and response measure; if you can't, question how important the characteristic really is.

Measuring only the happy path. Performance scenarios are always written for normal operation, and availability scenarios are never written at all. Environments where a dependency is down, a deployment is running or campaign traffic arrives are where you learn the most.

Nobody owning the target. A scenario without an owner, a test and an alert is just a well-meaning document. Next to every (H, *) scenario, write down who tracks it.

A Small Experiment

★☆☆ In the opening program, lower the share of slow requests from 0.08 to 0.03. How do the average, p95 and p99 change? Does the scenario pass? Then set the share to 0.06 and explain why p95 and p99 react differently.

★★☆ Read the tags in the MeterListener callback too and split measurements by the value of http.response.status_code. Add a branch to the endpoint that returns 500 for 3 percent of requests, calculate the error rate and add an error rate threshold to the scenario.

★★★ Write one availability and one modifiability scenario from your own system with all six parts and give each an (importance, difficulty) rating in a utility tree. Design a test or an alert that verifies the response measure automatically; if you can't, rewrite the scenario until it is measurable.

Once you write scenarios, you will notice something: the targets conflict. Accepting an order as PaymentPending for the availability scenario relaxes the "no unpaid orders" rule. The next post is about exactly these conflicts: the idea that architecture has no right answer, only the least worst option.