What Is System Design and How Do You Approach It? A 4-Step Method from Requirements to Deep Dive

Version note: The code in this post was built and run with C# 14 and .NET 10; the connection pool example ran with Microsoft.Data.SqlClient 7.1.1 against SQL (Structured Query Language) Server 2025.

Guess First

It is the first hour of the back-to-school campaign. On a normal day Bookstore's catalog search gets 20 requests per second; today it gets 200. The application is a single ASP.NET Core instance and a single SQL Server; CPU (Central Processing Unit) sits at 40% and memory is fine. The "best sellers" query on the campaign banner looks at a column with no index, the database slows down, and average search time climbs from 300 ms to 1.5 seconds. Ten minutes later users start seeing an error page instead of search results, and one line keeps repeating in the logs.

Which resource runs out first: CPU, ASP.NET Core's threads, or something else? And could you have found it with pen and paper before the system went down?

What Is System Design?

System design is deciding which parts a piece of software is made of, how those parts talk to each other, and how they behave under load, failure and growth. What separates it from design at the class and method level is scale: the units we talk about here are load balancers, caches, database replicas, queues and the network between them. In the terms of the Introduction to Architecture series post "What Is Software Architecture? How It Differs from Design and What an Architect Does", system design is the runtime and infrastructure face of architecture: which box runs where, which path a request takes, what happens when a box goes down.

This series aims at two things at once. The first is system design interviews: giving a structured answer to "design a URL (Uniform Resource Locator) shortener" in 45–60 minutes at a whiteboard. The second is real projects: finding out why Bookstore went down on campaign day and getting ready for the next one. Both use the same muscle; the difference is that in a real project you have to measure and leave your decisions in writing.

On the interview side the most widely used structure is the four steps in Alex Xu's System Design Interview (Vol. 1): understand the problem and establish the scope, propose a high-level design and get buy-in, do a deep dive, wrap up. Donne Martin's open-source System Design Primer suggests a similar order: write down use cases and constraints, sketch a high-level design, design the core components, scale. Below we apply the four steps to the campaign day problem.

Step 1: Clarifying Requirements and Narrowing the Scope

The first step is not drawing something, it is asking questions. Xu suggests spending 3–10 minutes of a 45-minute interview on this step; in a real project it means a few meetings and a document. We separate two kinds of requirements.

Functional requirements are what the system does: a user searches for a book, adds it to the cart, pays, sees the order status. Non-functional requirements (NFR) are how well it does it: for how many users, at what latency, with what availability, with what consistency. The Introduction to Architecture series post "Quality Attribute Scenarios: Turning "Make It Fast" into a Measurable Requirement" showed how to write NFRs with a source, a stimulus, an environment and a measure. At interview speed you do the same with one-line sentences: "catalog search p95 (the time 95% of requests stay under) below 300 ms at 200 requests per second".

For the campaign day problem these are the questions I would ask: Which flow is critical, search or checkout? How many times normal is peak traffic and how long does it last? Would a user accept slightly stale data instead of an error? If the payment provider slows down, can an order be put on hold? What are the budget and the team; for a team of three developers there is no point discussing a multi-region solution.

I write the answers down as assumptions, because every calculation that follows rests on them:

Assumptions (campaign day, Bookstore)
- normal: ~15,000 unique visitors/day, ~800 orders/day
- peak: 10x normal, ~2 hours; search ~200 req/s, checkout ~120 orders/min (2/s)
- targets: search p95 < 300 ms, checkout p95 < 800 ms, checkout availability 99.9%/month
- payment provider slower than 5 s -> order becomes PaymentPending, no error to user
- out of scope: multi-region, mobile app, recommendations

The last line matters at least as much as the first. Narrowing the scope lets you spend the remaining 35 minutes of an interview on what actually matters; in a real project, having "we are not doing this now" in writing is the answer to "why isn't it multi-region?" six months later.

Step 2: High-Level Design and Getting Buy-In

In the second step you draw the boxes and arrows: client, a CDN (Content Delivery Network) if there is one, load balancer, application servers, database, cache, queue, external systems. The goal is not detail but looking at the same picture as the person across from you. Bookstore today is very simple:

Browser -> Bookstore.Web (1 instance, ASP.NET Core)
             |-> SQL Server (catalog, orders, stock)
             |-> payment provider (HTTP, sync, 5 s timeout)
             |-> e-mail provider (SMTP, via in-process Channel<T> queue)

Showing this drawing and saying "this is the flow, this is where I will look for the bottleneck" is getting buy-in. In an interview this is also where you roughly write down the API (Application Programming Interface) and the data model: GET /api/books?q=, POST /api/orders and the tables. Back-of-the-envelope calculations belong to this step too; they are exactly the topic of the next post.

Step 3: Deep Dive: Calculate the Bottleneck

In the deep dive you open not every box but the one or two that carry the problem. On campaign day the critical question is: when search slows down, how many requests are waiting in the system at the same time? The answer is the most useful result of queueing theory, Little's law: the average number of items in a system = arrival rate × the time each item spends in the system (L = λ × W). The formula is independent of the arrival distribution and holds for any system in steady state.

// Little's law: requests in flight = arrival rate x time in system
Endpoint[] campaignPeak =
[
    new("search", 200, 0.30),
    new("search", 200, 1.50),
    new("checkout", 2, 0.80),
    new("checkout", 2, 7.00),
];

Console.WriteLine("Campaign peak, requests in flight (L = lambda x W)");
foreach (Endpoint e in campaignPeak)
    Console.WriteLine($"  {e.Name,-9}{e.RequestsPerSecond,5} req/s x {e.LatencySeconds:0.00} s = {e.InFlight,6:0.#} in flight");

record Endpoint(string Name, double RequestsPerSecond, double LatencySeconds)
{
    public double InFlight => RequestsPerSecond * LatencySeconds;
}

Campaign peak, requests in flight (L = lambda x W)
  search     200 req/s x 0.30 s =     60 in flight
  search     200 req/s x 1.50 s =    300 in flight
  checkout     2 req/s x 0.80 s =    1.6 in flight
  checkout     2 req/s x 7.00 s =     14 in flight

Traffic did not change, only the duration went up five times, and the number of search requests waiting inside at the same time went from 60 to 300. Each of these requests holds a database connection for the duration of its query. Microsoft.Data.SqlClient's connection pool opens at most 100 connections by default (Max Pool Size=100), and a request waiting for a free connection gets an exception after waiting for Connect Timeout (15 seconds by default). Checkout, even if the payment provider slows down to 7 seconds, reaches only 14 concurrent requests; on campaign day it is search, not checkout, that takes the system down.

Let's verify the calculation on a real pool. We simulate the slow query with a five-second WAITFOR DELAY, limit the pool to 5 connections and the wait to 3 seconds, and send 8 requests at the same time. It is a small-scale copy of the production pool of 100 facing 300 requests:

string cs = "Server=localhost;Database=Bookstore;Trusted_Connection=True;TrustServerCertificate=True;" +
            "Max Pool Size=5;Connect Timeout=3";

Stopwatch clock = Stopwatch.StartNew();
Task<string>[] calls = [.. Enumerable.Range(1, 8).Select(i => RunSearchAsync(i, cs, clock))];
foreach (string line in await Task.WhenAll(calls))
    Console.WriteLine(line);

static async Task<string> RunSearchAsync(int id, string cs, Stopwatch clock)
{
    try
    {
        await using SqlConnection connection = new(cs);
        await connection.OpenAsync();
        // a 5-second wait stands in for the slow search query
        await using SqlCommand command = new("WAITFOR DELAY '00:00:05'", connection);
        await command.ExecuteNonQueryAsync();
        return $"  request {id}: ok after {clock.Elapsed.TotalSeconds:0.0} s";
    }
    catch (InvalidOperationException ex)
    {
        return $"  request {id}: failed after {clock.Elapsed.TotalSeconds:0.0} s -> {ex.Message}";
    }
}

  request 1: ok after 5.2 s
  request 2: ok after 5.3 s
  request 3: ok after 5.3 s
  request 4: ok after 5.4 s
  request 5: ok after 5.5 s
  request 6: failed after 3.0 s -> Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
  request 7: failed after 3.0 s -> Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
  request 8: failed after 3.0 s -> Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

(In the copy we ran, the connection string points to our own test container; the rest of the code is identical.) This is the answer to the opening guess: the first resource to run out is neither CPU nor threads but the database connection pool, and the repeating log line is this message. Five requests got a connection and finished their query; the other three failed at the third second without ever starting their query. CPU staying at 40% is no surprise either: the waiting requests are inside an await, doing no work.

This is where the deep dive pays off: "let's buy a bigger server" does not solve this problem, because the bottleneck is not the server's power but how long the slow query holds a connection. The options follow directly from the calculation: reduce the query time (an index, preparing the query before the campaign), stop running the same query on every request (a cache; this series' post "Caching Strategies: Cache-Aside, Write-Through, Write-Behind and Cache Stampede"), grow the pool (as far as the database can handle; it is not free), and reject excess load early (load shedding; in the series' reliability post).

Step 4: Stating the Trade-offs and Wrapping Up

In the last step you state your choices and their costs out loud. For campaign day my proposal would be: read the "best sellers" list not from the database but from a cache refreshed every minute; the cost is that the list can be up to a minute old, and the marketing team accepts that. Add the missing index to the search query; the cost is a small extra on writes. Grow the pool only if the database has room for it. Leave the payment side alone; the calculation shows it is not the problem.

When wrapping up, naming the system's next bottleneck is a good habit too: if traffic goes up 30 times instead of 10, the write capacity of a single SQL Server and the single application instance become the problem; those are the topics of the series' posts "From One Server to Millions of Users: Vertical and Horizontal Scaling and the Stateless Tier" and "Scaling the Data Layer: Replication, Sharding and SQL vs NoSQL". There is no single right answer; as in the Introduction to Architecture series post "Trade-off Thinking: No Right Answer, Only the Least Worst Option", you pick the least worst option and give the reason.

The Difference Between an Interview and a Real Project

In an interview you make your decisions by talking for 45 minutes, and most of the numbers are assumptions. In a real project three things change. The first is measuring: production metrics replace the assumptions; load testing before the campaign verifies what Little's law says by measuring it. The second is writing: the high-level design and trade-offs go into a design doc, individual decisions into ADRs (Architecture Decision Record); the upcoming Architecture Decisions series covers this. The third is time: in an interview you draw the "final state" in one go, in reality a system grows step by step and each step fixes the bottleneck of the previous one.

There is one similarity as well: in both settings you ask questions first and draw second. A candidate who asks no questions gets marked "did not understand the requirements"; a team that asks no questions builds a beautiful architecture that solves the wrong problem.

Common Pitfalls

Starting to draw without asking about requirements. A whiteboard where Kafka and Kubernetes are drawn the moment someone says "design an e-commerce site" never sees the real campaign day problem, a single slow query.

Looking for the one right answer. The same question can have two good answers; what is assessed is not the answer you pick but knowing the alternatives and being able to state the cost.

Spending equal time on every box. The deep dive is about the component that carries the problem; explaining the load balancer algorithm for five minutes and never opening the database is putting your time in the wrong place.

Designing without numbers. "There is a lot of traffic" is not a requirement. Without requests per second, a latency target and an availability target, the question "cache or replica?" has no answer.

Talking in averages. What annoys users is not the average but the requests on the slow side; targets need to be written with p95 or p99. In the next post we will see this difference in numbers.

A Small Experiment

★☆☆ In the example, raise Max Pool Size to 8 and send the same 8 requests. Then send 16. Which requests failed, and how did the total time change compared to the first run? Explain the result with Little's law.

★★☆ Write an assumption list like the one in Step 1 for Bookstore's checkout flow: functional requirements, three NFRs and an "out of scope" line. Then calculate how many concurrent requests checkout produces at peak by calling the payment provider synchronously with a 5-second timeout.

★★★ Write two alternative solutions for campaign day (for example reading "best sellers" from a cache and adding a separate read replica for reads). For each, write in one paragraph, in ADR format, the bottleneck it solves, its cost and the cost of reversing it.

In this post the calculation itself was a single formula. In the next post we make back-of-the-envelope estimation systematic: from DAU (Daily Active Users) to peak QPS (queries per second), from storage to bandwidth and the "nines" math of availability, with an end-to-end table for two real products.