What Is Test Case Design? Test Conditions, Test Cases, Coverage Goals and Families of Techniques

Version note: The code in this post was built and run with .NET 10 and xUnit.net v3 4.0.3.

Guess First

Bookstore, the online bookstore used as the shared example across this site's series, is getting institutional orders. The rules were settled in the Requirements Analysis series: an institutional order contains at least 20 items in total, the institutional discount is 10 percent off the list price, discounts don't stack and the larger one wins, and each line is rounded separately with half a kuruş rounding up. Emre has written the code:

namespace Bookstore.Ordering;

public sealed record OrderLine(string Isbn, decimal ListPrice, int Quantity, decimal CampaignRate = 0m);

public sealed record PricedLine(string Isbn, int Quantity, decimal AppliedRate, decimal Total);

public abstract record PricingResult
{
    public sealed record Priced(IReadOnlyList<PricedLine> Lines, decimal Total) : PricingResult;

    public sealed record Rejected(string Reason) : PricingResult;
}

public static class InstitutionalPricing
{
    public const int MinimumQuantity = 20;
    public const decimal InstitutionalRate = 0.10m;

    public static PricingResult Price(IReadOnlyList<OrderLine> lines)
    {
        int totalQuantity = lines.Sum(line => line.Quantity);
        if (lines.Any(line => line.Quantity < MinimumQuantity))
        {
            return new PricingResult.Rejected(
                $"An institutional order needs at least {MinimumQuantity} items; this order has {totalQuantity}.");
        }

        List<PricedLine> priced = [.. lines.Select(PriceLine)];
        return new PricingResult.Priced(priced, priced.Sum(line => line.Total));
    }

    private static PricedLine PriceLine(OrderLine line)
    {
        decimal rate = InstitutionalRate;
        decimal total = Math.Round(line.ListPrice * line.Quantity * (1 - rate), 2, MidpointRounding.AwayFromZero);
        return new PricedLine(line.Isbn, line.Quantity, rate, total);
    }
}

He has written tests too. Five rows, five different quantities; that's not a small number:

namespace Bookstore.Ordering.Tests;

public sealed class InstitutionalPricingTests
{
    [Theory]
    [InlineData(25)]
    [InlineData(30)]
    [InlineData(40)]
    [InlineData(50)]
    [InlineData(100)]
    public void Institutional_discount_is_ten_percent(int quantity)
    {
        PricingResult result = InstitutionalPricing.Price([new OrderLine("9780441172719", 250m, quantity)]);

        PricingResult.Priced priced = Assert.IsType<PricingResult.Priced>(result);
        Assert.Equal(250m * quantity * 0.9m, priced.Total);
    }
}

Test run summary: Passed!
  total: 5
  failed: 0
  succeeded: 5
  skipped: 0
  duration: 500ms

All green. The code has two bugs. Before reading on, guess: how many of these five tests could have caught one of them? Looking at the rules one by one is enough to find the bugs.

The Answer: Five Tests, One Scenario

The answer is zero. The first bug is in the minimum check: the rule looks at the order total, the code looks at each line. A school order with 19 maths books and 1 science book should pass under the rule; this code rejects it. The second bug is in the discount: the campaign rate is never read, so a book on a 25 percent campaign is also sold at 10 percent off.

None of the five tests touches these cases, because all five ask the same question: does a single-line order with no campaign and more than 20 items get 10 percent off? Every number from 25 to 100 takes the same path through the code. Five tests were written, but there is only one scenario being tested. That is what usually happens when tests are picked by intuition: five tests of the same kind, zero tests at the boundary, and no tests for the rest of the rule.

This time we derive the tests for the same rule from the requirement, rule by rule:

namespace Bookstore.Ordering.Tests;

public sealed class InstitutionalPricingTests
{
    private const string Dune = "9780441172719";
    private const string Emma = "9780141439587";
    private const string TheHobbit = "9780547928227";
    private const string Middlemarch = "9780141439549";

    // TC-01..TC-04: the 20-item minimum applies to the order total
    public static TheoryData<int[], bool> MinimumQuantityCases => new()
    {
        { [20], true },
        { [19], false },
        { [19, 1], true },
        { [12, 7], false },
    };

    [Theory]
    [MemberData(nameof(MinimumQuantityCases))]
    public void Minimum_quantity_applies_to_the_order_total(int[] quantities, bool accepted)
    {
        OrderLine[] lines = [.. quantities.Select(quantity => new OrderLine(Dune, 250m, quantity))];

        PricingResult result = InstitutionalPricing.Price(lines);

        Assert.Equal(accepted, result is PricingResult.Priced);
    }

    // TC-05: a rejection states the order's total quantity
    [Fact]
    public void Rejection_states_the_order_total()
    {
        PricingResult result = InstitutionalPricing.Price(
            [new OrderLine(Dune, 250m, 12), new OrderLine(Emma, 84.95m, 7)]);

        PricingResult.Rejected rejected = Assert.IsType<PricingResult.Rejected>(result);
        Assert.Contains("this order has 19", rejected.Reason);
    }

    // TC-06, TC-07: discounts don't stack, the larger one wins
    public static TheoryData<string, decimal, decimal, int, decimal, decimal> LargerDiscountWinsCases => new()
    {
        { TheHobbit, 120m, 0.25m, 20, 0.25m, 1800.00m },
        { Middlemarch, 200m, 0.05m, 20, 0.10m, 3600.00m },
    };

    [Theory]
    [MemberData(nameof(LargerDiscountWinsCases))]
    public void Larger_discount_wins(
        string isbn, decimal listPrice, decimal campaignRate, int quantity, decimal expectedRate, decimal expectedTotal)
    {
        PricingResult result = InstitutionalPricing.Price([new OrderLine(isbn, listPrice, quantity, campaignRate)]);

        PricedLine line = Assert.Single(Assert.IsType<PricingResult.Priced>(result).Lines);
        Assert.Equal(expectedRate, line.AppliedRate);
        Assert.Equal(expectedTotal, line.Total);
    }

    // TC-08: each line is rounded separately, half a kurus rounds up
    [Fact]
    public void Each_line_is_rounded_half_up()
    {
        PricingResult result = InstitutionalPricing.Price(
            [new OrderLine(Emma, 84.95m, 3), new OrderLine(Dune, 250m, 17)]);

        PricingResult.Priced priced = Assert.IsType<PricingResult.Priced>(result);
        Assert.Equal([229.37m, 3825.00m], priced.Lines.Select(line => line.Total));
        Assert.Equal(4054.37m, priced.Total);
    }
}

Eight tests, only three more than before. Running them (stack traces trimmed):

failed Bookstore.Ordering.Tests.InstitutionalPricingTests.Larger_discount_wins(isbn: "9780547928227", listPrice: 120, campaignRate: 0.25, quantity: 20, expectedRate: 0.25, expectedTotal: 1800.00) (3ms)
  Assert.Equal() Failure: Values differ
  Expected: 0.25
  Actual:   0.10
failed Bookstore.Ordering.Tests.InstitutionalPricingTests.Each_line_is_rounded_half_up (0ms)
  Assert.IsType() Failure: Value is not the exact type
  Expected: typeof(Bookstore.Ordering.PricingResult+Priced)
  Actual:   typeof(Bookstore.Ordering.PricingResult+Rejected)
failed Bookstore.Ordering.Tests.InstitutionalPricingTests.Minimum_quantity_applies_to_the_order_total(quantities: [19, 1], accepted: True) (0ms)
  Assert.Equal() Failure: Values differ
  Expected: True
  Actual:   False

Test run summary: Failed!
  total: 8
  failed: 3
  succeeded: 5
  skipped: 0

Both bugs were caught. Look closely at the third failure: the rounding test did not fail because of rounding. The order with two lines of 3 and 17 items was rejected by the per-line minimum check. So the rounding rule had never actually been exercised in this code; one bug was hiding the test of another rule. In the next post we'll give this effect a name.

This post is about how the second test list was reached. Luck didn't produce it; a method did.

From Test Basis to Test Data: The Chain

The current ISTQB CTFL syllabus, v4.0.1, defines test analysis and test design as two separate steps. Analysis answers "what will we test?", design answers "how will we test it?". The distinction rarely comes up in day-to-day development, yet it is the best explanation of why picking tests by intuition falls short. The chain looks like this:

The test basis here is the acceptance criteria: the rules and examples laid out in the Requirements Analysis series' "Acceptance Criteria and Example Mapping" post. A rule, not code. If you derive a test from the code, you confirm what the code does; if you derive it from the rule, you check what the code should do.

A test condition is a testable piece that comes out of the test basis: "the minimum applies to the order total", "the larger discount wins". Each acceptance criterion usually yields one or more test conditions. The syllabus also notes that acceptance criteria can be seen as test conditions.

A coverage item makes concrete which parts of a test condition must be exercised. The technique decides which coverage items you pick: in equivalence partitioning each partition is a coverage item, in boundary value analysis each boundary value, in a decision table each rule column. The "minimum quantity" condition has four coverage items: the total being 20 or more, or below 20, in a single-line and in a multi-line order. Choosing the values on either side of the boundary, 19 and 20, is the job of boundary value analysis, the subject of the third post.

A test case covers one or more coverage items at once, with its inputs, preconditions and expected result. Test data is the concrete values that run that test case: 19 copies of Dune at 250 TRY and 1 copy of Emma at 84.95 TRY.

The whole chain for the institutional order rule:

Every row in the table has a reason, and the reason comes from the rule, not from the code. Emre's five tests fall into a single coverage item in the first row: "single line, total 20 or more". That is also why the test names and comments in the code should lead back to this table: every test from TC-01 to TC-08 says which rule it protects. This is the cheapest form of what the syllabus calls traceability.

The Coverage Goal: Covered Items, Not Test Count

The answer to "how many tests should I write?" is not a number but a ratio. Each technique defines its own coverage items, and coverage is the number of items exercised by at least one test case divided by the total number of items. Seen through this measure, the intuitive tests are plainly weak: the table has four test conditions and eight coverage items, and the five tests touch only one. That is 12.5 percent coverage. In the second list, eight tests cover all eight items: 100 percent.

Don't confuse this ratio with code coverage. Code coverage measures the share of lines or branches the tests executed and looks at the code; as we discussed in the Test Strategy series' "What Code Coverage Tells You and What It Doesn't" post, a high number proves nothing on its own. The coverage we are talking about here looks at items derived from the requirement. Emre's five tests executed most of the lines of the buggy code; the code had only two branches and the tests went through one of them. What was missing was not a line in the code but behaviour that wasn't in the code at all: there was no line reading the campaign rate that could have gone unexecuted. A measure that looks at the code cannot see code that was never written.

The coverage goal is set per technique, and "100 percent" is not always a sensible target. For a rule that calculates money, every column of the decision table is covered 100 percent; for the filter combinations of a report, combinatorial techniques give a lower but deliberate target. What matters is that a target exists: only then can "have we written enough tests?" be answered.

Families of Techniques: Which One Answers Which Question?

The syllabus splits test techniques into three families and covers collaboration-based approaches separately under their own heading. The eight posts in this series follow the same split:

• Black-box techniques, also called specification-based techniques: "What does the rule say, and which regions of the rule behave differently?" Equivalence partitioning, boundary value analysis, decision table testing and state transition testing. The test cases are independent of how the code is written; if the implementation changes but the behaviour stays the same, the tests remain valid.
• White-box techniques, also called structure-based techniques: "Which branch of the code never ran?" Statement and branch coverage. They derive tests by looking at the code; they see every path through the code even when the spec is missing or outdated, but they cannot see behaviour that was never written.
• Experience-based techniques: "Where do bugs usually show up in this kind of code?" Error guessing, exploratory testing and checklist-based testing. Their strength depends on the tester's experience; they catch what the other two families miss.
• Collaboration-based approaches: "How do we clarify the rule together so the bug is never written?" Collaborative user story writing, acceptance criteria and ATDD. The others focus on finding defects; this approach focuses on preventing them.

Every one of these families left a mark on the institutional order example. We derived the tests from the rule: black-box. We chose 19 and 20 for the minimum boundary: boundary value analysis. We made the clash between campaign and institutional discount a separate condition: a small decision table. Testing rounding on its own is a bit of error guessing; anyone who knows that Math.Round in .NET defaults to banker's rounding never forgets that row. And the rules themselves came out of the example mapping session in Requirements Analysis, a collaboration-based activity.

The series plan follows the same order: first the black-box techniques (equivalence partitioning, boundary value analysis, decision tables and state transitions), then combinations with pairwise and Microsoft's PICT tool, using branch coverage for test design, error guessing and exploratory testing, and in the last post, choosing techniques for a story.

This classification is not unique to ISTQB. Glenford Myers' The Art of Software Testing (first edition 1979, third edition with Badgett and Sandler in 2011) presents equivalence partitioning and boundary value analysis under black-box testing and coverage criteria under white-box testing. Lee Copeland's A Practitioner's Guide to Software Test Design (2004) works through the techniques one by one on example problems. The definitions in this series follow the syllabus, but the examples and code come from Bookstore.

Fixing the Code

Both fixes are one line each. The minimum check looks at the total, and the line rate becomes the larger of the campaign and institutional rates:

int totalQuantity = lines.Sum(line => line.Quantity);
if (totalQuantity < MinimumQuantity)
{
    return new PricingResult.Rejected(
        $"An institutional order needs at least {MinimumQuantity} items; this order has {totalQuantity}.");
}

decimal rate = Math.Max(InstitutionalRate, line.CampaignRate);

Test run summary: Passed!
  total: 8
  failed: 0
  succeeded: 8
  skipped: 0
  duration: 276ms

Because the tests were derived from the rule, they keep their meaning when the implementation changes. If someone splits InstitutionalPricing into strategies tomorrow, prices lines in parallel or reads the campaign rate from the Catalog module, these eight tests stay the same. The intuitive tests had a second weakness: they computed the expected value with a formula, 250m * quantity * 0.9m. When that formula is a copy of the formula in the code, the test makes the same mistake as the code.

Production Notes

Keep the list of test conditions somewhere. You don't need a separate document; comments at the top of the test class, numbers like TC-01 or the test names are enough. What matters is being able to find which tests are affected when a rule changes. When Hakan says tomorrow "the minimum is 25, not 20", what you look for is not every test mentioning "minimum" but the test cases of that condition.

Every bug found in production asks a question: which test condition was missing? When you close the bug ticket, add not only the test that reproduces the bug but also the missing condition and its other coverage items. Had the "multi-line order" condition been on the list from the start, the minimum bug would have been caught before code review.

Test design belongs in code review. In a pull request, "which conditions do these tests cover and which one is missing?" deserves as much attention as "do the tests pass?". Reading an eight-test list is faster than reading the code, and it shows a missing rule more easily.

Report covered items, not test counts. Saying "120 new tests" at the end of a sprint tells nobody anything. Saying "the four conditions and eight coverage items of the institutional order story are covered 100 percent" tells them what is actually protected.

Common Pitfalls

Confusing test count with quality. Five tests from the same partition don't give more confidence than one; they only add test time and maintenance cost. Before writing a new test, ask "which item does this test cover that none of the existing tests does?". If there is no answer, the test is redundant.

Not clarifying the requirement before choosing a technique. Is the "minimum 20 items" rule per book or per order total? Every test written before this is answered guesses one of the two readings. If the wrong guess is the same in the code and in the test, the test stays green. The session in the Requirements Analysis series' "Acceptance Criteria and Example Mapping" post put exactly this question on the table; test design starts there, not in the test project.

Copying the expected value from the code. The expected value is calculated by hand, from the rule: 84.95 × 3 = 254.85; with 10 percent off, 229.365; half a kuruş up: 229.37. Producing that number with a formula in the test means writing the calculation you want to test a second time.

Testing only the happy path. A rejected order, a book on a campaign and a rounding boundary are not on the happy path. Most rules live at exactly these edges.

Not checking why a test is red. In this post, the rounding test failed because of the minimum bug, not because of rounding. If you shrug off a red test with "something is broken anyway", you can't be sure that the test turning green after the fix really verifies its own rule.

A Small Experiment

★☆☆ Run the five intuitive tests against the fixed code and see that they all still pass. Then set MinimumQuantity to 25: which tests turn red and which stay green? Compare the result with the coverage table in this post.

★★☆ Add a new sentence to the rule: "The institutional discount applies only to printed books; e-books are sold at list price." Write the test conditions and coverage items first, then the tests and the code. How many rows did the coverage table gain?

★★★ Send an empty list and a line with zero quantity to InstitutionalPricing.Price. Does the requirement say anything about these cases? If not, write the question you would ask Selin in one sentence and add a test condition based on the answer.

In the next post we move on to the first technique: equivalence partitioning. We'll split order quantity, customer type and payment method into partitions, separate valid from invalid partitions, see how combining two invalid values in one test hides a bug, and turn the partition table straight into TheoryData.