White-Box and Experience-Based Techniques: Using Branch Coverage for Test Design, Error Guessing, Checklists and Exploratory Testing

Version note: The code in this post was built and run with .NET 10, ASP.NET Core 10, xUnit.net v3 4.0.3 and Microsoft.Testing.Extensions.CodeCoverage 18.12.0.

Guess First

In Bookstore, the online bookstore used as the shared example across this site's series, there's a new coupon: KITAP10. Selin's story is short: 10 percent off carts of at least 200 TRY; coupons don't apply to institutional customers, because they don't combine with the institutional discount. Emre's code:

namespace Bookstore.Ordering;

public sealed record Coupon(string Code, decimal Percent, decimal MinimumSubtotal, decimal? MaxDiscount = null);

public static class CouponCodes
{
    public static string Normalize(string code) => code.Trim().ToUpper();
}

public static class CouponCalculator
{
    public static decimal Discount(Coupon coupon, decimal subtotal, CustomerType customer)
    {
        if (customer == CustomerType.Institutional)
        {
            return 0m; // doesn't combine with the institutional discount
        }

        if (subtotal < coupon.MinimumSubtotal)
        {
            return 0m;
        }

        decimal discount = Math.Round(subtotal * coupon.Percent / 100m, 2, MidpointRounding.AwayFromZero);
        if (coupon.MaxDiscount is decimal max && discount > max)
        {
            discount = coupon.Percent;
        }

        return discount;
    }
}

Ayşe has derived the tests from the story with the techniques from the earlier posts: both sides of the minimum amount, a guest customer, an institutional customer.

// From the story: 10 percent on carts of at least 200 TRY; coupons don't apply to institutional customers
public static TheoryData<CustomerType, decimal, decimal> StoryCases => new()
{
    { CustomerType.Individual, 199.99m, 0m },
    { CustomerType.Individual, 200.00m, 20.00m },
    { CustomerType.Guest, 350.00m, 35.00m },
    { CustomerType.Institutional, 350.00m, 0m },
};

[Theory]
[MemberData(nameof(StoryCases))]
public void Discount_follows_the_story(CustomerType customer, decimal subtotal, decimal expected)
{
    Assert.Equal(expected, CouponCalculator.Discount(Kitap10, subtotal, customer));
}

The tests are green. How much does KITAP10 take off a 2,500 TRY cart?

The Answer: A Branch That Isn't in the Story

10 TRY. When the campaign team defined the coupon, they added a cap of "at most 100 TRY", and Emre put it into the code, but nobody wrote it into the story. When the discount exceeds the cap, the code returns the percentage instead of the cap: a 250 TRY discount that should drop to 100 TRY drops to 10 TRY.

Tests derived from the story couldn't see this branch, because the story has no such rule. All the techniques in the previous six posts are black-box: they derive test cases from the requirement, a model, a table. If the requirement is incomplete, so are they. In this post we look at two other sources: the code, which is white-box, and the tester's experience, which is experience-based.

Reading Branch Coverage for Test Design

The ISTQB CTFL v4.0.1 syllabus describes two white-box techniques. In statement testing, the coverage items are executable statements. In branch testing, the coverage items are branches, that is, transfers of control from one point to another in the code's control flow: the true and false sides of an if, each arm of a switch, the decision to enter or leave a loop. 100 percent branch coverage also gives 100 percent statement coverage; the reverse isn't true. What a coverage percentage means is the subject of the Test Strategy series' "What Code Coverage Tells You and What It Doesn't" post; here we use the report for one thing only: finding a branch that never ran and deriving a test case from it.

Because xUnit v3 tests run on Microsoft.Testing.Platform, coverage is collected by Microsoft's Microsoft.Testing.Extensions.CodeCoverage package. We add the package to the test project and ask for a report in Cobertura format:

> dotnet test --project tests/Bookstore.Ordering.Tests --coverage --coverage-output-format cobertura --coverage-output coverage.cobertura.xml

  In process file artifacts produced:
    - <repo>/TestResults/coverage.cobertura.xml

Test run summary: Passed!
  total: 108
  failed: 0
  succeeded: 108
  skipped: 0
  duration: 686ms

The CouponCalculator class's lines in the report:

<class line-rate="0.7857142857142857" branch-rate="0.75" complexity="8" name="Bookstore.Ordering.CouponCalculator" filename="<repo>/src/Bookstore.Ordering/CouponCalculator.cs">
  <lines>
    <line number="24" hits="1" branch="False" />
    <line number="25" hits="1" branch="True" condition-coverage="50% (2/4)">
      <conditions>
        <condition number="0" type="jump" coverage="50%" />
        <condition number="1" type="jump" coverage="50%" />
      </conditions>
    </line>
    <line number="26" hits="0" branch="False" />
    <line number="27" hits="0" branch="False" />
    <line number="28" hits="0" branch="False" />

Line 25 is if (coupon.MaxDiscount is decimal max && discount > max). There are two conditions, each with two outcomes; two of the four branches never ran. Lines 26–28, the body of the if, never ran either (hits="0"). We turn the report into a list of test cases: the first condition always came out true, because the coupon in the tests always has a cap; a coupon without a cap was never tried. The second condition always came out false; a discount above the cap was never tried.

The report tells us "run this line", but it doesn't tell us the expected result. Writing the expected result by reading the code would copy the code's bug into the test. So the first step is to go to Selin and ask about the cap: is the "at most 100 TRY" rule right, and why isn't it in the story? Once the answer comes, we test the boundary with boundary value analysis as in the earlier post: 1,000.00 TRY where the discount is exactly 100 TRY, 1,000.10 TRY which exceeds the cap by one kuruş, and 2,500 TRY which clearly exceeds it.

// From the coverage report: a discount above the cap and a coupon without a cap had never run
public static TheoryData<decimal, decimal> CapCases => new()
{
    { 1000.00m, 100.00m },
    { 1000.10m, 100.00m },
    { 2500.00m, 100.00m },
};

[Theory]
[MemberData(nameof(CapCases))]
public void Discount_is_capped_at_the_maximum(decimal subtotal, decimal expected)
{
    Assert.Equal(expected, CouponCalculator.Discount(Kitap10, subtotal, CustomerType.Individual));
}

[Fact]
public void Coupon_without_a_maximum_is_not_capped()
{
    Coupon uncapped = Kitap10 with { MaxDiscount = null };

    Assert.Equal(250.00m, CouponCalculator.Discount(uncapped, 2500.00m, CustomerType.Individual));
}

failed Bookstore.Ordering.Tests.CouponCalculatorTests.Discount_is_capped_at_the_maximum(subtotal: 1000.10, expected: 100.00) (1ms)
  Assert.Equal() Failure: Values differ
  Expected: 100.00
  Actual:   10
failed Bookstore.Ordering.Tests.CouponCalculatorTests.Discount_is_capped_at_the_maximum(subtotal: 2500.00, expected: 100.00) (0ms)
  Assert.Equal() Failure: Values differ
  Expected: 100.00
  Actual:   10

Test run summary: Failed!
  total: 8
  failed: 2
  succeeded: 6
  skipped: 0
  duration: 317ms

The fix is one word:

discount = max;

After the fix, CouponCalculator has 100 percent line and branch coverage.

What White-Box Can't See: Missing Code

There are two more lines in the same report. The CouponCodes class has 100 percent coverage, and so does CheckoutService. Both have bugs, as we'll see shortly. This isn't a contradiction; the syllabus states the weakness of white-box techniques plainly: because they take the whole implementation into account they can find defects even with a vague or outdated specification, but they can't see something that was never written in the code. If there's no if that rejects an empty cart, there's no unexecuted branch for it either; the report says 100 percent.

To find missing code, you need to look not at the code but at knowledge about how software breaks. The syllabus groups this under experience-based techniques: error guessing, checklist-based testing and exploratory testing.

Error Guessing and a Defect Taxonomy

Error guessing means anticipating errors, defects and failures based on the tester's knowledge: how the application behaved in the past, the mistakes developers tend to make, the failures seen in similar applications. The syllabus organizes this knowledge into categories such as input, output, logic, computation, interfaces and data; its systematic form is called a fault attack: drawing up a list of possible errors and designing tests that will expose each one.

The Bookstore team's list is compiled from bugs in this series and earlier projects:

• Null, empty string, empty list; zero and negative quantities.
• The Turkish I: `ToUpper()` and `ToLower()` depend on the current culture; in tr-TR, "i" becomes "İ" when uppercased.
• Time zones: the three hours between UTC and Istanbul time, a date shifting at the day boundary, as in the third post.
• Concurrent requests: two events reaching the same order at the same time, as in the fifth post.
• Double clicks and retries: the same request arriving twice.
• Money: rounding, a negative total, a lost kuruş.

CouponCodes.Normalize matches the second item exactly. Bookstore's API runs with the tr-TR culture because it formats prices for Turkey; if a customer types the coupon in lowercase as "kitap10", ToUpper() produces "KİTAP10" and the coupon isn't found. The story doesn't say this, the coverage report says 100 percent, but the item on the list is a test case on its own:

using System.Globalization;

namespace Bookstore.Ordering.Tests;

public sealed class CouponCodeTests
{
    // Error guessing: the Turkish I, upper/lower case and leading/trailing spaces
    [Theory]
    [InlineData("en-US", "kitap10")]
    [InlineData("tr-TR", "kitap10")]
    [InlineData("tr-TR", " Kitap10 ")]
    [InlineData("tr-TR", "KITAP10")]
    public void Coupon_code_is_normalized_the_same_way_in_every_culture(string culture, string code)
    {
        CultureInfo original = CultureInfo.CurrentCulture;
        try
        {
            CultureInfo.CurrentCulture = CultureInfo.GetCultureInfo(culture);

            Assert.Equal("KITAP10", CouponCodes.Normalize(code));
        }
        finally
        {
            CultureInfo.CurrentCulture = original;
        }
    }
}

failed Bookstore.Ordering.Tests.CouponCodeTests.Coupon_code_is_normalized_the_same_way_in_every_culture(culture: "tr-TR", code: "kitap10") (11ms)
  Assert.Equal() Failure: Strings differ
              ↓ (pos 1)
  Expected: "KITAP10"
  Actual:   "KİTAP10"
              ↑ (pos 1)
failed Bookstore.Ordering.Tests.CouponCodeTests.Coupon_code_is_normalized_the_same_way_in_every_culture(culture: "tr-TR", code: " Kitap10 ") (0ms)
  Assert.Equal() Failure: Strings differ
              ↓ (pos 1)
  Expected: "KITAP10"
  Actual:   "KİTAP10"
              ↑ (pos 1)

Test run summary: Failed!
  total: 12
  failed: 2
  succeeded: 10
  skipped: 0
  duration: 378ms

The code normalizes an identifier that is never shown to the user; it should be independent of the culture:

public static string Normalize(string code) => code.Trim().ToUpperInvariant();

Changing the culture inside a try/finally in the test and restoring it is deliberate too: CurrentCulture belongs to the thread, and if it isn't restored it affects tests that run later on the same thread.

Checklist-Based Testing

As long as a defect taxonomy lives in one person's head, the quality of error guessing depends on that person. A checklist makes the knowledge the team's. The syllabus's advice on checklists is practical: items are written as questions, each item can be checked separately and directly, things that can be checked automatically don't go on the list, the list is updated regularly based on defect analysis, and it isn't allowed to grow too long. The team's checkout list:

The list doesn't replace tests; it makes the question "did we think of this?" systematic during a new feature or a code review. When an item turns into an automated test, it leaves the list or becomes the question "is there a test for it?".

Exploratory Testing: Charters and Time-Boxes

In exploratory testing, test design, execution and evaluation happen at the same time; the tester picks the next test as they learn the system. What separates it from "clicking around without a plan" is structure: the Session-Based Test Management article that James and Jon Bach published in 2000 defines that structure. Work is split into uninterrupted sessions of fixed length; a charter sets each session's purpose; the tester takes notes during the session, keeping the bugs and the questions they find apart; at the end of the session, a debriefing goes over the results with stakeholders. The PROOF mnemonic Bach uses for debriefings gives the headings of that conversation: Past, what was done; Results, what was found; Obstacles, what got in the way; Outlook, what comes next; and Feelings, whether the tester is confident in the outcome.

For writing charters, the template from Elisabeth Hendrickson's Explore It! (2013) is handy: say in one sentence what you'll explore, with which resources, and what information you're looking for. The charter for Ayşe's one-hour session:

"Explore the checkout API with different cart contents, coupon spellings and repeated requests to discover cases where an order is created with the wrong amount or more than once."

A charter isn't a list of test cases; it gives direction but not steps. Ayşe runs the API locally and sends requests, choosing the next request based on each result. Selected requests from the session, with the real responses:

POST /api/checkout
{"customer":"Individual","lines":[{"isbn":"9780441013593","quantity":2,"unitPrice":150.00}],"couponCode":"kitap10"}

HTTP/1.1 201 Created
Location: /api/orders/1001
{"orderId":1001,"subtotal":300.00,"discount":30.00,"total":270.00}

# The same request, twice in a row
{"orderId":1003,"subtotal":300.00,"discount":30.00,"total":270.00}
{"orderId":1002,"subtotal":300.00,"discount":30.00,"total":270.00}

POST /api/checkout
{"customer":"Guest","lines":[],"couponCode":null}

HTTP/1.1 201 Created
Location: /api/orders/1004
{"orderId":1004,"subtotal":0,"discount":0,"total":0}

POST /api/checkout
{"customer":"Guest","lines":[{"isbn":"9780141439518","quantity":-3,"unitPrice":95.00}]}

{"orderId":1006,"subtotal":-285.00,"discount":0,"total":-285.00}

POST /api/checkout
{"customer":"Individual","lines":[{"isbn":"9780547928227","quantity":1,"unitPrice":120.00}],"couponCode":"KİTAP10"}

{"type":"https://tools.ietf.org/html/rfc4918#section-11.2","title":"Unprocessable Entity","status":422,"detail":"Unknown coupon 'KİTAP10'."}

The lowercase "kitap10" works now; the error guessing fix is in place. But the session found other things. Ayşe's session notes:

• Bug: when the same request arrives twice, two orders are opened, 1002 and 1003. A customer who double-clicks the button, or whose connection drops and retries, may pay twice.
• Bug: an empty cart opens a 0 TRY order with 201 Created; a line with quantity 0 does the same.
• Bug: a negative quantity makes the total negative, -285 TRY. If the payment side reads that as a refund, it's a serious loss of money.
• Question: "KİTAP10" typed in uppercase on a Turkish keyboard isn't found. Should coupon codes use the letter I at all, or should İ be accepted too? For Selin.
• Question: when an institutional customer enters a coupon, the discount silently becomes 0; the response doesn't say why. For Selin.

Bugs and questions are kept apart on the session sheet, because questions wait for a decision, not a code change. In the debriefing Selin answers the second question right away: the response should carry a warning, and that's a new story. The first question goes to the campaign team.

From Session Notes to Regression Tests

The most commonly lost output of an exploratory session is the findings never turning into tests. We turn each bug into a test at the lowest level that can reproduce it. All three bugs are in CheckoutService; there's no need for HTTP:

// Exploratory session findings, as regression tests
[Fact]
public void Empty_cart_is_rejected()
{
    CheckoutResult result = NewService().PlaceOrder(new(CustomerType.Guest, [], null));

    Assert.Null(result.Order);
    Assert.Equal("Cart is empty.", result.Error);
}

[Theory]
[InlineData(0)]
[InlineData(-3)]
public void Line_quantity_must_be_positive(int quantity)
{
    CheckoutResult result = NewService().PlaceOrder(
        new(CustomerType.Guest, [new CartLine("9780141439518", quantity, 95m)], null));

    Assert.Null(result.Order);
    Assert.Equal("Quantity must be positive.", result.Error);
}

[Fact]
public void Same_idempotency_key_returns_the_same_order()
{
    CheckoutService service = NewService();
    CheckoutCommand command = new(CustomerType.Individual, [Dune], "KITAP10");

    CheckoutResult first = service.PlaceOrder(command, idempotencyKey: "checkout-7f3a");
    CheckoutResult second = service.PlaceOrder(command, idempotencyKey: "checkout-7f3a");

    Assert.Equal(first.Order, second.Order);
}

The fix for double clicks is an idempotency key: the client generates a key for each checkout attempt and reuses the same key when it resends the request. To make the test compile, we first add the parameter to PlaceOrder without using it yet:

failed Bookstore.Ordering.Tests.CheckoutServiceTests.Line_quantity_must_be_positive(quantity: 0) (8ms)
  Assert.Null() Failure: Value is not null
  Expected: null
  Actual:   PlacedOrder { OrderId = 1001, Subtotal = 0, Discount = 0, Total = 0 }
failed Bookstore.Ordering.Tests.CheckoutServiceTests.Line_quantity_must_be_positive(quantity: -3) (0ms)
  Assert.Null() Failure: Value is not null
  Expected: null
  Actual:   PlacedOrder { OrderId = 1001, Subtotal = -285, Discount = 0, Total = -285 }
failed Bookstore.Ordering.Tests.CheckoutServiceTests.Empty_cart_is_rejected (0ms)
  Assert.Null() Failure: Value is not null
  Expected: null
  Actual:   PlacedOrder { OrderId = 1001, Subtotal = 0, Discount = 0, Total = 0 }
failed Bookstore.Ordering.Tests.CheckoutServiceTests.Same_idempotency_key_returns_the_same_order (1ms)
  Assert.Equal() Failure: Values differ
  Expected: PlacedOrder { OrderId = 1001, Subtotal = 300, Discount = 30, Total = 270 }
  Actual:   PlacedOrder { OrderId = 1002, Subtotal = 300, Discount = 30, Total = 270 }

Test run summary: Failed!
  total: 6
  failed: 4
  succeeded: 2
  skipped: 0
  duration: 366ms

The fix is two checks at the start of PlaceOrder and a lookup by key inside the lock:

if (command.Lines.Count == 0)
{
    return new CheckoutResult(null, "Cart is empty.");
}

if (command.Lines.Any(line => line.Quantity <= 0))
{
    return new CheckoutResult(null, "Quantity must be positive.");
}

lock (_lock)
{
    // A second request with the same key doesn't open a new order; it returns the first one
    if (idempotencyKey is not null && _ordersByIdempotencyKey.TryGetValue(idempotencyKey, out PlacedOrder? existing))
    {
        return new CheckoutResult(existing, null);
    }

    PlacedOrder order = new(_nextOrderId++, subtotal, discount, subtotal - discount);
    if (idempotencyKey is not null)
    {
        _ordersByIdempotencyKey[idempotencyKey] = order;
    }

    return new CheckoutResult(order, null);
}

The API takes the key from a header:

app.MapPost("/api/checkout", (
    CheckoutCommand command,
    [FromHeader(Name = "Idempotency-Key")] string? idempotencyKey,
    CheckoutService checkout) =>
{
    CheckoutResult result = checkout.PlaceOrder(command, idempotencyKey);
    return result.Order is { } order
        ? Results.Created($"/api/orders/{order.OrderId}", order)
        : Results.Problem(detail: result.Error, statusCode: StatusCodes.Status422UnprocessableEntity);
});

We try the session's findings once more through the API. A request sent twice with the same Idempotency-Key: checkout-7f3a header returns a single order, and an empty cart and a negative quantity get 422:

{"orderId":1001,"subtotal":300.00,"discount":30.00,"total":270.00}
{"orderId":1001,"subtotal":300.00,"discount":30.00,"total":270.00}

{"type":"https://tools.ietf.org/html/rfc4918#section-11.2","title":"Unprocessable Entity","status":422,"detail":"Cart is empty."}
{"type":"https://tools.ietf.org/html/rfc4918#section-11.2","title":"Unprocessable Entity","status":422,"detail":"Quantity must be positive."}

Test run summary: Passed!
  total: 149
  failed: 0
  succeeded: 149
  skipped: 0
  duration: 339ms

This idempotency is a dictionary held in memory and works for a single server instance. With several instances the key has to be stored in the database or a shared cache; that's tested at the integration level, not the unit level.

Production Notes

Read the coverage report in the pull request. Don't look at the total percentage; look at unexecuted branches in the changed files. The cap branch in this post was a small red area lost inside a 95 percent total.

Don't make coverage a target. Once the target is a number, tests get written to raise that number, and expected results get filled in by reading the code. The report produces a list of questions; the answers come from the requirement and from Selin.

Keep the error guessing list alive. When a production bug's root cause is analyzed, ask "is this a category?". If it is, it goes on the list and the checklist; if it's a one-off, it only becomes a regression test.

Put exploratory sessions on the calendar. Exploratory testing left as "we'll explore if there's time" never happens. One or two sessions for a new feature go into the sprint plan with their charters; questions from the debriefing go to the backlog.

Common Pitfalls

Calling test design done because coverage is 100 percent. CheckoutService and CouponCodes were at 100 percent and both had bugs. Coverage measures code that runs; it can't measure code that was never written.

Reading the expected result from the code. If you write a test's expected value by looking at the expression in the code to close an empty branch in the coverage report, the line discount = coupon.Percent gets "verified" too. The expected value comes from the rule.

Mistaking exploratory testing for clicking around without a plan. Exploration without a charter, a time-box and notes is an afternoon nobody remembers the next day. The charter gives direction, the time-box sets the cost, and the notes keep the findings from getting lost.

Not turning findings into tests. If a bug found in an exploratory session is fixed without a regression test, the same bug silently comes back in the next refactoring.

Leaving error guessing to the experienced person only. Once the taxonomy and the checklist are written down, someone new to the team asks the same questions. Experience moves from the person to the list.

A Small Experiment

★☆☆ Delete the if (customer == CustomerType.Institutional) check in Discount and regenerate the coverage report. What happens to branch coverage? Which test turns red?

★★☆ Assume the answer to Ayşe's second question: if an institutional customer enters a coupon, the order is still opened, but the response carries the warning "Coupons don't apply to institutional orders.". Extend CheckoutResult; write the test first, then the code.

★★★ Write your own charter for the shipping fee from the fourth post and run a 30-minute session: call ShippingFeeCalculator from a console app or a small endpoint. Split your findings into bugs and questions, and turn each bug into a regression test. Do you need to add a new item to the checklist?

We've seen seven techniques in seven posts. In the last post we bring them all together: when a story arrives, which technique do we pick, how do we remove the duplicates among the tests the techniques produce, and what do we need to check when we ask AI for test cases? The example is the story the Requirements Analysis series left for the second release: partial shipment for institutional orders.