Equivalence Partitioning: Splitting Inputs into Classes, Valid and Invalid Partitions, Coverage and Turning Them into xUnit Theories

Version note: The code in this post was built and run with .NET 10 and xUnit.net v3 4.0.3.

Guess First

In Bookstore, the online bookstore used as the shared example across this site's series, every order heading to checkout first goes through a validator. Selin's story has four rules: the order quantity must be positive; an institutional order must contain at least 20 items in total; paying on account is open only to institutional customers; a guest customer pays by credit card only. The validator returns every error it finds at once, because the form shows them all together.

namespace Bookstore.Ordering;

public enum CustomerType
{
    Individual,
    Institutional,
    Guest,
}

public enum PaymentMethod
{
    CreditCard,
    BankTransfer,
    OnAccount,
}

public enum OrderError
{
    UnknownCustomerType,
    QuantityMustBePositive,
    BelowInstitutionalMinimum,
    PaymentMethodNotAllowed,
}

public sealed record OrderRequest(CustomerType Customer, int TotalQuantity, PaymentMethod Payment);

public static class OrderRequestValidator
{
    public static IReadOnlyList<OrderError> Validate(OrderRequest request)
    {
        if (!Enum.IsDefined(request.Customer))
        {
            return [OrderError.UnknownCustomerType];
        }

        List<OrderError> errors = [];
        if (request.TotalQuantity <= 0)
        {
            errors.Add(OrderError.QuantityMustBePositive);
        }
        else if (request.Customer == CustomerType.Institutional
                 && request.TotalQuantity < InstitutionalPricing.MinimumQuantity)
        {
            errors.Add(OrderError.BelowInstitutionalMinimum);
        }

        if (!IsAllowed(request.Customer, request.Payment))
        {
            errors.Add(OrderError.PaymentMethodNotAllowed);
        }

        return errors;
    }

    private static bool IsAllowed(CustomerType customer, PaymentMethod payment) => payment switch
    {
        PaymentMethod.OnAccount => customer == CustomerType.Institutional,
        _ => true,
    };
}

Emre wrote three tests for invalid orders and says he deliberately picked the "hard" cases: every row has more than one thing wrong.

[Theory]
[InlineData(CustomerType.Guest, 0, PaymentMethod.BankTransfer)]
[InlineData(CustomerType.Guest, 5, PaymentMethod.OnAccount)]
[InlineData(CustomerType.Institutional, -1, PaymentMethod.CreditCard)]
public void Invalid_requests_are_rejected(CustomerType customer, int quantity, PaymentMethod payment)
{
    IReadOnlyList<OrderError> errors = OrderRequestValidator.Validate(new OrderRequest(customer, quantity, payment));

    Assert.NotEmpty(errors);
}

All three pass. One of the four rules is missing from the code entirely. Which one, and why don't these three tests notice?

The Answer: One Error Hides Another

The missing rule is the guest's payment restriction: IsAllowed only looks at paying on account and lets a guest pay by bank transfer. All three tests look as if they include this rule, but none of them actually exercises it.

In the first row, the guest orders 0 items and pays by bank transfer. The validator returns QuantityMustBePositive for the 0 items, the list isn't empty, and the test passes. This test can never tell whether the payment rule works; the quantity error got there first. The second row is sneakier: the guest pays on account, and that single value breaks two rules at once. The on-account rule produces the error, and the guest rule stays invisible again. The third row doesn't touch the payment rule at all.

This effect is called defect masking. We saw it in the previous post too: the rounding test failed because of the minimum bug, without ever exercising rounding. When a test has more than one invalid value, its result only tells you something about the value caught first. A weak assertion like Assert.NotEmpty completes the picture: since it doesn't check which error came back, it cannot tell a test that passes for the wrong reason.

The cure is a technique: equivalence partitioning, and its simplest rule, that every invalid partition is exercised on its own in its own test.

Finding Partitions

Equivalence partitioning splits the possible values of an input into groups that the code under test is expected to treat the same way. These groups are called partitions. The assumption behind it is simple: if a value chosen from a partition reveals a defect, any other value from the same partition reveals it too. That's why one test per partition is considered enough. The ISTQB syllabus sets two conditions: partitions must not overlap, and none of them may be empty.

Partitions come from the rules. For the order quantity, the rules say: zero and below are invalid for every customer; 1 to 19 is valid for individuals and guests but invalid for institutions; 20 and above is valid for everyone. Here is the first lesson: whether a partition is valid can depend on another input. 12 items on its own is neither valid nor invalid; together with an institutional customer it falls into an invalid partition. That's why "institutional and 1–19 items" is a partition in its own right.

Enums look easy at first: each value is a partition. For CustomerType that's individual, institutional and guest. But in C# an enum is really a number and can carry an undefined value. If the JSON arriving at the API says "payment": 9, System.Text.Json converts it to (PaymentMethod)9 without complaint; we tried it, and even with JsonStringEnumConverter registered, numeric values are accepted by default. "Undefined enum value" is therefore a real invalid partition, and the one most often forgotten.

The payment method teaches the second lesson: split invalid partitions by rule, not by error code. Paying on account being closed to non-institutional customers is one rule; a guest paying only by card is another. Both produce PaymentMethodNotAllowed, but separate rules live in separate lines of code and can break separately. Had we merged them into a single "payment not allowed" partition and picked paying on account as its representative, we would have missed the missing guest rule again. That is exactly what Emre's second test did.

What valid and invalid mean varies from team to team. The syllabus says so explicitly: for some teams an invalid value is one the code should reject, for others it is one the spec doesn't define at all. In this post an invalid partition means a partition for which the validator should return an error code. What matters is that the team uses the same definition.

The Partition Table

The partitions that come out of the rules, each with an ID:

Fourteen partitions, eight valid and six invalid. The table shows one more thing: the int type protects us from non-numeric input such as "abc". That partition didn't disappear; it just moved to another layer. It now lives in the API's model binding, in the 400 response covered in the ASP.NET Core Web API series. Which partition is tested at which layer is the subject of the Test Strategy series' "What to Test at Which Level?" post.

Each Choice Coverage

In equivalence partitioning the coverage items are the partitions themselves. Coverage is the number of partitions exercised by at least one test divided by the total number of partitions. When there are several inputs, the simplest criterion is Each Choice coverage: every partition of every input appears in at least one test. It doesn't care about combinations; it never asks whether "institutional customer and bank transfer" were tried together.

For valid partitions this saves a lot, because one test can cover one valid partition of each of the three inputs at once. Eight valid partitions are covered by three tests: individual, 5 items, bank transfer (EP1, EP6, EP10); institutional, 25 items, on account (EP2, EP7, EP11); guest, 3 items, credit card (EP3, EP6, EP9). Three tests, because the input with the most partitions has three valid ones.

Invalid partitions get no such saving. Each invalid partition is exercised in its own test, with the other inputs taken from valid partitions. Myers' The Art of Software Testing (third edition, 2011) sets this rule separately for valid and invalid classes: valid classes can be combined in one test, each invalid class needs its own. The reason is the masking we saw above. When one invalid value triggers the validator, the check for a second invalid value in the same test either never runs or its result gets lost behind the first error.

In total: three valid, six invalid, nine tests, and all fourteen partitions. Emre's three tests did touch invalid partitions, but each had more than one invalid value: in both tests that included EP13, another invalid partition produced its error first.

From the Table to TheoryData

The table turns into test code almost as is. We covered the mechanics of TheoryData, MemberData and TheoryDataRow in the Unit Test series' "Data-Driven Tests in xUnit v3" post; here we only look at how the table carries over:

namespace Bookstore.Ordering.Tests;

public sealed class OrderRequestValidatorTests
{
    // Each Choice: every valid partition at least once (EP1-EP3, EP6, EP7, EP9-EP11)
    public static TheoryData<CustomerType, int, PaymentMethod> ValidPartitions => new()
    {
        { CustomerType.Individual, 5, PaymentMethod.BankTransfer },
        { CustomerType.Institutional, 25, PaymentMethod.OnAccount },
        { CustomerType.Guest, 3, PaymentMethod.CreditCard },
    };

    [Theory]
    [MemberData(nameof(ValidPartitions))]
    public void Valid_partitions_are_accepted(CustomerType customer, int quantity, PaymentMethod payment)
    {
        IReadOnlyList<OrderError> errors = OrderRequestValidator.Validate(new OrderRequest(customer, quantity, payment));

        Assert.Empty(errors);
    }

    // One invalid partition per test; all other values come from valid partitions
    public static IEnumerable<TheoryDataRow<CustomerType, int, PaymentMethod, OrderError>> InvalidPartitions =>
    [
        new(CustomerType.Individual, -2, PaymentMethod.CreditCard, OrderError.QuantityMustBePositive) { Label = "EP5" },
        new(CustomerType.Institutional, 12, PaymentMethod.OnAccount, OrderError.BelowInstitutionalMinimum) { Label = "EP8" },
        new(CustomerType.Individual, 5, PaymentMethod.OnAccount, OrderError.PaymentMethodNotAllowed) { Label = "EP12" },
        new(CustomerType.Guest, 3, PaymentMethod.BankTransfer, OrderError.PaymentMethodNotAllowed) { Label = "EP13" },
        new((CustomerType)7, 5, PaymentMethod.CreditCard, OrderError.UnknownCustomerType) { Label = "EP4" },
        new(CustomerType.Individual, 5, (PaymentMethod)9, OrderError.PaymentMethodNotAllowed) { Label = "EP14" },
    ];

    [Theory]
    [MemberData(nameof(InvalidPartitions))]
    public void Each_invalid_partition_is_rejected_with_its_own_error(
        CustomerType customer, int quantity, PaymentMethod payment, OrderError expected)
    {
        IReadOnlyList<OrderError> errors = OrderRequestValidator.Validate(new OrderRequest(customer, quantity, payment));

        Assert.Equal([expected], errors);
    }
}

Two details come from the table. The invalid tests use Assert.Equal([expected], errors) rather than NotEmpty: since exactly one error is expected, the assertion proves the test fails or passes for the right reason. Each row's Label is the partition ID; the test output shows this label instead of the argument list, and you get from a red row back to the table in one step. The representative values were also chosen "from the middle" on purpose: -2, 12, 25. The boundary itself, 0, 1, 19 and 20, is the next post's job.

Against the old code:

failed Bookstore.Ordering.Tests.OrderRequestValidatorTests.Each_invalid_partition_is_rejected_with_its_own_error [EP13] (1ms)
  Assert.Equal() Failure: Collections differ
                              ↓ (pos 0)
  Expected: <generated>      [PaymentMethodNotAllowed]
  Actual:   List<OrderError> []
failed Bookstore.Ordering.Tests.OrderRequestValidatorTests.Each_invalid_partition_is_rejected_with_its_own_error [EP14] (0ms)
  Assert.Equal() Failure: Collections differ
                              ↓ (pos 0)
  Expected: <generated>      [PaymentMethodNotAllowed]
  Actual:   List<OrderError> []

Test run summary: Failed!
  total: 17
  failed: 2
  succeeded: 15
  skipped: 0

Two reds, two separate bugs. EP13 shows the missing guest rule. EP14 found a bug nobody was looking for: an undefined payment method falls into the _ => true arm and is accepted. Tomorrow an order arriving with "payment": 9 in its JSON would have made it all the way to the payment step. The fix is to write IsAllowed over a tuple:

private static bool IsAllowed(CustomerType customer, PaymentMethod payment) => (customer, payment) switch
{
    _ when !Enum.IsDefined(payment) => false,
    (_, PaymentMethod.OnAccount) => customer == CustomerType.Institutional,
    (CustomerType.Guest, _) => payment == PaymentMethod.CreditCard,
    _ => true,
};

The order of the arms is a rule too: undefined values first, on account before the guest rule. If a guest pays on account, both rules give the same result, so the order doesn't change the outcome here; but to know this isn't an accident, that combination needs a test too. I'll leave that to A Small Experiment.

Partitions on the Output Side and in Strings

Partitions aren't only found in inputs. The validator's output splits into partitions too: an empty list, each error code, and more than one error. Every error code appears at least once as an expected result in the tests above. "More than one error" doesn't appear in any test yet, even though that's exactly what the form shows. So we give it its own test:

// Output partition: multiple errors are reported together
[Fact]
public void All_errors_are_reported_together()
{
    IReadOnlyList<OrderError> errors = OrderRequestValidator.Validate(
        new OrderRequest(CustomerType.Guest, 0, PaymentMethod.BankTransfer));

    Assert.Equal([OrderError.QuantityMustBePositive, OrderError.PaymentMethodNotAllowed], errors);
}

Test run summary: Passed!
  total: 18
  failed: 0
  succeeded: 18
  skipped: 0
  duration: 284ms

This test has exactly the same input as Emre's first test. The difference is not in the input but in what the test asserts: here two invalid values arriving together is the very rule under test, and the assertion expects both errors by name. Combining invalid values isn't forbidden; combining them without asserting the outcome precisely is the problem.

For string inputs, format rules define the partitions. Typical partitions for an ISBN are: a valid 13-digit ISBN, a valid ISBN written with hyphens, 13 digits with a wrong check digit, a value containing non-digit characters, a length other than 13, and an empty or null value. Each takes a different code path and is expected to produce a different error message. Which values to try for the "length other than 13" partition, 12, 14 or both, is the question boundary value analysis answers.

Production Notes

Let the partition table live with the code. A table left on a wiki page goes stale at the first rule change. Keeping the IDs as Labels in the test data moves the table into the test class itself: to see which partition has no test, looking at the test file is enough.

When a new enum value is added, the partition table changes too. If PaymentMethod.Wallet arrives tomorrow, the _ => true arm of IsAllowed silently opens it to every customer. Writing the last arm of a switch expression as "everything else is valid" means putting new values into a valid partition by default. In code dealing with money and permissions, prefer the opposite: list what's allowed explicitly and reject the rest.

Invalid partitions multiply at the API boundary. Model binding, JSON converters and libraries such as FluentValidation each take some partitions on themselves. Knowing which partition is rejected at which layer keeps you from testing the same invalid value three times in three layers, or in none.

Rejected requests in production logs are the real-world counterpart of the partition table. A week of logs grouped by error code often reveals a partition that isn't in the table, for example a quantity that isn't negative but is int.MaxValue.

Common Pitfalls

Combining two invalid values in the same test. The main subject of this post: if the test fails, you lose which value triggered it; if it passes, you lose which check ran. Each invalid partition in its own test, with the other inputs valid.

Testing an invalid partition with a weak assertion. Assert.NotEmpty(errors) or Assert.Throws<Exception> won't catch a test that passes for the wrong reason. Assert the expected error code, or the exception type and message, precisely.

Merging partitions by error code. The same error code can come from different rules. Split partitions by rule; two rules producing the same code are two partitions.

Mistaking valid partitions for combinations. Even with 100 percent Each Choice coverage, "institutional customer paying 20 items by bank transfer" may never have been tried. When combinations matter and how to pick them with pairwise is the subject of the sixth post.

Forgetting undefined enum values. A C# enum isn't a closed set; every enum value coming from outside is a number. Without an Enum.IsDefined check and its test, a partition like EP14 is silently treated as valid.

A Small Experiment

★☆☆ Run Emre's three tests against the fixed code; they all still pass. Write down which partitions each row covers, using the IDs in the table, and mark the masked partitions.

★★☆ A guest paying on account breaks two rules at once. Write a separate test for this combination and swap the two arms in IsAllowed. Does the test tell you anything? If not, explain why.

★★★ Add PaymentMethod.Wallet and assume you discussed the rule with Selin: a digital wallet is open to individual and guest customers and closed to institutional ones. Update the partition table first, then the tests, and the code last. Which test would have been red in the old version with the _ => true arm?

In this post we deliberately picked representative values from the middle of the partitions. In the next post we'll do the opposite: with boundary value analysis we'll look at the boundary itself and both sides of it. 19, 20 and 21; the last second of a campaign and the three hours between Istanbul and UTC; 499.99 and 500.00 TRY; ISBNs with 12, 13 and 14 digits.