Combinatorial Testing: Managing Configuration Explosion with Pairwise, Orthogonal Arrays and PICT

Version note: The code in this post was built and run with .NET 10, xUnit.net v3 4.0.3 and PICT 3.7.4.

Guess First

In Bookstore, the online bookstore used as the shared example across this site's series, a store pickup option has been added to checkout. When an order is confirmed, the system decides two things: within how many business days the order leaves for the carrier or the store, and which confirmation e-mail template the customer gets. Four parameters drive the decision: customer type, meaning individual, institutional or guest; payment method, meaning credit card, bank transfer or on account; delivery type, meaning standard, express or store pickup; and language, Turkish or English.

namespace Bookstore.Ordering;

public enum DeliveryType
{
    Standard,
    Express,
    StorePickup,
}

public sealed record CheckoutRequest(CustomerType Customer, PaymentMethod Payment, DeliveryType Delivery, string Language);

public sealed record CheckoutPlan(int DispatchInBusinessDays, string ConfirmationTemplate);

public static class ConfirmationTemplates
{
    // One confirmation e-mail template per delivery type and language
    public static readonly IReadOnlySet<string> Names = new HashSet<string>
    {
        "standard.tr",
        "standard.en",
        "express.tr",
        "express.en",
        "store-pickup.tr",
    };
}

public static class CheckoutPlanner
{
    public const int MaxDispatchInBusinessDays = 3;

    public static CheckoutPlan Plan(CheckoutRequest request)
    {
        int days = request.Delivery == DeliveryType.Express ? 0 : 1;
        if (request.Payment == PaymentMethod.BankTransfer)
        {
            days++; // wait for the bank transfer to clear
        }

        if (request.Customer == CustomerType.Institutional)
        {
            days++; // wait for the institution's purchase approval
        }

        string template = $"{TemplatePrefix(request.Delivery)}.{request.Language}";
        if (!ConfirmationTemplates.Names.Contains(template))
        {
            throw new InvalidOperationException($"No confirmation template named '{template}'.");
        }

        return new CheckoutPlan(days, template);
    }

    private static string TemplatePrefix(DeliveryType delivery) => delivery switch
    {
        DeliveryType.Standard => "standard",
        DeliveryType.Express => "express",
        DeliveryType.StorePickup => "store-pickup",
        _ => throw new ArgumentOutOfRangeException(nameof(delivery)),
    };
}

Emre applies each choice coverage from the second post: every value of every parameter should appear in at least one test. The parameter with the most values has three, so three rows are enough:

Customer	Payment	Delivery	Language
Individual	BankTransfer	StorePickup	tr
Guest	CreditCard	Express	en
Institutional	OnAccount	Standard	tr

Test run summary: Passed!
  total: 3
  failed: 0
  succeeded: 3
  skipped: 0
  duration: 309ms

All eleven values were tried and all three tests are green. Can a customer who picks store pickup in the English UI complete their order?

The Answer: The Bug Is Where Two Values Meet

They can't. When store pickup was added last month, the Turkish confirmation e-mail was written and the English one was forgotten. There is no template named store-pickup.en, and Plan throws an exception. The each choice tests tried StorePickup and they tried English, but never both in the same row.

What triggers the bug isn't a single value but two values arriving together. Equivalence partitioning treated each parameter on its own; decision tables looked at combinations of conditions, but for rules where we know the outcome depends on them. Here the parameters aren't expected to affect each other; no rule was written between language and delivery type. The bug sits at an intersection nobody thought of as a rule. This post's technique, combinatorial testing, exists to cover intersections like this systematically.

The Cost of Every Combination

The first answer that comes to mind is to try every combination. MatrixTheoryData in xUnit v3 does that in one line; its mechanics are in the Unit Test series' "Data-Driven Tests in xUnit v3" post:

// Full combination: 3 x 3 x 3 x 2 = 54 rows, no constraints
public static MatrixTheoryData<CustomerType, PaymentMethod, DeliveryType, string> FullMatrix =>
    new(Enum.GetValues<CustomerType>(), Enum.GetValues<PaymentMethod>(), Enum.GetValues<DeliveryType>(), ["tr", "en"]);

failed Bookstore.Ordering.Tests.CheckoutPlannerComparisonTests.Every_combination_gets_a_plan(customer: Guest, payment: BankTransfer, delivery: StorePickup, language: "en") (0ms)
failed Bookstore.Ordering.Tests.CheckoutPlannerComparisonTests.Every_combination_gets_a_plan(customer: Institutional, payment: BankTransfer, delivery: StorePickup, language: "en") (0ms)
failed Bookstore.Ordering.Tests.CheckoutPlannerComparisonTests.Every_combination_gets_a_plan(customer: Individual, payment: CreditCard, delivery: StorePickup, language: "en") (0ms)

Test run summary: Failed!
  total: 54
  failed: 9
  succeeded: 45
  skipped: 0
  duration: 255ms

Fifty-four combinations, nine red. The bug was caught, but there are two problems. First, six of the nine reds are orders that can't happen in reality: a guest can't pay by bank transfer or on account, on-account payment is only for institutional customers, and institutional orders aren't picked up in store. Twenty-four of the fifty-four rows are impossible combinations like these. Second, the number grows fast. At the unit level 54 tests run in under a second, and a full matrix is a genuine choice here. But when you test the same checkout in a browser on three different engines, the number of valid combinations becomes 90; 90 E2E tests, each walking an order flow end to end, means a pipeline that takes minutes. Every new parameter multiplies this number by its value count.

The Pairwise Assumption, t-Wise and Orthogonal Arrays

Pairwise testing rests on an assumption: most bugs come from the value of a single parameter or from the interaction of two parameters. There is measurement behind this assumption. The 2004 study by Kuhn, Wallace and Gallo at NIST, published in IEEE Transactions on Software Engineering, and later research report that in the systems studied most failures were triggered by one or two parameters, that those needing three or more became progressively rarer, and that the observed interaction degree didn't go beyond six. NIST calls this observation the "interaction rule". The same sources also warn: since some bugs only appear when three or more conditions come together, pairwise on its own isn't always enough.

The goal of pairwise is this: every value pair of any two parameters should appear side by side in at least one test. Because a single test row covers many pairs at once, the number of rows is much smaller than the full combination. The generalization is called t-wise: t = 1 is each choice, t = 2 is pairwise, t = 3 is every triple, and t equal to the number of parameters is the full combination. For this post's model, the numbers produced by the same tool are: each choice 3 rows, pairwise 11, 3-wise 22, full combination 30.

Orthogonal arrays are an older and stricter form of the same idea. The L9 array that Taguchi used in experimental design covers four three-valued parameters with nine rows in a balanced way: every value pair of every two columns appears exactly once. Pairwise asks for "at least once" and doesn't seek balance. The difference matters in practice: an orthogonal array wants every parameter to have the same number of values and doesn't like constraints. Place L9 on the customer type, payment and delivery columns and four of the nine rows turn out to be impossible combinations; deleting them takes the balance and some pairs with them. For a business rule with constraints, a pairwise tool is more practical.

Models and Constraints with PICT

Microsoft's PICT tool reads a plain-text model file and generates the test rows. Its latest release is v3.7.4; the GitHub release ships pict.exe for Windows, and on Linux and macOS it is built from source with CMake. A model file contains the parameters first, then the constraints:

# Bookstore checkout: customer type, payment, delivery and language
Customer: Individual, Institutional, Guest
Payment:  CreditCard, BankTransfer, OnAccount
Delivery: Standard, Express, StorePickup
Language: tr, en

# Guests pay by credit card only; on-account payment is for institutional customers only
IF [Customer] = "Guest" THEN [Payment] = "CreditCard";
IF [Payment] = "OnAccount" THEN [Customer] = "Institutional";
# Institutional orders are not picked up in store
IF [Customer] = "Institutional" THEN [Delivery] <> "StorePickup";

The constraints repeat the rules of OrderRequestValidator from the second post and the delivery rule. The PICT documentation explains well why you write them into the model instead of filtering after generation: if you delete an impossible row afterwards, the valid pairs that row covered are lost as well. A tool that knows the constraints places the valid pairs in other rows.

> pict checkout.pict
Customer	Payment	Delivery	Language
Institutional	BankTransfer	Standard	en
Institutional	OnAccount	Express	tr
Individual	BankTransfer	StorePickup	tr
Individual	CreditCard	Express	en
Institutional	BankTransfer	Express	en
Individual	CreditCard	Standard	tr
Guest	CreditCard	StorePickup	en
Guest	CreditCard	Express	tr
Institutional	CreditCard	Standard	en
Guest	CreditCard	Standard	en
Institutional	OnAccount	Standard	en

> pict checkout.pict /s
Combinations:   40
Generated tests:11
Generation time:0:00:00

The /s option prints the model's statistics: after the constraints there are 40 value pairs to cover, and PICT has packed them into 11 rows. The seventh row is the order where a guest picks store pickup in the English UI. That pair wasn't in each choice's three rows; in pairwise it has to be.

Seeds, Versions and Reproducibility

As long as the model and options stay the same, PICT produces the same output on every run. The /r option randomizes generation and writes the seed it used to the error output; /r:N reproduces the same output with the same seed. The seed also changes the row count, because packing pairs into as few rows as possible is a hard problem and the tool uses a heuristic algorithm:

seed 1: 10
seed 2: 12
seed 3: 10
seed 4: 11
seed 5: 11

All of them cover the same 40 pairs; the only difference is how efficiently they're packed. The version affects the output too: the main branch of PICT on GitHub contains unreleased performance improvements and produces different rows for the same model. That's why we commit the generated file to the repository together with the model file; the test's input depends on a reviewed file, not on how PICT happened to behave that day.

If you want to keep a row no matter what, for example a combination that once found a bug, you give a seed file with the /e option. The file has the same format as PICT's output; PICT places these rows first and builds the remaining pairs on top of them. When we try it with Individual, CreditCard, StorePickup, en, one of the combinations that triggers the bug, that becomes the first row and the total is still 11 rows.

From PICT Output to TheoryData

PICT's output is tab-separated text: the first line holds the parameter names, every following line is a test case. We put the file in the test project and copy it to the build output:

<ItemGroup>
  <None Include="Pict\*.txt" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

MemberData reads the file and turns each line into a TheoryDataRow:

namespace Bookstore.Ordering.Tests;

public sealed class CheckoutPlannerPairwiseTests
{
    // PICT output: the first line holds the parameter names, every following line is a test case (tab-separated)
    public static IEnumerable<TheoryDataRow<CustomerType, PaymentMethod, DeliveryType, string>> Pairs =>
        File.ReadLines(Path.Combine(AppContext.BaseDirectory, "Pict", "checkout-pairs.txt"))
            .Skip(1)
            .Select(line => line.Split('\t'))
            .Select(v => new TheoryDataRow<CustomerType, PaymentMethod, DeliveryType, string>(
                Enum.Parse<CustomerType>(v[0]), Enum.Parse<PaymentMethod>(v[1]), Enum.Parse<DeliveryType>(v[2]), v[3]));

    [Theory]
    [MemberData(nameof(Pairs))]
    public void Every_valid_pair_gets_a_plan(
        CustomerType customer, PaymentMethod payment, DeliveryType delivery, string language)
    {
        CheckoutPlan plan = CheckoutPlanner.Plan(new CheckoutRequest(customer, payment, delivery, language));

        // Oracle: no separate expected value per row; properties that must hold for every combination
        Assert.EndsWith($".{language}", plan.ConfirmationTemplate);
        Assert.InRange(plan.DispatchInBusinessDays, 0, CheckoutPlanner.MaxDispatchInBusinessDays);
    }
}

Enum.Parse ties the values in the model file to the C# types: if a value's name is misspelled in the model, the test stops with a clear error while loading the data. Notice the asserts too: we don't write a separate expected value for each row. The hard part of combinatorial testing is the oracle, that is, what decides whether a test's result is correct. Writing an expected value per row means rewriting the decision table. Instead we check properties that must hold for every combination: a plan is produced, the confirmation e-mail is in the customer's language, and the order leaves within three business days at the latest. Pairwise scans the combinations; the correct result of each combination is tested by the techniques from the earlier posts.

failed Bookstore.Ordering.Tests.CheckoutPlannerPairwiseTests.Every_valid_pair_gets_a_plan(customer: Guest, payment: CreditCard, delivery: StorePickup, language: "en") (0ms)
  Xunit.MicrosoftTestingPlatform.XunitException: System.InvalidOperationException : No confirmation template named 'store-pickup.en'.

Test run summary: Failed!
  total: 11
  failed: 1
  succeeded: 10
  skipped: 0
  duration: 379ms

One of the eleven rows is red, and the red row carries exactly the missing pair. The fix is adding the missing template:

"store-pickup.tr",
"store-pickup.en",

Test run summary: Passed!
  total: 131
  failed: 0
  succeeded: 131
  skipped: 0
  duration: 397ms

The Browser Dimension and Playwright Projects

At the unit level the difference between 11 and 54 is a few milliseconds. The real gain of pairwise is where each row is expensive: E2E tests running in a browser. We add one more parameter at the top of the model:

# Bookstore checkout E2E: browser, customer type, payment, delivery and language
Browser:  Chromium, Firefox, WebKit
Customer: Individual, Institutional, Guest

The constraints are the same. The valid full combination is 90 rows, pairwise is 13 rows. In Playwright Test, browsers are usually defined as projects in playwright.config.ts, and every test runs once per project; the details are in the Integration and E2E Testing series' "Getting Started with Playwright" post. There are two ways to go. The first is to take the browser out of the model: 11 rows run in three projects, making 33 tests, and every row is tried in every browser. The second is to keep the browser in the model: 13 rows are generated and each project runs only its own browser's rows; the tests compare the browser name in the row with the project name and skip the others. The second way runs fewer tests but treats the browser like any other parameter: on-account payment gets tried in Firefox and may not be tried in WebKit. Where browser differences are a risk, such as the 3D Secure redirect on the payment page, the first way makes sense; where the browser is an ordinary parameter, the second.

Production Notes

Put the model file through review. Constraints are business rules; a wrong constraint silently leaves a valid combination out of testing. If the line "institutional orders are not picked up in store" changes one day, the model must change too, or the new valid combinations will never be tested.

Commit the generated file and regenerate it when you change the model. If the test input is generated on the fly during a build, the test set changes silently whenever the PICT version or the seed changes. Pinning rows that once caught a bug with /e keeps you from losing them in a new generation.

Build the value lists with equivalence partitioning. PICT doesn't find the values itself; every value in the model represents a partition. If you add a continuous parameter such as the cart total to the model, write a few of the boundary values from the third post as its values.

Keep negative tests separate. When a value is prefixed with ~, PICT treats it as invalid and puts at most one invalid value in each row; it solves the masking problem from the second post at the tool level. Still, because invalid values have a different expected outcome, keeping them in a separate model and a separate test is more readable.

Common Pitfalls

Using pairwise in place of full testing of critical rules. Pairwise tries each pair once; for rules about money, permissions and refunds, once isn't enough. The decision table for the shipping fee and the boundary values for the institutional discount stay where they are. Pairwise is added on top of them, to scan the intersections nobody wrote down as rules.

Skipping constraints and filtering the output by hand. Deleting impossible rows afterwards also deletes the valid pairs covered in those rows. What's left is no longer pairwise, and nothing tells you so.

Trying to write a separate expected value for every row. As the number of combinations grows, this turns into rewriting the decision table in test code. The oracle of combinatorial testing is usually properties that hold for every row.

Treating pairwise as a guarantee. A bug triggered by three parameters together may not be caught by a pairwise set. You can raise risky parameters to 3-wise in a sub-model or add critical triples as seed rows.

A Small Experiment

★☆☆ Run the model with /o:3. How many rows come out? Which row has the store pickup, English and bank transfer triple?

★★☆ Add a Coupon: None, FreeShipping, Percent10 parameter and the constraint "express can't be chosen for an order paid by bank transfer" to the model. How much does the pairwise row count change? How much does the full combination count change?

★★★ Add a bug to CheckoutPlanner that makes dispatch four days for an institutional express order paid by bank transfer. How many parameters does this bug depend on coming together? Does the pairwise set catch it? If it doesn't, make it catch the bug by adding a sub-model or a seed row to the model.

So far we've always derived test cases from requirements and models. In the next post we change direction: first we'll look at the code and find skipped branches in a branch coverage report, then move to experience-based techniques: error guessing, checklists and a one-hour exploratory testing session for checkout.