Data-Driven Tests in xUnit v3: Theory, MemberData, TheoryData, TheoryDataRow and MatrixTheoryData

Version note: The code in this post was built and run with .NET 10 and xUnit.net v3 4.0.2.

Predict First

A volume discount is coming to Bookstore: 10% on orders of 100 EUR and above, rounded to the cent. The code and the test that checks the rounding:

namespace Bookstore.Pricing;

public sealed class DiscountPolicy
{
    // Volume discount: 10% on orders of 100 EUR and above
    public decimal VolumeDiscount(decimal subtotal) =>
        subtotal >= 100m ? Math.Round(subtotal * 0.10m, 2) : 0m;
}

public sealed class VolumeDiscountTests
{
    private readonly DiscountPolicy policy = new();

    [Theory]
    [InlineData(0, 0)]
    [InlineData(99.99, 0)]
    [InlineData(100, 10)]
    [InlineData(100.01, 10)]
    [InlineData(250, 25)]
    public void Ten_percent_from_100_euros(decimal subtotal, decimal expected)
    {
        Assert.Equal(expected, policy.VolumeDiscount(subtotal));
    }

    public static TheoryData<decimal, decimal> Rounding => new()
    {
        { 123.45m, 12.35m },
        { 100.05m, 10.01m },
        { 199.99m, 20.00m },
    };

    [Theory]
    [MemberData(nameof(Rounding))]
    public void Discount_is_rounded_to_the_cent(decimal subtotal, decimal expected)
    {
        Assert.Equal(expected, policy.VolumeDiscount(subtotal));
    }
}

There are eight rows in total. How many pass? The expected values from accounting were calculated by hand: 10% of 123.45 is 12.345, rounded to the cent that is 12.35.

The Answer: Banker's Rounding

failed Bookstore.Pricing.Tests.VolumeDiscountTests.Discount_is_rounded_to_the_cent(subtotal: 123.45, expected: 12.35) (7ms)
  Assert.Equal() Failure: Values differ
  Expected: 12.35
  Actual:   12.34
failed Bookstore.Pricing.Tests.VolumeDiscountTests.Discount_is_rounded_to_the_cent(subtotal: 100.05, expected: 10.01) (0ms)
  Assert.Equal() Failure: Values differ
  Expected: 10.01
  Actual:   10.00

Test run summary: Failed!
  total: 8
  failed: 2
  succeeded: 6

Two of the eight rows fail, and both are values sitting exactly at the midpoint: 12.345 and 10.005. The default of Math.Round is MidpointRounding.ToEven, also called "banker's rounding": a midpoint value is rounded to the nearest even number. 12.345 → 12.34 (4 is even), 10.005 → 10.00 (0 is even). Accounting expects "round half up". The fix is one line:

subtotal >= 100m ? Math.Round(subtotal * 0.10m, 2, MidpointRounding.AwayFromZero) : 0m;

What found this bug was the choice of data: "random" amounts like 199.99 or 250 would not have shown it. This post explains the mechanics of data-driven tests with the tools of xUnit v3, but the real lesson is this: which rows you write matters more than how many.

InlineData and Choosing Data

[Theory] runs the same test method with different arguments; each row is reported as a separate test, and when one fails the others keep running. It is the exact replacement for the loop test of the previous post. [InlineData] is the simplest source: the values live inside the attribute.

Attribute arguments cannot be decimal in C#, so [InlineData(99.99, 0)] actually carries a double, and xUnit converts it to the parameter type. Values like 99.99 cause no trouble, but to avoid conversion surprises with long fractional values it is safer to write decimal tests with TheoryData.

Two techniques help when choosing rows. Equivalence partitioning: split the input space into classes in which the code behaves the same and pick one representative from each. The volume discount has two classes: below 100 (no discount) and 100 and above (discount). Boundary value analysis: bugs cluster at the edges of the classes, so try the boundary itself and both sides of it. With a threshold of 100 that is 99.99, 100 and 100.01, plus 0 as the lower end. These are exactly the rows that turn red when > is written instead of >=, or when the threshold is 1000 instead of 100.

Rounding is a boundary problem too: ".5" values are the boundary of rounding. That is why a rounding test should include values that sit exactly at the midpoint, such as 12.345 and 10.005. The boundary of a rule is not only a threshold; rounding, an empty collection, the largest value and null are boundaries as well.

MemberData, ClassData and Type-Safe TheoryData

When values do not fit in an attribute (decimal, objects, long lists), [MemberData] points to a static property or method. The Rounding property above returns TheoryData<decimal, decimal>. The difference from the IEnumerable<object[]> often seen in older code is type safety: a row with the wrong type, such as { 123.45m, "12.35" }, does not compile. The xUnit analyzers also compare the types of MemberData with the test parameters.

[ClassData(typeof(...))] takes the same kind of data from a separate class; it helps when the same data is used in several test classes. For data used inside a single class, MemberData is more readable because it keeps the data next to the test. v3 also lets the data source return Task<> or ValueTask<>; if you need to read the data asynchronously from a file, that door is open.

TheoryDataRow: Per-Row Metadata

One of the most useful additions in v3 is that each row can carry its own metadata. TheoryDataRow<...> provides skip, display name, label, timeout, traits and explicit information at row level:

public sealed class TierRateTests
{
    private readonly DiscountPolicy policy = new();

    public static TheoryData<CustomerTier, decimal> TierRates =>
    [
        new TheoryDataRow<CustomerTier, decimal>(CustomerTier.Regular, 0m),
        new TheoryDataRow<CustomerTier, decimal>(CustomerTier.Silver, 0.02m)
        {
            TestDisplayName = "Silver tier gets 2 percent",
        },
        new TheoryDataRow<CustomerTier, decimal>(CustomerTier.Gold, 0.05m)
        {
            Label = "gold",
            Traits = new() { ["Category"] = ["Loyalty"] },
        },
    ];

    [Theory]
    [MemberData(nameof(TierRates))]
    public void Card_payment_uses_tier_rate(CustomerTier tier, decimal expected)
    {
        Assert.Equal(expected, policy.TierRate(tier, PaymentMethod.Card));
    }
}

In the output of dotnet test --list-tests, the three rows show up in three different ways:

  Bookstore.Pricing.Tests.TierRateTests.Card_payment_uses_tier_rate(tier: Regular, expected: 0)
  Silver tier gets 2 percent(tier: Silver, expected: 0.02)
  Bookstore.Pricing.Tests.TierRateTests.Card_payment_uses_tier_rate [gold]

TestDisplayName replaces the method name, and the arguments are still appended. Label, which came with 4.0, replaces the argument list with a short label; it improves readability a lot for rows whose arguments are long objects. Thanks to the trait on the Gold row, dotnet test --filter-trait "Category=Loyalty" ran only that row (total: 1). Skip and Timeout can be given per row in the same way: you can skip a single row with a known bug without switching off the whole theory.

MatrixTheoryData: Combinations

The tier discount has two dimensions: customer tier (Regular, Silver, Gold) and payment method (Card, BankTransfer, GiftCard). No tier discount applies when paying with a gift card. For a rule like "the rate never exceeds 5% in any combination", instead of writing every combination by hand, MatrixTheoryData produces the cross product:

public static MatrixTheoryData<CustomerTier, PaymentMethod> AllTiersWithGiftCard =>
    new([CustomerTier.Regular, CustomerTier.Silver, CustomerTier.Gold], [PaymentMethod.GiftCard]);

public static MatrixTheoryData<CustomerTier, PaymentMethod> EveryCombination =>
    new(Enum.GetValues<CustomerTier>(), Enum.GetValues<PaymentMethod>());

[Theory]
[MemberData(nameof(EveryCombination))]
public void Tier_rate_is_never_above_five_percent(CustomerTier tier, PaymentMethod payment)
{
    Assert.InRange(policy.TierRate(tier, payment), 0m, 0.05m);
}

[Theory]
[MemberData(nameof(AllTiersWithGiftCard))]
public void Gift_card_payment_gets_no_tier_rate(CustomerTier tier, PaymentMethod payment)
{
    Assert.Equal(0m, policy.TierRate(tier, payment));
}

EveryCombination produced 3 × 3 = 9 rows, AllTiersWithGiftCard 3 × 1 = 3 rows. MatrixTheoryData can combine 2 to 5 data sets. The thing to watch is that combinations grow multiplicatively: 5 dimensions with 4 values each is 1024 tests. A matrix is good for invariants that hold "in every combination"; if every combination needs its own expected value, explicit rows are more readable. The more powerful tool for searching for invariants across the whole input space is property-based testing; we will see it with FsCheck in post 12.

Dynamic Skip, Explicit and Serialization

In v2 the only way to skip a test was [Fact(Skip = "...")], fixed at compile time. v3 lets you decide at run time:

public sealed class SkipAndExplicitTests
{
    public static bool IsCi => Environment.GetEnvironmentVariable("CI") == "true";

    [Fact(Skip = "Gift card rules are being rewritten (BS-412)")]
    public void Gift_card_balance_covers_shipping() { }

    [Fact(SkipUnless = nameof(IsCi), Skip = "Runs only on the CI server")]
    public void Price_list_export_matches_the_published_catalog() { }

    [Fact]
    public void Turkish_culture_formats_discount_with_comma()
    {
        Assert.SkipWhen(
            OperatingSystem.IsBrowser(),
            "ICU data is not available in the browser");

        string text = 12.5m.ToString("N2", new System.Globalization.CultureInfo("tr-TR"));

        Assert.Equal("12,50", text);
    }

    [Fact(Explicit = true)]
    public void Recalculate_every_historical_order() { }
}

skipped Bookstore.Pricing.Tests.SkipAndExplicitTests.Price_list_export_matches_the_published_catalog
  Runs only on the CI server
skipped Bookstore.Pricing.Tests.SkipAndExplicitTests.Gift_card_balance_covers_shipping
  Gift card rules are being rewritten (BS-412)
skipped Bookstore.Pricing.Tests.SkipAndExplicitTests.Recalculate_every_historical_order
  Not run (due to explicit test filtering)

SkipUnless and SkipWhen take the name of a public static bool property; the skip message lives in Skip. Setting SkipUnless and SkipWhen together on the same test causes a run-time error. Assert.Skip, Assert.SkipUnless and Assert.SkipWhen make the decision inside the test body. The xUnit documentation points out an important detail: a dynamically skipped test still goes through the whole pipeline, which means the class is created and disposed. Keep that in mind if it has an expensive constructor.

Explicit = true runs the test only when explicitly requested. It suits a long-running recalculation or a check that should only be triggered by hand. On the command line, --explicit on runs everything and --explicit only runs only explicit tests. In the other direction, if you want no skipped tests at all in CI (Continuous Integration), --fail-skips on counts skipped tests as failed.

Finally, serialization. For Test Explorer to run a single row of a theory, xUnit must be able to serialize that row's data during discovery. Built-in types, enums and the Guid, Index, Range and Uri types added in v3 are serializable; your own records are not, by default:

public sealed class BookDiscountTests
{
    public static TheoryData<Book, decimal> Books =>
    [
        (new Book("978-0321125217", "Domain-Driven Design", 55m), 0m),
        (new Book("978-0134757599", "Refactoring", 120m), 12m),
    ];

    [Theory]
    [MemberData(nameof(Books))]
    public void Single_book_volume_discount(Book book, decimal expected)
    {
        Assert.Equal(expected, new DiscountPolicy().VolumeDiscount(book.Price));
    }
}

--list-tests showed this theory as a single row (Bookstore.Pricing.Tests.BookDiscountTests.Single_book_volume_discount), and 28 tests were discovered in total; when run, the two rows ran separately and the total went up to 29. The consequence: you cannot run a single row of this theory from Test Explorer. The fix is either to pass the data as serializable types (making the book's ISBN (International Standard Book Number) and price separate parameters) or to register a serializer for your type through v3's IXunitSerializer extension point. In most cases the first is simpler and makes the test more readable.

In Production

Theory rows are separate tests and their durations add up. Building a 3-dimensional matrix with 10 values each means 1000 tests; even at 2 ms each, that is 2 seconds. The speed of the unit test suite is a feature for the team; use matrices for invariant checks and justify the row count.

Skips are debt. Put a ticket number in the [Fact(Skip = ...)] text (like BS-412) and track the number of skipped tests in CI. A test skipped for months should be either fixed or deleted; a test that stays yellow forever protects nothing.

Culture-dependent tests. For tests that depend on culture, like the tr-TR formatting test, pass the culture explicitly; v3 also lets you fix the culture for all tests with the culture setting in xunit.runner.json. The default is the machine's culture; the test that passes on the developer's machine and fails in CI starts here.

Common Pitfalls

Stuffing unrelated scenarios into a single theory. A theory with five parameters like [InlineData(50, "Gold", "Card", true, 2.5)], where each row tries a different rule, is the loop test in disguise. If there are more than three parameters and the rows try different rules, split them into separate theories by rule.

Computing the expected value with production code. A theory written as Assert.Equal(Math.Round(subtotal * 0.10m, 2), policy.VolumeDiscount(subtotal)) could never have found the bug at the top of this post: the test does the same banker's rounding and approves 12.34. The expected value is a constant calculated by hand by someone who knows the rule.

Choosing only "happy" values. A discount theory written with 150, 200 and 250 has three rows but a single equivalence class. Without boundaries, rounding points and extreme values, the row count gives a misleading sense of confidence.

Expecting to run a single row from Test Explorer with non-serializable data. You notice it not on the day you write the theory but three months later, when you need to debug one row. Simple parameters and building the object inside the test remove this problem.

A Small Experiment

★☆☆ Add a single InlineData row to the Ten_percent_from_100_euros theory that would have caught the bug before the rounding fix. Write down why the row is a boundary value.

★★☆ Add a row to the TierRates data: Gold tier, but a scenario that has to be skipped because of a known bug. Skip only that row with Skip and confirm from the dotnet test output that the others ran.

★★★ Make BookDiscountTests runnable row by row from Test Explorer: first by turning the parameters into simple types, then by writing an IXunitSerializer for Book. Compare the amount of code and the readability of the test names in the two solutions.

In the next post we look at how tests are created and destroyed: a new class instance for every test, IAsyncLifetime, class, collection and assembly fixtures, and the full parallelism that came with 4.0.