A Safety Net Before Refactoring: Characterization Tests, Approval Testing and Finding Blind Spots

Version note: The code in this post was built and run with C# 14 and .NET 10 (SDK 10.0.402), with tests on xUnit.net v3 4.0.3, Verify.XunitV3 32.0.0 and Microsoft.Testing.Extensions.CodeCoverage 18.12.0.

Guess First: An Invoice Without Tests

Bookstore's InvoiceRenderer class builds the text of an order's invoice: header, item lines, gift wrap, coupon, shipping, tax, payment instructions. The production file is 400 lines; in this post we work with a 111-line shortened version that preserves its behavior, and all output comes from that version. It has no tests at all. It starts like this:

public string Render(Order order)
{
    var sb = new StringBuilder();
    var now = DateTime.Now;
    sb.AppendLine("BOOKSTORE INVOICE");
    sb.AppendLine("Invoice no: INV-" + now.ToString("yyyyMMdd") + "-" + order.Id.ToString("D6"));
    sb.AppendLine("Date: " + now.ToString("d"));
    sb.AppendLine("Customer: " + order.Customer.Name + " <" + order.Customer.Email + ">");
    if (order.Customer.IsVip)
        sb.AppendLine("VIP customer - thank you for your loyalty!");
    sb.AppendLine(new string('-', 52));
    // ...
    foreach (var item in order.Items)
    {
        // ...
        var lineTotal = item.Book.Price * item.Quantity;
        subtotal += lineTotal;
        if (item.Book.Format == BookFormat.Ebook)
            tax += lineTotal * 0.20m;
        else
            tax += lineTotal * 0.10m;
        // ...
        sb.AppendLine(title.PadRight(26) + item.Quantity.ToString().PadLeft(3) + " x " + item.Book.Price.ToString("N2").PadLeft(8) + lineTotal.ToString("N2").PadLeft(12));
    }
    // ... coupon, shipping, totals, payment instructions

The task is to break this class apart in the coming posts. But first we need a safety net that tells us whether we broke something when we touch it. Two questions: what should the first test you write for this method verify? And how many things do you see in this output that could turn a test that is green today red tomorrow, or on another machine?

Characterization Tests: What Does the Code Do Today?

In his 2004 book Working Effectively with Legacy Code, Michael Feathers calls these characterization tests: tests that record what the code actually does today rather than what it should do. The goal isn't to prove correctness but to notice change. If the code contains a bug, the test pins that bug as well; this is a deliberate choice, because during a refactoring the only thing we want is for behavior not to change.

The simplest way to write a characterization test by hand is to write it without knowing the expected value and let the test tell you. We start with an empty expectation:

[Fact]
public void Total_line_of_a_domestic_order()
{
    string text = new InvoiceRenderer().Render(Samples.DomesticOrder());
    string totalLine = text.Split(Environment.NewLine).Single(l => l.StartsWith("Total"));

    Assert.Equal("", totalLine);
}

Assert.Equal() Failure: Strings differ
Expected: ""
Actual:   "Total                                        1,007"···
           ↑ (pos 0)

The failure message gives the real value; paste it into the expectation and the test goes green, and from then on any change to the total line is caught. This works well for a single value, but the invoice is 20 lines of text, and pinning each line with its own assertion is tiring and misses the relationships between lines (which line is there, which one is missing).

Approval Testing: Approve the Whole Output Once

Approval testing (or the golden master approach) scales the same idea: save the whole output to a file, have a human look at it once and approve it, and on every later run compare the new output with the approved file. In .NET the established tool for this is Verify. Its setup and the snapshot review flow were covered in the Unit Test series post "Snapshot and Property-Based Testing: Extending Unit Tests with Verify and FsCheck"; that post chose 32.0.0 because of the maintenance fee check in the 33.x versions, and we use the same version here.

[Fact]
public Task Domestic_order_invoice()
{
    string text = new InvoiceRenderer().Render(Samples.DomesticOrder());
    return Verify(text);
}

On the first run there is no file to compare against, so the test goes red and Verify leaves a .received.txt file:

VerifyException : Directory: tests/Bookstore.Invoicing.Tests
New:
  - Received: InvoiceApprovalTests.Domestic_order_invoice.received.txt
    Verified: InvoiceApprovalTests.Domestic_order_invoice.verified.txt

BOOKSTORE INVOICE
Invoice no: INV-20261114-000042
Date: 11/14/2026
Customer: Ada Lovelace <ada@example.com>
----------------------------------------------------
Dune                        1 x   450.00      450.00
Emma                        2 x   180.00      360.00
  + gift wrap                                    9.80
Sapiens: A Brief Hist...    1 x   300.00      300.00
----------------------------------------------------
Subtotal                                     1,119.80
Discount (WELCOME10)                          -111.98
Shipping                                         FREE
Total                                        1,007.82
  incl. tax                                    141.00
Categories: Science Fiction, Classics, History
Please pay by bank transfer until 11/21/2026.
Reference: 000042

When you have read the file and said "yes, this is today's invoice", you save it as .verified.txt and commit it to the repo. .received.txt files go into .gitignore. The test is green now. But the second opening question is still unanswered.

The Answer: Green Today, Red Tomorrow

I ran the same test again with the system time zone set to UTC. On my clock the date was 14 November; in UTC it was still 13 November:

$ TZ=UTC dotnet test
NotEqual:
  - Received: InvoiceApprovalTests.Domestic_order_invoice.received.txt
    Verified: InvoiceApprovalTests.Domestic_order_invoice.verified.txt

Received: InvoiceApprovalTests.Domestic_order_invoice.received.txt
BOOKSTORE INVOICE
Invoice no: INV-20261113-000042
Date: 11/13/2026
...
Please pay by bank transfer until 11/20/2026.

If your CI server runs in UTC, this test breaks every night at certain hours, and the next day it breaks everywhere anyway. The second problem is culture. Running the same test on a machine with Turkish regional settings writes numbers and dates differently:

Date: 14.11.2026
...
Subtotal                                     1.119,80
Discount (WELCOME10)                          -111,98
Total                                        1.007,82

So the opening output had at least three fragile points: the date in the invoice number, the invoice date and the bank transfer due date; plus the culture-dependent format of numbers and dates. The fix has two layers. We pin the culture in the test project; in xUnit.net v3 an xunit.runner.json file is enough:

{
  "culture": "en-US"
}

The dates we remove from the output before comparison, without touching the production code. Verify's scrubbers exist for this:

return Verify(text)
    .ScrubInlineDates("yyyyMMdd", CultureInfo.InvariantCulture)
    .ScrubInlineDates("M/d/yyyy", CultureInfo.InvariantCulture);

Invoice no: INV-Date_1-000042
Date: Date_1
...
Please pay by bank transfer until Date_2.

Verify gives the same date the same number: the date in the invoice number and the invoice date are the same day, so both become Date_1; the due date is a different day, so it becomes Date_2. The fact "these two dates are equal" is kept, but "the due date is exactly seven days later" is lost. That's a compromise. The permanent fix is to give the code its clock from outside; in the fifth post, once DateTime.Now moves to TimeProvider, these scrubbers become unnecessary. For now the goal is to build the net without touching the code. The test is now green in UTC, in Turkish culture and in the Los Angeles time zone.

Non-deterministic output has other sources too: rows from an SQL query without ORDER BY, the file order of Directory.EnumerateFiles, a collection processed in parallel, randomly generated Guids. The question is the same each time: is this difference a real behavior change, or noise in the measurement? If it is noise, it gets scrubbed or sorted; if it is behavior, it stays in the snapshot.

Sweeping the Input Space with Combinations

A single invoice pins a single path. InvoiceRenderer's behavior, however, branches on country, coupon, quantity and payment method. Instead of writing a separate test for every combination, we collect the cartesian product of the inputs in one snapshot with Verify's combination support:

[Fact]
public Task Totals_for_country_coupon_quantity_and_payment() =>
    Combination()
        .Verify(
            RenderTotals,
            ["TR", "DE", "US"],
            [null, "WELCOME10", "FIX40", "BOGUS"],
            [1, 3],
            ["card", "transfer"])
        .ScrubInlineDates("M/d/yyyy", CultureInfo.InvariantCulture);

RenderTotals builds an order for each combination, renders the invoice and squeezes only the coupon message and the totals block into one line; the item lines are already approved in the main snapshot. A few lines of the 3 × 4 × 2 × 2 = 48-line approved file:

TR, FIX40    , 3, card    : Subtotal 540.00 | Discount (FIX40) -40.00 | Shipping FREE | Total 500.00 | incl. tax 54.00 | Paid by card.,
TR, BOGUS    , 3, card    : Coupon BOGUS is not valid and was ignored. | ---------------------------------------------------- | Subtotal 540.00 | Shipping FREE | Total 540.00 | incl. tax 54.00 | Paid by card.,

Choosing input values is a design task. A "normal" value, a boundary value and an invalid value for each parameter is a good start. Here FIX40 combined with three copies of Emma produces a net amount of exactly 500.00, right on the free shipping threshold. The mutation section below shows how I found that value and why it was needed.

Finding Blind Spots: Coverage

We built the net, but we don't know where its holes are. The first measure is coverage. We add the Microsoft Testing Platform coverage extension from the Test Strategy series post "What Code Coverage Tells You and What It Doesn't: Line, Branch Coverage and Mutation Testing" to the test project and collect coverage in Cobertura format:

dotnet test --coverage --coverage-output-format cobertura --coverage-output coverage.cobertura.xml

The numbers in the report say little on their own; we need to know which lines never ran. We get that with a small script without a project file, using .NET 10's file-based app support:

using System.Xml.Linq;

// Usage: dotnet run tools/uncovered.cs -- <cobertura.xml> <FileName.cs>
string reportPath = args[0];
string fileName = args[1];

XDocument report = XDocument.Load(reportPath);
XElement file = report.Descendants("class")
    .First(c => ((string?)c.Attribute("filename"))?.EndsWith(fileName) == true);
string[] source = File.ReadAllLines((string)file.Attribute("filename")!);

foreach (XElement line in file.Descendants("line").DistinctBy(l => (int)l.Attribute("number")!))
{
    int number = (int)line.Attribute("number")!;
    int hits = (int)line.Attribute("hits")!;
    string? branches = (string?)line.Attribute("condition-coverage");
    bool partial = branches is not null && !branches.StartsWith("100%");
    if (hits == 0 || partial)
    {
        string label = hits == 0 ? "never run" : $"branches {branches}";
        Console.WriteLine($"{number,4} {label,-22} {source[number - 1].Trim()}");
    }
}

With the first invoice and the first combination set (which then had only country, coupon and payment method), InvoiceRenderer's line coverage came out at 86.7 percent and branch coverage at 88 percent. The script's list:

  15 branches 50% (1/2)     if (order.Customer.IsVip)
  16 never run              sb.AppendLine("VIP customer - thank you for your loyalty!");
  19 branches 50% (1/2)     if (order.Items.Count == 0)
  20 never run              {
  21 never run              sb.AppendLine("No items.");
  22 never run              return sb.ToString();
  46 branches 50% (1/2)     if (item.Book.Format != BookFormat.Print)
  47 never run              {
  48 never run              sb.AppendLine("  (gift wrap not available for digital items)");
  49 never run              }
  67 branches 50% (1/2)     if (discount > subtotal)
  68 never run              discount = subtotal;
  97 branches 50% (1/2)     else if (order.PaymentMethod == "transfer")
 103 never run              {
 104 never run              sb.AppendLine("Payment method: " + order.PaymentMethod);
 105 never run              }
 107 branches 50% (1/2)     if (physical == 0)
 108 never run              sb.AppendLine("Your download links have been sent to " + order.Customer.Email + ".");

The list reads like a shopping list: a VIP customer, an empty order, gift wrap on a digital item, a coupon larger than the order, a payment method other than card or transfer, an order with only digital items. Two orders cover most of it: a VIP order made of an ebook and an audiobook, with gift wrap requested, a fixed 900 coupon and paid by "voucher", plus an empty order. After two new snapshots, line and branch coverage are both 100 percent.

Testing the Tests: Deliberate Mutations

100 percent coverage says every line ran at least once; it doesn't say an assertion checked the result of that line. To see this, I planted small bugs in the code on purpose and ran the tests after each one. This is mutation testing done by hand; Stryker.NET does the same work automatically and was covered in the Test Strategy post.

The invalid coupon message line showed up as executed in the coverage report: the BOGUS combinations ran it. But at that point RenderTotals took the output from the Subtotal line onwards, and the message came before it; nobody looked at the output of the line that ran. The boundary mutation survived because no input landed on exactly 500. I changed the combination test based on these two findings: included the coupon message, added the quantity dimension and the FIX40 coupon that makes 540 − 40 = 500. When I applied the two mutations again, both turned the combination test red:

failed Bookstore.Invoicing.Tests.InvoiceCombinationTests.Totals_for_country_coupon_quantity_and_payment
Test run summary: Failed!

A few deliberate mutations tell you a lot about the quality of the net, even before you install Stryker.NET. Trying two or three mutations in the area you plan to change, say in the shipping code if you are about to extract shipping, is cheap insurance.

Taking Down the Scaffolding: From Characterization Tests to Behavior Tests

Characterization tests are like construction scaffolding: they protect you while you change the structure, but they aren't designed to stay. A snapshot says "this is what the invoice looks like today"; it doesn't say "free shipping starts at 500". When it turns red, you have to read the diff to understand what changed and why.

Once the net is in place, refactoring can begin. As a first step I extracted the shipping calculation into a separate class called ShippingFees; this commit changes only production code and the five snapshot tests stay green. Right after that, in a separate commit, we describe the rule, which can now be called directly, with named tests:

[Theory]
[InlineData(500.00, 0)]
[InlineData(499.99, 39.90)]
public void Domestic_shipping_is_free_from_500(decimal netAmount, decimal expected) =>
    Assert.Equal(expected, ShippingFees.For("TR", printedBooks: 1, netAmount));

[Fact]
public void Eu_shipping_adds_two_per_extra_printed_book() =>
    Assert.Equal(16.50m, ShippingFees.For("DE", printedBooks: 3, netAmount: 100m));

$ git log --oneline -2
354ca18 test: describe shipping rules with behavior tests
4af1750 refactor: extract ShippingFees from InvoiceRenderer

As the refactoring progresses, each rule gets its own behavior test this way and the load carried by the snapshots shrinks. In the end one or two snapshots remain to protect the invoice layout (alignment, line order); the calculation rules are described by named tests, not snapshots. Deleting broad snapshots like the combination test once the refactoring is done is a legitimate decision too.

Common Pitfalls

Approving non-deterministic output. If dates, time zones, culture, Guids or collections without guaranteed ordering end up in the snapshot, the test will one day break on its own. After a while the team starts approving red snapshots without thinking and the net stops working. Running the test once in a different time zone and culture before the first approval catches most of these problems.

Forgetting that you pinned wrong behavior as "correct". In the approved invoice of the digital VIP order the tax line says 82.00. By Bookstore's rule audiobooks are digital products too and are taxed at 20 percent; the correct value would be 104.00, because the code only checks for Ebook. On top of that it shows 82.00 tax on a total of 0.00. The job of a characterization test is to preserve this bug; but if you don't write it down somewhere, in a few months the snapshot will read as "correct behavior". Record the oddities you find in an issue and make the fix separately from the refactoring, in its own commit, updating the snapshot deliberately.

Approving a .received file without reading it. The "accept all" convenience that comes with a diff tool approves a wrong output with one click as well. Approval is a review: look at every line, and if you are unsure about one, go to the production code.

A snapshot that is too broad or too narrow. Approving the whole order object as JSON breaks tests even when an unrelated property is added. Approving only the total line misses behaviors like the coupon message. The snapshot should draw the boundary of the behavior you want to protect.

Treating coverage as the goal. When we reached 100 percent coverage, two mutations were still alive. Coverage is a tool that shows which code doesn't run; it doesn't show which code is protected.

A Small Experiment

★☆☆ Pick a method in your own project that produces text or JSON and has no tests. First pin one of its lines with the Assert.Equal("", ...) technique, then approve the same output as a whole with Verify. Compare the maintenance cost of the two.

★★☆ Run your approved snapshot test again with TZ=UTC and Turkish culture, and on Windows in a session where you changed them with Set-TimeZone and Set-Culture in PowerShell. How many different breaking points did you find, and for each one did you choose scrubbing, pinning, or a seam in the production code?

★★★ Reproduce this post's mutation table in your own code: make five mutations in the area you plan to change and run the tests after each one. What is the smallest input change that kills the surviving mutations? Then run Stryker.NET on the same area and compare with what you found by hand.

The safety net is ready. In the next post we'll open up PricingEngine.Calculate step by step with the IDE's automated refactorings: compiling and testing after every step, moving callers gradually with parallel change when a signature changes, and recognizing the places where automated tools aren't safe.