What Code Coverage Tells You and What It Doesn't: Line, Branch Coverage and Mutation Testing

Version note: The code in this post was built and run with .NET 10 (SDK 10.0.402), xUnit.net v3 4.0.2, Microsoft.Testing.Extensions.CodeCoverage 18.12.0, ReportGenerator 5.5.12, Stryker.NET 5.0.0, Angular 22.2 and Vitest 5.1.

Guess First

The Bookstore's pricing rests on three rules: 10 percent off at 500 TL and above, an extra 5 percent for Gold customers, free shipping at 300 TL and above. We write a single test for these rules; it creates price quotes for three different carts but never looks at the result, not one Assert. What will line coverage be? Branch coverage? And a tool that deliberately breaks the code: out of 15 broken versions, for how many will it report that this test went red?

Line, Branch and Method Coverage

Code coverage measures which parts of the code run while the tests run. It has three common units. Line coverage is the share of lines that ran. Branch coverage is whether both directions of every conditional (if, ?:, &&, switch) ran at least once. Method coverage is the share of methods called at least once. A one-line rule shows the difference best:

public static class ShippingFee
{
    public static decimal For(decimal subtotal, bool express) =>
        express ? 60m : subtotal >= 300m ? 0m : 30m;
}

[Fact]
public void Standard_shipping_is_free_over_300() =>
    Assert.Equal(0m, ShippingFee.For(400m, express: false));

When we run only this test with coverage, the relevant lines of the report are:

<line number="23" hits="1" branch="True" condition-coverage="50% (2/4)">

|Bookstore.Pricing.ShippingFee|100%|50%|

The line ran, so line coverage is 100 percent. But only two of the four directions were exercised: express shipping and a cart under 300 TL never ran. Someone looking at line coverage would think this method is "tested"; branch coverage says half of it never ran. That is why, if you pick one of the two, branch coverage is the more honest metric. The coverage page on Microsoft Learn gives the same example: in code with two branches, a test that verifies only one reports 50 percent branch coverage.

Measuring It in .NET: The MTP Extension and ReportGenerator

In a test project that runs on Microsoft Testing Platform (MTP), coverage comes as an extension package:

<ItemGroup>
  <PackageReference Include="xunit.v3" Version="4.0.2" />
  <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="18.12.0" />
</ItemGroup>

Because --coverage is an MTP option, dotnet test has to run in MTP mode; with the .NET 10 SDK that means adding "test": { "runner": "Microsoft.Testing.Platform" } to the global.json at the solution root. As the output format we ask for Cobertura XML (Extensible Markup Language), because that is the common language of reporting tools:

$ dotnet test --coverage --coverage-output-format cobertura --coverage-output coverage.cobertura.xml
Running tests from tests/Bookstore.Pricing.Tests/bin/Debug/net10.0/Bookstore.Pricing.Tests.dll (net10.0|arm64)
tests/Bookstore.Pricing.Tests/bin/Debug/net10.0/Bookstore.Pricing.Tests.dll (net10.0|arm64) passed (184ms)

  In process file artifacts produced:
    - TestResults/coverage.cobertura.xml

To make the XML readable we install ReportGenerator not as a global tool but as a local tool in the repo's tool manifest, so the same version runs in CI (Continuous Integration) and for every developer:

dotnet new tool-manifest
dotnet tool install dotnet-reportgenerator-globaltool --version 5.5.12
dotnet reportgenerator -reports:TestResults/coverage.cobertura.xml -targetdir:coverage-report -reporttypes:"TextSummary;Html"

As an open-source alternative, Coverlet's MTP version coverlet.MTP produces the same Cobertura output; which one you pick makes no difference on the reporting side. The setup details are in the upcoming Unit Test series, and adding a coverage report to the pipeline is covered in the GitHub Actions series post ".NET CI with GitHub Actions: setup-dotnet, global.json, NuGet Cache, Test Reports and Coverage".

Coverage in Angular

In Angular's Vitest-based test builder, coverage is a single flag. After adding the @vitest/coverage-v8 package as a dev dependency and running it:

$ npx ng test --watch=false --coverage
 % Coverage report from v8
---------------|---------|----------|---------|---------|-------------------
File           | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
---------------|---------|----------|---------|---------|-------------------
All files      |   88.37 |    95.65 |   77.77 |   89.47 |
 cart-store.ts |   93.75 |     92.3 |   85.71 |      90 | 33
 cart.ts       |   85.18 |      100 |      50 |   88.88 | 13
---------------|---------|----------|---------|---------|-------------------

89 percent looks good, but there are only two files in the list. app.ts is missing entirely, because no test loads it and the report counts only the files the tests load. When we tell it to look at all the application code, the table changes:

$ npx ng test --watch=false --coverage --coverage-include 'src/app/**/*.ts'
-------------------|---------|----------|---------|---------|-------------------
File               | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-------------------|---------|----------|---------|---------|-------------------
All files          |   63.33 |    95.65 |      50 |      50 |
 app.config.ts     |       0 |      100 |     100 |       0 | 6
 app.routes.ts     |       0 |      100 |     100 |       0 | 3
 app.ts            |       0 |      100 |       0 |       0 | 7-33
 cart-store.ts     |   93.75 |     92.3 |   85.71 |      90 | 33
 cart.ts           |   85.18 |      100 |      50 |   88.88 | 13
 runtime-config.ts |       0 |      100 |     100 |       0 | 8
-------------------|---------|----------|---------|---------|-------------------

Line coverage dropped from 89.47 percent to 50 percent; nothing in the code changed. If you are going to report coverage as a number, pin down the denominator first.

Goodhart's Law: 87.5 Percent Without Assertions

The observation first made by economist Charles Goodhart in 1975 and later summed up by anthropologist Marilyn Strathern: "When a measure becomes a target, it ceases to be a good measure." Let's look at the test from Guess First:

[Fact]
public void Creates_quotes()
{
    PriceQuote.Create(600m, CustomerTier.Gold, express: false);
    PriceQuote.Create(100m, CustomerTier.Standard, express: true);
    PriceQuote.Create(100m, CustomerTier.Standard, express: false);
}

  Line coverage: 87.5%
  Covered lines: 14
  Uncovered lines: 2
  Coverable lines: 16
  Total lines: 35
  Branch coverage: 100% (8 of 8)
  Covered branches: 8
  Total branches: 8
  Method coverage: 66.6% (4 of 6)
  Full method coverage: 66.6% (4 of 6)
  Covered methods: 4
  Fully covered methods: 4
  Total methods: 6

Bookstore.Pricing                    87.5%
  Bookstore.Pricing.DiscountPolicy   88.8%
  Bookstore.Pricing.PriceQuote       83.3%
  Bookstore.Pricing.ShippingFee       100%

The answer to the first two opening questions: 87.5 percent line and 100 percent branch coverage. This test stays green with code that computes the discount wrong, flips the shipping fee or produces a negative total. That is exactly the point of the 2012 "TestCoverage" post by Fowler (Martin Fowler, chief scientist at Thoughtworks): coverage is a good tool for finding untested code and a bad number for measuring how good the tests are. Fowler says that in a team that tests thoughtfully he would expect coverage in the upper 80s or 90s, and he would be suspicious of 100 percent as a sign of tests written to satisfy the metric. In the CI/CD (Continuous Integration / Continuous Delivery) Fundamentals series post "Testing in the Pipeline" we also saw from the pipeline side how a test without assertions inflates coverage.

Mutation Testing: Testing the Tests

Coverage asks "did this line run?" Mutation testing asks "if this line were wrong, would any test notice?" The tool injects small bugs into the code: it turns >= into >, += into -=, replaces a condition with a constant true. Each broken version is called a mutant. The tests run against the mutant; if at least one test fails, the mutant is killed, if they all pass, it survived. The mutation score is the ratio of killed mutants to testable mutants.

In .NET the tool for this is Stryker.NET. We install it as a local tool too and run it from the test project's folder. Since our test project runs on MTP, we choose the runner explicitly; the default is still vstest:

dotnet tool install dotnet-stryker --version 5.0.0
cd tests/Bookstore.Pricing.Tests
dotnet stryker --test-runner mtp -r cleartexttree -r progress -r html

The result for the test without assertions:

[10:16:41 INF] 18 mutants created
[10:16:42 INF] 2     mutants got status NoCoverage.   Reason: Not covered by any test.
[10:16:42 INF] 3     mutants got status Ignored.      Reason: Removed by block already covered filter
[10:16:42 INF] 13    total mutants will be tested
Killed:    1
Survived: 12
[10:16:43 INF] The final mutation score is 6.67 %

The answer to the third question: out of 15 mutants (12 survived, 2 not covered at all, 1 killed) the test failed for only one, a score of 6.67 percent. The single killed mutant is interesting too:

    ├── [Killed] Arithmetic mutation on line 32
    │   ├── [-] subtotal * DiscountPolicy.RateFor(subtotal, tier)
    │   └── [+] subtotal / DiscountPolicy.RateFor(subtotal, tier)

A standard customer's discount rate is 0, the division by zero threw an exception and the test failed. That is the only kind of bug a test without assertions catches: an exception. Behind the 87.5 percent line coverage, all we really knew was "this code doesn't blow up."

The Metric That Finds the Boundaries

Let's write the test properly: three scenarios, each verifying the discount and the total.

[Fact]
public void Gold_customer_over_threshold_gets_fifteen_percent_and_free_shipping()
{
    PriceQuote quote = PriceQuote.Create(600m, CustomerTier.Gold, express: false);

    Assert.Equal(90m, quote.Discount);
    Assert.Equal(510m, quote.Total);
}

The other two verify a 100 TL standard cart with shipping (total 130) and with express (total 160). Coverage is 93.7 percent line, 100 percent branch. Stryker, however, says two mutants are still alive:

Killed:   13
Survived:  2
[10:17:34 INF] The final mutation score is 86.67 %
    ├── [Survived] Equality mutation on line 11
    │   ├── [-] subtotal >= ThresholdAmount
    │   └── [+] subtotal > ThresholdAmount
    ├── [Survived] Equality mutation on line 23
    │   ├── [-] subtotal >= 300m
    │   └── [+] subtotal > 300m

This is something coverage can never show: no test looks at a cart of exactly 500 TL or exactly 300 TL. If someone accidentally turned >= into >, a customer with a 500 TL cart would lose the discount and every test would stay green. Let's add the boundaries:

[Theory]
[InlineData(499.99, 0)]
[InlineData(500, 0.10)]
public void Discount_starts_exactly_at_threshold(decimal subtotal, decimal expected) =>
    Assert.Equal(expected, DiscountPolicy.RateFor(subtotal, CustomerTier.Standard));

The same Theory for shipping with 299.99 and 300. The result:

[10:17:47 INF] 18 mutants created
[10:17:47 INF] 3     mutants got status Ignored.      Reason: Removed by block already covered filter
[10:17:47 INF] 15    total mutants will be tested
Killed:   15
Survived:  0
[10:17:49 INF] The final mutation score is 100.00 %

Line coverage did not change at all in this step; it stayed at 93.7 percent. The quality of the tests went up, and coverage could not see it.

In Production

Use coverage to find untested places. The way of reading it that Fowler suggests: open the report, look at the uncovered lines and ask "does it worry me that this line is untested?" If the signature check in the payment callback is not covered, you should worry; if a DTO's (Data Transfer Object) ToString is not covered, you don't need to.

If you set a target, set it on new code. An 80 percent threshold for the whole codebase pushes a team with old, hard-to-test code into writing meaningless tests. Diff coverage (also called patch coverage or "coverage on new code") looks only at the lines changed in a pull request: "at least 80 percent of the new lines in this PR (pull request) must be tested." Tools such as Codecov and SonarQube report it directly. How to enforce the threshold as a quality gate in the pipeline is in the same CI/CD Fundamentals post.

Mutation testing is slow, so apply it only to changed code as well. For every mutant the relevant tests run again; in a large project a full run can take hours. Stryker's --since option looks at the git diff. On a branch that changes the gift wrap fee:

$ dotnet stryker --test-runner mtp --since:gift-wrap
[10:18:36 INF] 1 files changed
[10:18:36 INF] Changed file src/Bookstore.Pricing/GiftWrap.cs
[10:18:36 INF] 18    mutants got status Ignored.      Reason: Removed by since filter
[10:18:36 INF] 18    total mutants are skipped for the above mentioned reasons
[10:18:36 INF] 5     mutants will be tested because: Mutant changed compared to target commit
[10:18:36 INF] 5     total mutants will be tested
Killed:   3
Survived: 2

Out of 23 mutants only the 5 in the changed file were tested, and the two survivors immediately showed that the new gift wrap rule's fee cap and its 3-book boundary were never tested. One more observation: in the same experiment, when a test file also changed, Stryker re-tested the old file's 15 mutants too, with the reason "One or more covering tests changed". Also, in this small project each run took about 5 seconds and most of that was the build; the gain shows up in large projects where test time exceeds build time. Stryker's MTP runner is still marked as preview in 5.0.0, and it says so with a warning at the start of each run; it is wise not to set a pipeline-breaking threshold (--break-at) before comparing its results with the vstest runner.

Common Pitfalls

Getting tests written because of a coverage threshold. Once the threshold becomes the target, the team writes tests like Creates_quotes above. Coverage goes up, confidence doesn't. If there is a threshold, there should also be a rule to reject tests without assertions in code review.

Counting generated code and DTOs in the denominator. Microsoft's extension already skipped auto-properties and the compiler-generated members of records; when we measured, a DTO with six properties added just one line, for Currency's initial value. But Entity Framework Core (EF Core) migrations, clients generated from an OpenAPI definition and source generator output mean hundreds of lines. Keep them out with the [ExcludeFromCodeCoverage] attribute or tool settings.

Getting hung up on 100 percent. The DiscountPolicy class stayed at 88.8 percent after every test. The reason is the line public const decimal ThresholdAmount = 500m;: a decimal constant is created in IL (Intermediate Language) by a static constructor, but since the value is baked into the places that use it, that constructor never runs. The report shows this line as "uncovered." Writing a test to cover a line like that is pure waste produced by a 100 percent target.

Not checking the denominator. As in the Angular example, a report that skips files that were never loaded can say 89 percent when the real figure is 50 percent.

Turning the mutation score into a new 100 percent target. Some mutants don't change behavior (equivalent mutants), some live in unimportant code. The list of surviving mutants is a to-do list to be read like a coverage report; once it is reduced to a single number, Goodhart's law hits it too.

A Small Experiment

★☆☆ Collect coverage for your own project and examine five of the uncovered lines. How many of them really worry you?

★★☆ Write the tests that kill the two surviving mutants of GiftWrap.FeeFor. What should the fee be for exactly 3 and 4 books?

★★★ Run Stryker.NET on the riskiest file in your project (money, security, frequently changed). Compare its mutation score and list of surviving mutants with the same file's line coverage. Which told you more?

We have reached the end of the Test Strategy series: we covered the types and the two axes, the shape that fits your system, what to test at which level, and how to measure whether your tests actually verify anything. The next step is to put these decisions into code: the upcoming Unit Test series covers the mechanics of unit testing, from setting up a test project with xUnit.net v3 to test doubles and the places where mocks do harm.