Testing in the Pipeline: Fast Feedback, Quality Gates and Flaky Tests

Version note: The examples in this post were run with .NET 10 (SDK 10.0.402), xUnit 2.9.3, coverlet 10.1.0, Testcontainers 4.16.0, PostgreSQL 18, Angular 22.2, Playwright 1.65 and act 0.2.97.

Guess First

You add a single test to Bookstore's test project that contains no Assert at all: it sends requests to three endpoints and never looks at the result. What happens to line coverage and branch coverage? Second question: two tests that calculate prices fail in 2 out of 30 runs. Does adding a "retry twice if it fails" setting to the pipeline solve the problem? You will see the answers in the real output in the middle of the post.

Turning the Test Pyramid into Pipeline Order

What the test pyramid says is simple: many fast, cheap tests at the bottom, a few slow, expensive tests at the top. In a pipeline this becomes an ordering; the cheapest check goes first, because the earlier a failure is caught, the faster the feedback. The ten-minute build target in Fowler's CI (Continuous Integration) article comes from the same place: the time in which a developer can see the result "before moving on to something else" after a commit.

The layers in Bookstore and the times I measured in my runs:

The numbers are small because Bookstore is small. The proportions stay similar in real projects: e2e (end-to-end) tests become the biggest item of pipeline time even though there are few of them. Giving each layer a time budget works well: "if unit tests go over 2 minutes or integration over 5 minutes, we talk".

Integration Tests: Against a Real PostgreSQL

A unit test isolates the logic of the code; but only a real database tells you whether the SQL (Structured Query Language) is written correctly and whether the migration creates the table correctly. Imitating PostgreSQL with an in-memory provider or SQLite misses PostgreSQL-specific behaviour. Testcontainers starts a real PostgreSQL container when the test begins and removes it when it ends:

using Bookstore.Api.Data;
using Npgsql;
using Testcontainers.PostgreSql;

namespace Bookstore.Api.IntegrationTests;

public sealed class BookStockRepositoryTests : IAsyncLifetime
{
    // fixed name and port so the local run does not clash with other containers; drop these two lines in CI
    private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder("postgres:18-alpine")
        .WithName("cicdwork-pg-it")
        .WithPortBinding(18120, 5432)
        .Build();

    public async Task InitializeAsync()
    {
        await _postgres.StartAsync();
        await using NpgsqlDataSource dataSource = NpgsqlDataSource.Create(_postgres.GetConnectionString());
        await using NpgsqlCommand command = dataSource.CreateCommand("""
            CREATE TABLE books (id int PRIMARY KEY, title text NOT NULL, stock int NOT NULL);
            INSERT INTO books VALUES (1, 'Refactoring', 12), (2, 'Continuous Delivery', 4), (3, 'Accelerate', 0);
            """);
        await command.ExecuteNonQueryAsync();
    }

    public Task DisposeAsync() => _postgres.DisposeAsync().AsTask();

    [Fact]
    public async Task CountInStock_ignores_sold_out_books()
    {
        await using NpgsqlDataSource dataSource = NpgsqlDataSource.Create(_postgres.GetConnectionString());
        BookStockRepository repository = new(dataSource);

        long count = await repository.CountInStockAsync();

        Assert.Equal(2, count);
    }
}

I am showing the code exactly as I ran it locally. The WithName and WithPortBinding lines are only there to avoid clashing with other containers on my machine; in a pipeline a fixed port means two jobs running in parallel on the same runner break each other. Without those two lines Testcontainers picks a free port and GetConnectionString() returns it.

$ dotnet test tests/Bookstore.Api.IntegrationTests
Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 1 s - Bookstore.Api.IntegrationTests.dll (net10.0)

One second, including starting the container. If the image is not on the runner, the first run adds the download time; that shows why caching the image in the integration job, or choosing a small (alpine) image, matters. I covered testing the HTTP (Hypertext Transfer Protocol) layer end to end with WebApplicationFactory in the ASP.NET Core Web API (Application Programming Interface) series post "Integration Testing an ASP.NET Core Web API: End-to-End Tests with WebApplicationFactory".

Lining the Layers Up in the Pipeline

Bookstore's test workflow puts the layers in separate jobs:

name: test

on:
  pull_request:
  push:
    branches: [main]

jobs:
  unit:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          global-json-file: global.json
      - run: dotnet format --verify-no-changes
      - run: dotnet build -c Release -warnaserror
      - run: >-
          dotnet test tests/Bookstore.Api.Tests -c Release --no-build
          -p:CollectCoverage=true -p:Threshold=80 -p:ThresholdType=line

  integration:
    needs: unit
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          global-json-file: global.json
      - run: dotnet test tests/Bookstore.Api.IntegrationTests -c Release

  e2e:
    needs: unit
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      matrix:
        shard: [1, 2]
    defaults:
      run:
        working-directory: web/bookstore-web
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - if: github.event_name == 'pull_request'
        run: npx playwright test --grep @smoke --shard=${{ matrix.shard }}/2
      - if: github.event_name == 'push'
        run: npx playwright test --shard=${{ matrix.shard }}/2

integration and e2e both depend only on unit, so once the unit layer is green they start in parallel. On a PR only the e2e tests tagged @smoke run; on a push to main, all of them. I ran the unit job with act:

[test/unit]   ✅  Success - Main dotnet format --verify-no-changes [11.370395917s]
[test/unit]   |     0 Warning(s)
[test/unit]   |     0 Error(s)
[test/unit]   ✅  Success - Main dotnet build -c Release -warnaserror [3.084910625s]
[test/unit]   | Passed!  - Failed:     0, Passed:     6, Skipped:     0, Total:     6, Duration: 115 ms - Bookstore.Api.Tests.dll (net10.0)
[test/unit]   | | Bookstore.Api | 81.25% | 50%    | 62.5%  |
[test/unit]   ✅  Success - Main dotnet test tests/Bookstore.Api.Tests -c Release --no-build -p:CollectCoverage=true -p:Threshold=80 -p:ThresholdType=line [1.325313333s]
[test/unit] 🏁  Job succeeded

I did not run the integration and e2e jobs in act: the first needs access to Docker from inside the job container, the second needs browser dependencies installed. I ran their test commands locally, as you can see above and below; the whole workflow passed actionlint. GitHub's Ubuntu runners come with Docker ready, so Testcontainers needs no extra setup there.

Look at why the cheap check comes first: dotnet format took 11 seconds, the tests 1.3 seconds. Here the format check looks more expensive than the tests; but on a commit with a formatting error, build and test never start, and in real projects test time is measured in minutes while the format check stays in seconds.

Quality Gates

A quality gate enforces a quality condition beyond passing tests, in a way that breaks the pipeline. Bookstore has four.

Turning warnings into errors

I added a line that produces a nullable reference type warning and built first normally, then with -warnaserror. The second build quietly succeeded, because the project had not changed and the compiler never ran; the incremental build did not reproduce the warning. With --no-incremental:

$ dotnet build src/Bookstore.Api -c Release -warnaserror --no-incremental
src/Bookstore.Api/Catalog/TitleFormatter.cs(5,45): error CS8602: Dereference of a possibly null reference. [src/Bookstore.Api/Bookstore.Api.csproj]
    1 Error(s)

Every pipeline job starts from a clean checkout, so this problem does not exist there; but it may be one reason a developer says "I don't get the warning locally". Putting TreatWarningsAsErrors into Directory.Build.props to turn warnings into errors permanently is more consistent than a pipeline-only flag.

A coverage threshold, and why it misleads on its own

With coverlet and the threshold raised to 85, the local run:

$ dotnet test tests/Bookstore.Api.Tests -p:CollectCoverage=true -p:Threshold=85 -p:ThresholdType=line
| Total   | 80%  | 50%    | 62.5%  |
.../coverlet.msbuild/10.1.0/buildMultiTargeting/coverlet.msbuild.targets(73,5): error : The minimum line coverage is below the specified 85 [tests/Bookstore.Api.Tests/Bookstore.Api.Tests.csproj]

Now let's add the test from the opening question:

[Fact]
public async Task Endpoints_do_not_throw()
{
    HttpClient client = factory.CreateClient();
    await client.GetAsync("/api/books");
    await client.GetAsync("/api/books/1");
    await client.GetAsync("/api/info");
}

Passed!  - Failed:     0, Passed:     7, Skipped:     0, Total:     7, Duration: 149 ms - Bookstore.Api.Tests.dll (net10.0)
| Module        | Line   | Branch | Method |
| Bookstore.Api | 82.85% | 83.33% | 75%    |

Line coverage went from 80 to 82.85, branch coverage from 50 to 83.33. The test verifies nothing; it passes even if the endpoint returns 500. Coverage measures "which lines ran", not "which lines were verified". A threshold is a useful alarm for finding areas without tests; once it becomes a target, it produces tests like this one. Keeping the threshold a little below the current value to prevent a drop is healthier than forcing a high number.

Bundle budget

On the Angular side the same idea applies to bundle size. When I lowered the budget in angular.json to 150 kB to try it out:

$ npx ng build
Application bundle generation failed. [0.757 seconds] - 2026-11-08T03:33:13.367Z
✘ [ERROR] bundle initial exceeded maximum budget. Budget 150.00 kB was not met by 41.16 kB with a total of 191.16 kB.

A budget catches someone adding a 2 MB library without noticing. I covered the details, together with SPA (Single Page Application) routing, in the Angular series post "Angular Build and Deploy: ng build, Environment Files, Budgets and SPA Routing".

Making the result visible on the PR

Searching for the failing test among 300 lines of a red job's log is a waste of time. That is what actions which produce test results as JUnit XML (Extensible Markup Language) or TRX (Visual Studio Test Results) and show a summary on the PR are for, along with Jenkins' junit step. In the Playwright config I turned on the JUnit report in CI: reporter: process.env['CI'] ? [['list'], ['junit', { outputFile: 'test-results/e2e.xml' }]] : 'list'.

Flaky Tests

A flaky test is one that sometimes passes and sometimes fails without the code changing. Bookstore had two tests for the campaign discount:

public static class Campaign
{
    // global, mutable state shared by every test in the process
    public static decimal DiscountRate { get; set; }

    public static decimal Apply(decimal price) => Math.Round(price * (1 - DiscountRate), 2);
}

public class BlackFridayTests
{
    [Fact]
    public async Task Black_friday_price_is_discounted()
    {
        Campaign.DiscountRate = 0.20m;
        await Task.Delay(10);
        Assert.Equal(40.00m, Campaign.Apply(50.00m));
        Campaign.DiscountRate = 0m;
    }
}

public class RegularPriceTests
{
    [Fact]
    public void Regular_price_is_not_discounted() =>
        Assert.Equal(50.00m, Campaign.Apply(50.00m));
}

Each test is correct on its own. But xUnit runs different test classes in parallel by default; if RegularPriceTests runs right when BlackFridayTests has set the discount rate, it sees the wrong price. The simplest way to detect a flaky test is to run the same test many times against unchanged code:

fails=0
for i in $(seq 1 30); do
    dotnet test -c Release --no-build > run$i.log 2>&1 || fails=$((fails+1))
done
echo "failed runs: $fails/30"

failed runs: 2/30

  Failed Bookstore.Pricing.Tests.RegularPriceTests.Regular_price_is_not_discounted [3 ms]
   Assert.Equal() Failure: Values differ
Expected: 50.00
Actual:   40.00
Failed!  - Failed:     1, Passed:     1, Skipped:     0, Total:     2, Duration: 15 ms - Bookstore.Pricing.Tests.dll (net10.0)

2 out of 30 runs, about 7 percent. Back to the opening question: if you add "retry twice" to the pipeline, most red builds disappear, but the bug does not. A retry hides a race condition in the test code; if the same race exists in production code (static state, a shared cache), it hides that too. A retry is a plaster: reasonable for transient infrastructure problems (a registry returning 503 for a moment), not for a problem in the test logic.

The real fix is to remove the shared state:

public sealed class Campaign(decimal discountRate)
{
    public decimal Apply(decimal price) => Math.Round(price * (1 - discountRate), 2);
}

The tests now create their own Campaign instance. The same loop this time gave failed runs: 0/30.

If the fix cannot happen right away, quarantine: mark the test with [Trait("Category", "Quarantine")], exclude it from the main pipeline with --filter "Category!=Quarantine", keep running it in a separate job, and give every quarantined test an owner and a deadline. Quarantine without an owner is a polite name for quietly deleting the test.

Parallelisation and Sharding

When test time grows, splitting the tests across several machines (sharding) shortens wall-clock time. Playwright's --shard option does this, but it has a trap. Bookstore's two e2e tests are in the same file; with fullyParallel off:

$ CI=1 npx playwright test --shard=1/2
Running 2 tests using 1 worker, shard 1 of 2
  ✓  1 e2e/home.spec.ts:3:1 › home page shows the store name @smoke (215ms)
  ✓  2 e2e/home.spec.ts:8:1 › home page reads the runtime config (107ms)
  2 passed (2.6s)
$ CI=1 npx playwright test --shard=2/2
[WebServer] Prebundling has been configured but will not be used because caching has been disabled.
$ echo $?
0

With this setting Playwright splits tests by file; since there is only one file, the second shard ran no tests and exited with 0. In a pipeline that means a job that looks green but does nothing. When I added fullyParallel: true to the config, the tests were split per test:

Running 1 test using 1 worker, shard 1 of 2
  ✓  1 e2e/home.spec.ts:3:1 › home page shows the store name @smoke (217ms)
  1 passed (2.5s)
Running 1 test using 1 worker, shard 2 of 2
  ✓  1 e2e/home.spec.ts:8:1 › home page reads the runtime config (132ms)
  1 passed (2.3s)

When you move to sharding, verify that each shard really runs tests, for example by adding up the test counts in the reports.

In Production

Keep a history of test results. Whether a single red run is flaky or real can only be told by the pass rate of the same test over its last 50 runs. Most CI tools have plugins or test analytics features that show this history.

Watch out for time-dependent tests. A test that looks at DateTime.Now may fail after 14:00 or at the end of the month. Using the TimeProvider abstraction that came with .NET 8 in the code and passing a FakeTimeProvider in the test prevents this class of flaky test from the start.

By default Testcontainers starts a small helper container called Ryuk for cleanup; it removes the containers even if the test process crashes. In my local runs I disposed the containers explicitly and did not want an extra container, so I turned it off with TESTCONTAINERS_RYUK_DISABLED=true. Leave it on in CI, so that a job cut off halfway does not leave orphaned database containers behind.

Common Pitfalls

Running the whole e2e suite on every commit: the pipeline grows to 40 minutes and developers stop waiting. Smoke on PRs, the full set on main and at night, is a good balance for most teams.

Skipping integration tests on PRs: when people say "it's too slow, let it run on main", SQL and migration errors are only caught after the merge and main breaks.

Making the coverage percentage a target: as you saw above, the number is easily inflated with tests that verify nothing.

Adding retries to everything: it makes flaky tests invisible and postpones fixing them forever.

Not verifying sharding: an empty shard finishes green too.

A Small Experiment

★☆☆ Measure coverage in your own test project, then add a test without assertions and measure again. How much did it go up?

★★☆ Run your own test suite 30 times with the loop script above. Did any test fail? If so, was the cause static state, time or ordering?

★★★ Move one of your project's integration tests to Testcontainers and add it to the pipeline as a separate job. Compare the durations of the unit job and the integration job and write a time budget for each.

In the next post we will look at where the artifact goes: the dev, test, staging and production environments, promoting the same artifact between them, and why "it worked on staging" is not enough.