Flaky E2E Tests: Isolation, Parallelism, Retries, Network Mocking and Visual Comparison

Version note: The code in this post was built and run with .NET 10, Angular 22.2, Playwright 1.65, @axe-core/playwright 4.14, Testcontainers for .NET 4.16 and SQL Server 2025.

Guess First

The search test from post 6 failed twice in the first runs and passed when run again. We found the cause of one in post 7: a book created by another test running in parallel. The other was in the very first, cold-start run, and I had not kept the error message. My suspicion was a race between the request sent when the page opens and the search request. When the catalog page opens it requests all books, and when the search button is pressed it requests the search:

constructor() {
  this.search('');
}

protected search(term: string): void {
  this.books.set(null);
  this.api.search(term).subscribe((books) => this.books.set(books));
}

To imitate an API that starts slowly, I delayed the full-list request sent on page open by 150 milliseconds with page.route; I left the search request alone:

await page.route(
  (url) => url.pathname === '/api/books' && url.searchParams.get('search') === '',
  async (route) => {
    await new Promise((resolve) => setTimeout(resolve, delay));
    await route.continue();
  },
);

await catalog.open();
await catalog.search(book.title.toLowerCase());

await expect(catalog.results).toHaveCount(1);
await expect(catalog.results.getByRole('heading')).toHaveText(book.title);

How many times does this test fail when run six times? Zero, six or something in between?

The Answer: It Was the Application, Not the Test, That Was Flaky

Three of the six runs failed:

Error: expect(locator).toHaveCount(expected) failed

Locator:  getByRole('list', { name: 'Search results' }).getByRole('listitem')
Expected: 1
Received: 57
Timeout:  5000ms

Call log:
  - Expect "toHaveCount" getByRole('list', { name: 'Search results' }).getByRole('listitem') with timeout 5000ms
  - waiting for getByRole('list', { name: 'Search results' }).getByRole('listitem')
    2 × locator resolved to 0 elements
      - unexpected value "0"
    12 × locator resolved to 57 elements
       - unexpected value "57"

57 is every book in the database. Two requests are in flight at the same time: the full list from page open and the search. search opens a new subscription each time, and whichever response arrives last stays on the screen. If the search response arrives first, the list shows a single book for a moment, then the delayed full list overwrites it. Whether the test failed depended on the order of the two responses and on the moment the assertion sampled the list. The same thing happens to a real user: a user searching on a slow connection sees the result for a moment, then the list goes back to the full list on its own.

The fix is in the application:

protected search(term: string): void {
  this.books.set(null);
  // Cancel the previous request if it is still running; a late response must not overwrite the new result
  this.pending?.unsubscribe();
  this.pending = this.api.search(term).subscribe((books) => this.books.set(books));
}

When the subscription of an HttpClient request is closed, the request is cancelled and its response is never processed. After the fix, the test variants with delays of 0, 150, 300, 600 and 1200 milliseconds passed in 37 out of 37 runs. That is the first question to ask before labeling a flaky test a "test problem": is the test catching nondeterminism that a user could also run into?

In his article "Eradicating Non-Determinism in Tests" (2011), Martin Fowler groups the sources of nondeterminism into lack of isolation, asynchronous behavior, remote services, time and resource leaks. In E2E, animation and platform differences join them. The ones I ran into in Bookstore:

Time: Time Zones and page.clock

The order page says "You can cancel this order until 04:54."; the time is the order's creation time plus 30 minutes, formatted in the browser's time zone. Since Bookstore's users are in Türkiye, timezoneId: 'Europe/Istanbul' was added to the config; the browser opens with this time zone on every machine. The first test I wrote computed the expected value like this:

const deadline = new Date(Date.parse(order.createdAt) + 30 * 60 * 1000);
const expected = deadline.toLocaleTimeString('en-GB', { hour: '2-digit', minute: '2-digit' });
await expect(page.getByText(`You can cancel this order until ${expected}.`)).toBeVisible();

It passed on my machine. CI runners usually run in UTC; running the same test with TZ=UTC gave:

Error: expect(locator).toBeVisible() failed
Locator: getByText('You can cancel this order until 01:54.')
Expected: visible
Timeout: 5000ms
Error: element(s) not found

The page's accessibility tree said paragraph: You can cancel this order until 04:54.. The browser used Istanbul time, while the test process used its own time zone. This is the most typical case of "works on my machine": the test depended on a value that changes from machine to machine, the process's time zone. The fix is to compute the expected value in the browser's time zone as well:

// Compute the expected time in the browser's time zone; don't rely on the test process's time zone
const expected = deadline.toLocaleTimeString('en-GB', { hour: '2-digit', minute: '2-digit', timeZone: timezoneId });

timezoneId comes from the same constant as the config. The test also passed with TZ=UTC and TZ=America/New_York.

When you need to control time itself, Playwright has page.clock. The toast shown after adding to the cart disappears after three seconds; testing that by really waiting three seconds is both slow and unstable at the boundaries. The clock is installed before the page opens and the test moves time forward itself:

test('the cart toast disappears after three seconds', async ({ data, catalog, page }) => {
  const book = await data.book('Emma', 39.5);
  await page.clock.install();

  await catalog.open();
  await catalog.search(book.title);
  await catalog.addToCart(book.title);
  await expect(page.getByRole('status')).toBeVisible();

  await page.clock.runFor(2_900);
  await expect(page.getByRole('status')).toBeVisible();
  await page.clock.runFor(200);
  await expect(page.getByRole('status')).toBeHidden();
});

The test finished in 543 milliseconds and states precisely that the toast is still visible at 2.9 seconds and gone at 3.1 seconds. install replaces Date, setTimeout and friends in the browser with fakes; time still flows, but it can be fast-forwarded with runFor. According to the documentation, install must happen before any other clock-related calls. This clock is only the browser's clock; the API's time does not change. Server-side time is controlled in the integration tests with FakeTimeProvider from post 4.

Parallelism, Workers and Sharding

Playwright runs tests in parallel in separate worker processes; the default number of workers is half the number of logical CPU cores. fullyParallel: true spreads tests from the same file as well. With the servers up, I measured the suite with different worker counts (11 tests, including setup):

workers=1   11 passed (6.1s)
workers=2   11 passed (4.1s)
workers=4   11 passed (3.4s)
default     11 passed (3.3s)

Beyond four workers there is no gain; a single API and a single database are shared, and the longest test (the order flow) takes close to two seconds on its own. Parallelism tests the assumption that the tests are independent of each other: the "Dune" failure in post 7 only showed up in a parallel run. If a group of tests really has to run in order, there is test.describe.configure({ mode: 'serial' }), but that is usually a sign that the tests share some state.

Sharding splits the suite across several machines:

$ npx playwright test --shard=1/2
Running 6 tests using 5 workers, shard 1 of 2
  6 passed (1.9s)
$ npx playwright test --shard=2/2
Running 6 tests using 5 workers, shard 2 of 2
  6 passed (2.0s)

11 tests became 12 across two shards: looking with --list shows that the setup project is in both shards, because each shard runs the project it depends on itself. Each shard gets its own HTML report; in CI, the shards are given the blob reporter and the results are merged into a single report with npx playwright merge-reports. Since 1.65, a run in which no tests are found in any shard fails with "No tests found". Bookstore's E2E suite does not need sharding at this size; since the container and ng serve start-up is paid again in each shard, two shards spend more machine time in total.

What Do Retries Hide?

The config has retries: 2 for CI. I ran the search bug from before the fix ten times, delaying the initial load randomly between 0 and 300 milliseconds:

$ npx playwright test race2 --repeat-each 10 --retries 2
  ✘   2 [chromium] › e2e/race2.spec.ts:3:1 › search survives a slow initial load (5.5s)
  ✓  12 [chromium] › e2e/race2.spec.ts:3:1 › search survives a slow initial load (retry #1) (398ms)
  ...
  ✘  14 [chromium] › e2e/race2.spec.ts:3:1 › search survives a slow initial load (retry #1) (5.4s)
  ✘  16 [chromium] › e2e/race2.spec.ts:3:1 › search survives a slow initial load (retry #2) (5.3s)

  1 failed
  3 flaky
  7 passed (18.7s)

Playwright counts a test that fails on the first attempt and passes on a retry separately as "flaky". The report is honest, but it is not the pipeline's decision: in three of the ten runs the retry hid the failure completely, and without the single remaining "failed", the run would have ended successfully with the default settings. A real bug in the application would have stayed a yellow label in the report. The way to prevent that is the --fail-on-flaky-tests flag or failOnFlakyTests: true in the config; every flaky test turns the run red.

The value of retries is in diagnosis: with trace: 'on-first-retry', the attempt right after a failed first attempt is recorded, and you get the trace of the flaky test. My rule is this: retries are on in CI, the flaky count is reported on every run, a test that turns out flaky is linked to an issue and quarantined (in Playwright with test.fixme, or with a tag in a separate project), and it stays there until fixed. The flaky policy in the pipeline is in the CI/CD Fundamentals series post "Testing in the Pipeline: Fast Feedback, Quality Gates and Flaky Tests".

page.route: When Is It Legitimate?

In this post I used page.route for three purposes, and all three are legitimate. The first is reproducing timing: the delay above made an ordering that happened only occasionally on cold start reproducible every time. The second is failure states that are hard to produce in the end-to-end setup. The fake payment provider in the E2E host always approves; to test what the customer sees during a payment outage, I change the response on the browser side:

test('a payment outage is shown to the customer', async ({ data, catalog, cart, page }) => {
  const book = await data.book('Emma', 39.5);
  // The fake payment provider always approves in E2E; we produce the failure on the browser side
  await page.route('**/api/orders', (route) =>
    route.fulfill({
      status: 503,
      contentType: 'application/problem+json',
      json: { status: 503, title: 'Payment provider unavailable' },
    }),
  );

  await catalog.open();
  await catalog.search(book.title);
  await catalog.addToCart(book.title);
  await cart.open();
  await cart.placeOrder('Izmir');

  await expect(cart.error).toHaveText('Payment provider unavailable');
  await expect(page.getByRole('button', { name: 'Place order' })).toBeEnabled();
});

This test does not verify that the API produces a 503, but how Angular shows a 503. That the API really returns 503 with this title during a payment outage is in the integration test from post 4; together, the two tests close the chain. The third use is fixing the data of visual tests; that is in the next section.

The line is this: the mocked request must not be the subject of the test. If you mock /api/orders while testing the order flow, you are no longer testing the API, the database or the contract between them; what remains is a slow component test running in a browser. An E2E suite that mocks the entire API becomes a more expensive copy of the component tests in the Frontend Testing series.

Visual Comparison and Accessibility

toHaveScreenshot compares a screenshot of the page with a baseline pixel by pixel. My first attempt used live data: the test created and searched for its own book and compared the page. The first run wrote the baseline, the second failed:

Error: expect(page).toHaveScreenshot(expected) failed

  281 pixels (ratio 0.01 of all image pixels) are different.

The difference was the unique suffix in the book title; the very thing that separates tests from each other in post 7 makes the visual test different on every run. A visual test needs fixed data. Here page.route is legitimate, because the subject of the test is not the data itself but how it looks:

// A visual test needs fixed data: the book list comes from this array instead of the API
const books = [
  { id: 1, isbn: '978-0441172719', title: 'Dune', price: 64.9, stock: 7 },
  { id: 2, isbn: '978-0141439587', title: 'Emma', price: 39.5, stock: 3 },
  { id: 4, isbn: '978-0141439549', title: 'Middlemarch', price: 48.75, stock: 5 },
  { id: 3, isbn: '978-0547928227', title: 'The Hobbit', price: 52, stock: 0 },
];

test.beforeEach(async ({ page }) => {
  await page.route((url) => url.pathname === '/api/books', (route) => route.fulfill({ json: books }));
  await page.goto('/');
  await expect(page.getByRole('list', { name: 'Search results' }).getByRole('listitem')).toHaveCount(4);
});

test('catalog page', async ({ page }) => {
  await expect(page).toHaveScreenshot('catalog.png');
});

test('toast after adding to cart', async ({ page }) => {
  await page.getByRole('listitem').filter({ hasText: 'Emma' }).getByRole('button', { name: 'Add to cart' }).click();
  await expect(page).toHaveScreenshot('catalog-toast.png');
});

The sliding toast caused no trouble: toHaveScreenshot disables animations by default (finite animations are fast-forwarded to the end, infinite ones are reset to their initial state) and waits until it gets two identical screenshots in a row. In the toast test's log, intermediate screenshots showed that wait. The default threshold is 0.2; it decides how large a color difference must be for two pixels to count as "different". maxDiffPixels and maxDiffPixelRatio decide how many different pixels are allowed; both are unset by default, so a single pixel breaks the test.

The snapshot file name carries the browser and the platform: catalog-chromium-win32.png on Windows, catalog-chromium-linux.png on Linux. I copied a baseline produced on a developer machine into Playwright's Linux Docker image (mcr.microsoft.com/playwright:v1.65.0-noble) and compared:

  ✘  1 [chromium] › e2e/visual.spec.ts:17:1 › catalog page (464ms)
  ✘  2 [chromium] › e2e/visual.spec.ts:21:1 › toast after adding to cart (714ms)
    Error: expect(page).toHaveScreenshot(expected) failed
      26966 pixels (ratio 0.03 of all image pixels) are different.

Same page, same data, same Chromium version: the whole difference comes from font rendering. Silencing it with maxDiffPixelRatio: 0.05 is tempting, but when I changed the button color in the accessibility fix below, the visual test reported a difference of 8410 pixels (1%). A threshold that tolerates the platform difference also misses a real design change. The solution is to generate the baselines in the environment CI uses and to run the visual tests only there. In Bookstore the visual tests run with a separate config, serving the built application with a small static server; the main config leaves them out with testIgnore:

// Visual tests run only in Playwright's Linux image; baselines are generated in that environment
export default defineConfig({
  testDir: './e2e',
  testMatch: /visual\.spec\.ts/,
  reporter: [['list'], ['html', { open: 'never' }]],
  use: { baseURL: 'http://localhost:4300', timezoneId },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
  webServer: {
    command: 'node e2e/support/static-server.mjs dist/bookstore-web/browser 4300',
    url: 'http://localhost:4300',
  },
});

$ npm run build -- --configuration development
$ docker run --rm -v ${PWD}:/work -w /work mcr.microsoft.com/playwright:v1.65.0-noble npx playwright test -c playwright.visual.config.ts --update-snapshots
A snapshot doesn't exist at /work/e2e/visual.spec.ts-snapshots/catalog-chromium-linux.png, writing actual.
  ✓  1 [chromium] › e2e/visual.spec.ts:17:1 › catalog page (255ms)
A snapshot doesn't exist at /work/e2e/visual.spec.ts-snapshots/catalog-toast-chromium-linux.png, writing actual.
  ✓  2 [chromium] › e2e/visual.spec.ts:21:1 › toast after adding to cart (584ms)
  2 passed (1.2s)

${PWD} works in PowerShell too. Running again without the flag, six runs out of six passed. The baselines are committed; a changed baseline shows up in the pull request as an image diff, and the reviewer confirms that the change was intentional.

A quick accessibility check with axe

@axe-core/playwright runs the open-source axe engine on the page's current DOM:

test('the catalog page has no detectable WCAG A/AA violations', async ({ page }) => {
  await page.goto('/');
  await expect(page.getByRole('list', { name: 'Search results' })).toBeVisible();

  const results = await new AxeBuilder({ page }).withTags(['wcag2a', 'wcag2aa']).analyze();

  expect(results.violations.map((v) => `${v.id}: ${v.nodes.length} node(s)`)).toEqual([]);
});

- Array []
+ Array [
+   "color-contrast: 173 node(s)",
+ ]

WCAG requires a contrast of 4.5:1 for normal-size text; the white button text on the blue background was 2.77:1. The 173 elements were the "Add to cart" buttons in a catalog filled with test data and the "Search" button. When I darkened the button color the axe test passed and both visual tests caught the change; I updated the baselines in the container on purpose. An automated check can find only part of the WCAG violations: keyboard flow, focus order and the screen reader experience need manual testing. The details are in the Accessibility (a11y) series.

In Production

Measure the flaky rate and make it visible. Playwright's report counts flaky tests separately; writing that number somewhere on every run (the CI summary, a dashboard) shows the trend. Running changed tests a few times with --repeat-each 10 is a cheap check to see whether a new test is flaky before merging; the --only-changed flag selects only the changed test files.

Don't raise timeouts as a fix for flakiness. The race above would have failed at the same rate with a 5-second expect timeout as with 30 seconds; waiting does not change anything once the list has settled in the wrong state. A timeout should be raised only for an operation that is really slow but finishes correctly.

Keep the number of visual tests small. Every screenshot is a maintenance cost: a font, a browser version or a small CSS change means a baseline update. In Bookstore only two screens have visual tests; for screens whose design changes often, role and text assertions are enough.

Common Pitfalls

Making retries the permanent fix. Retries turn the report green but don't fix the failure; in the example above, what the retry hid was not a weakness in the test but a real race condition in the application. With failOnFlakyTests off, flaky tests pile up over time and the team stops reading the report.

Mocking everything and turning E2E into a component test. You get a fast and stable suite, but a break in the API contract, a missing migration or a wrong authorization setting no longer shows up in any test. Mocks are only for situations that are not the subject of the test and cannot be produced in the real setup.

Deleting a flaky test or forgetting it with skip. A quarantine needs a deadline and an owner; otherwise coverage shrinks silently. Look at the trace before deleting: both flaky tests in this post said something about the application or the test data.

Setting the visual test threshold according to the platform difference. A threshold that tolerates a 3% platform difference misses a 1% color change. Fix the environment, not the threshold.

A Small Experiment

★☆☆ Revert the unsubscribe line in catalog-page.ts and run the delay test with values between 0 and 1200 milliseconds using --repeat-each 10. At which delay does it fail most? Why do very short and very long delays fail less?

★★☆ Write a deterministic regression test for the search race: verify with page.on('requestfailed') that the delayed initial load request is cancelled. Without waitForTimeout.

★★★ Design a GitHub Actions job for CI: two shards, the blob reporter, a single HTML report with merge-reports, and a step that writes the number of flaky tests to the job summary. Run the visual tests as a separate job in the same image.

Throughout the series we used WebApplicationFactory, Testcontainers and Playwright in their current versions. But most tests in a team were written before these versions. The last post is about reading old integration and E2E tests: WebApplicationFactory from .NET 6 to 11, Testcontainers 3 to 4, Protractor to Playwright, and the breaking changes in Playwright's recent versions.