Async Flakiness & Determinism

1 The Hook

A team at a fictional NZ logistics company, Kahu Freight, had a Playwright suite of 300 tests. It was green on every developer’s machine. But in CI, roughly one run in five went red — and never the same test twice. One day it was the checkout test, the next the login test, the next a dashboard assertion. Re-running the pipeline usually went green.

So the team did the natural thing: they turned on retries. retries: 2 in the config. Now a test that failed got two more shots, and the pipeline went green almost every time. Problem solved — or so it looked. Velocity was fine, the board was green, everyone moved on.

Three months later a real bug shipped to production: under load, the checkout page occasionally rendered the “Pay” button before the cart total finished loading, and customers could submit a payment for $0.00. When the team went back through their history, they found something uncomfortable: the checkout test had been failing intermittently in CI for months — it had actually been catching this exact race condition — but the retries swallowed the failure every time. The flaky test wasn’t noise. It was a real defect waving a flag, and the team had taped over the flag.

That is the whole lesson in one story. Flakiness is not a nuisance to be silenced with retries — it is a signal that your test and your app disagree about when something is ready. The job is not to make the red go away. The job is to make the test deterministic: same input, same result, every single time. This lesson teaches you why tests go flaky and how to build waits that don’t.

2 The Rule

3 The Analogy

4 Why Tests Go Flaky

Flakiness feels mysterious because it’s intermittent, but it comes from a short list of causes. Almost all of them are a mismatch between the test’s timing and the app’s timing. Learn the categories and you can name any flake on sight:

• Timing / race conditions (the big one): the test acts or asserts before the app has finished an async operation — a fetch, a render, an animation. Fast machines hide it; a loaded CI runner exposes it. This is the majority of all flakiness.
• Hard-coded waits: sleep(2000) is a bet that the app will always be ready in two seconds. On a slow run it isn’t, and the test fails; on a fast run you waste two seconds. Either way it’s wrong.
• Shared state / test order dependence: test B only passes if test A ran first and left data behind. Run them in a different order, or in parallel, and B fails. Each test must set up and tear down its own world.
• Non-deterministic test data: a test that depends on “today’s date,” a random value, or a record that another test might also touch. The data shifts under the test.
• Animations and transitions: the element is in the DOM but still sliding into place, so a click lands on the wrong spot or is intercepted.
• Network variance: a real third-party call that’s usually fast but occasionally slow, or returns slightly different data.

5 The Hard-Wait Anti-Pattern

The instinct of every new automator hitting a flaky test is to add a sleep. It even works — for a while. This is the single most damaging habit in UI automation, so look hard at why it’s wrong.

This code says: wait exactly two seconds, then check. It is wrong in both directions at once. If the total loads in 2.5 seconds on a slow CI run, the test fails even though the app is perfectly fine — a false red. If it loads in 100ms, you’ve thrown away 1.9 seconds for nothing — multiply that across 300 tests and your suite crawls. A fixed wait is a guess about timing, and the app’s timing is exactly the thing you can’t guarantee.

The fix is a web-first assertion: an assertion that automatically retries until the condition is true or a timeout is hit. It waits for the state you care about, not for a number of milliseconds.

This version passes as soon as the total is correct — whether that takes 100ms or 2.5s — and only fails if it never becomes correct within the timeout. It is faster on quick runs and robust on slow ones. The lesson: never wait for time; always wait for a condition. Playwright’s expect, Cypress’s built-in retry-ability, and Selenium’s explicit WebDriverWait with expected conditions all exist precisely so you never have to sleep.

6 Deterministic Wait Architecture

Killing sleeps is the start. A genuinely stable suite is designed to be deterministic. Five principles:

• Assert on conditions, never on time. Use web-first / auto-retrying assertions everywhere. If you ever type waitForTimeout or sleep, treat it as a bug to be removed.
• Wait for the right signal. Wait for the element to be actionable (visible, stable, enabled) — or for the network response, or for a loading spinner to disappear — not just for it to exist in the DOM. “Present” and “ready” are different states.
• Isolate state. Every test creates the data it needs and cleans up after itself. No test depends on another test’s leftovers or on running in a particular order. This is what makes parallel execution safe.
• Control the non-deterministic inputs. Freeze the clock for date-dependent tests, seed any randomness, and pin test data. Don’t let “today’ or Math.random() decide whether your test passes.
• Mock the unstable edges. For UI tests, stub flaky third-party calls so the test exercises your code, not the reliability of someone else’s API. Test the real integration separately, deliberately, in its own suite.

Put together, these turn the “pass 50 times in a row” challenge from luck into design. A test that waits for conditions, owns its data, and controls its inputs has no reason to vary — so it doesn’t.

7 The Retry Trap

Test runners let you auto-retry failed tests, and used wrongly that feature is how Kahu Freight shipped a $0.00 checkout. So be precise about what retries are for.

A retry that turns red into green hides information. If a test only passes on the second attempt, something is non-deterministic — and that something might be a real bug in the app, not in the test. Blanket retries paper over exactly the signal you built the test to catch.

Retries have a legitimate, narrow use: as a safety net and a measurement tool, not a fix. The healthy pattern is:

• Track flakiness, don’t silence it. Configure the runner to record when a test passed only on retry, and treat that count as a defect backlog — not as “green, move on.”
• Quarantine, then fix. Move a known-flaky test out of the blocking suite so it stops eroding trust, but with a ticket to fix it — not to forget it.
• Reserve retries for genuinely external instability you don’t control (a shared environment, a third party), and even then, log it.

The line to hold: a retry should buy you visibility while you fix the cause, never permission to ignore it. The moment retries become the fix, your green pipeline is lying to you.

8 Common Mistakes

9 Now You Try

Three graded exercises. Write your answer, run it for AI feedback, then compare to the model answer.

Flakiness sources:
1. Two fixed sleeps (waitForTimeout 3000 and 2000) — guesses about timing that break on slow/fast runs.
2. textContent() + expect(total).toBe(...) — a one-shot read with no auto-retry; it captures whatever is on screen at that instant, before the total may have loaded.
3. Order dependence / shared state — relies on a separate 'add item to cart' test having run first and left an item. Fails alone or in parallel.
4. Non-deterministic data — asserts the order date equals today's real date, which changes every day and across timezones/DST.

Deterministic rewrite:
test('checkout shows total', async ({ page }) => {
  // own your data: seed a cart with a known item via API/fixture, don't depend on another test
  await seedCartWithItem(page, { sku: 'ABC', price: 4200 });
  // freeze the clock so the date is fixed
  await page.clock.setFixedTime(new Date('2026-06-05T10:00:00+12:00'));

  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();   // auto-waits for actionable

  await expect(page.locator('#total')).toHaveText('$42.00');       // web-first, auto-retries
  await expect(page.locator('#order-date')).toHaveText('05/06/2026'); // fixed, not new Date()
});

Key moves: no sleeps; web-first assertions that wait for the condition; the test seeds its own data; the clock is frozen so the date is deterministic. Full marks identify the sleeps, the one-shot read, the order dependence, AND the date — and replace all four.

(a) Almost certainly a timing/race condition. CI runners are slower and more loaded than a dev laptop, so async operations (fetch, render, animation) take longer and the test asserts before the app is ready. "Different assertion each time" fits a general readiness/race problem rather than one specific broken locator. Shared-state-under-parallelism is the second candidate if CI runs tests in parallel and local doesn't.

(b) Because retries hide non-determinism, and non-determinism is sometimes a real app bug (the Kahu Freight $0.00 checkout). Going green on retry tells you nothing about WHY it failed — you've silenced a signal, not fixed a cause. The pipeline is now lying about stability.

(c) Reproduce the load locally: run the test with --repeat-each 50 (and under CPU throttling / parallel workers) to make it fail on demand. Then read the trace from a failed run to see exactly what the test was waiting for and what state the app was in. Replace any sleeps with web-first assertions; wait for the real signal (network response / spinner gone / actionable element). Check whether the app itself has a race (e.g. button enabled before data loads) — that's a product bug to log, not a test fix. Keep retries only as a tracked flakiness metric while you fix it.

Marking: full marks name timing/race as the likely category WITH the CI-load reasoning, explain that retries hide a possibly-real bug, and give a concrete reproduce-then-trace plan rather than "add waits".

Lazy-loading list: don't assume the item is present — scroll it into view (or use a locator that auto-scrolls, like Playwright's getByRole which scrolls before acting) and wait for the specific item locator to be visible. Wait for the condition "this item exists and is visible", not a scroll-then-sleep.

Animated button: wait for it to be actionable, not merely present. Auto-retrying click/assertions wait for the element to be visible AND stable (not moving) AND enabled before acting, which rides out the 600ms slide. Avoid clicking immediately after it appears in the DOM.

Variable 200–4000ms backend: use web-first assertions with a timeout comfortably above the worst case (e.g. wait up to 10s for the result), so a slow 4000ms response still passes. Better: wait for the actual network response or for the loading spinner to disappear, so you key off the real signal rather than any fixed number. Don't sleep 4000ms — that's slow on fast runs and still risky.

What I avoid: every waitForTimeout/sleep; one-shot reads (textContent then assert); assuming DOM-present == ready; depending on scroll position or another test's state.

Prove determinism: run it with --repeat-each 50 (ideally under throttling and in parallel). Green 50/50 = deterministic; one failure = a latent race to fix before it ships.

Marking: full marks give a CONDITION-based wait for each of the three hazards (scroll+visible, actionable/stable, response/spinner with a generous timeout), explicitly reject sleeps, and include the repeat-each proof. Any "add a 5-second sleep" answer misses the point.

10 Self-Check

Click each question to reveal the answer.

11 Interview Prep

Real questions asked in NZ automation interviews. Read the model answers, then practise your own version.