Contents

What a flaky test is

A flaky test is one that, on unchanged code, sometimes passes and sometimes fails. Run it — green. Run it again — red. Nothing changed.

This is sneakier than a normal failure. A failing deterministic test honestly says "there's a bug here." A flaky test lies both ways — sometimes a false alarm, sometimes a false sense of safety.

Why it's expensive

The direct cost is the time spent triaging false failures. But the real price is higher: flaky tests destroy trust in tests as a signal.

The decay path is familiar to many teams:

  1. Tests sometimes fail "on their own."
  2. Developers get used to re-running the suite without looking.
  3. Re-running becomes a habit: red means "probably just flaky."
  4. One day red is a real bug, but it gets re-run and shipped to production.

The moment a team starts ignoring a red run, tests stop doing their main job.

The five main causes

In practice, the vast majority of flaky tests trace back to five sources.

1. Timing and asynchrony

The most common cause by far. The test waits for a fixed duration instead of an actual condition.

// BAD: hoping 500ms is enough
await sleep(500);
expect(element.textContent).toBe("Loaded");

// GOOD: wait for the condition itself
await waitFor(() => expect(element.textContent).toBe("Loaded"));

sleep is a bet on machine speed. It passes on a fast laptop and fails on a busy CI agent. The fix: wait for a specific state, not for time.

2. Shared mutable state

Tests share something live: a database, a file, a global variable, a cache. One test leaves debris, the next one fails.

// A global counter the tests never reset
let cache = {};

test("A", () => { cache.user = 1; /* ... */ });
test("B", () => { expect(cache).toEqual({}); }); // fails after A

The fix: isolation. Each test sets up and tears down its own state (beforeEach / afterEach) instead of relying on cleanliness from its neighbors.

3. Execution-order dependency

A specific but dangerous variant of the above. Tests pass in one order and fail in another because one silently prepares data for the next.

This hurts especially when the runner randomizes order or runs tests in parallel. A good sign of a healthy suite is that it passes in random order. Many frameworks can shuffle it (--shuffle); turn it on and fix whatever surfaces.

4. External dependencies

The test hits a real third-party service, the network, or the system clock. Any of those can blink.

// BAD: the test depends on the real date
test("discount is active today", () => {
  expect(isDiscountActive()).toBe(true); // breaks after the promo deadline
});

// GOOD: time under control
test("discount is active on the promo day", () => {
  jest.useFakeTimers().setSystemTime(new Date("2026-06-23"));
  expect(isDiscountActive()).toBe(true);
});

The fix: at the unit and integration level, mock or stub external dependencies. Leave real external calls to a small number of deliberate e2e tests, accepting that they'll be less stable.

5. Non-determinism in the data itself

The test relies on an order nobody guaranteed. The classic case is asserting on a collection that's actually unordered.

// BAD: row order from the DB isn't guaranteed without ORDER BY
expect(users).toEqual([alice, bob]);

// GOOD: compare ignoring order
expect(users).toEqual(expect.arrayContaining([alice, bob]));
expect(users).toHaveLength(2);

Same category: relying on hash key order, on random ids, or on locale when sorting strings.

Catching flakiness systematically

Fixing them one by one isn't enough — you need a process that surfaces them.

Run a suspect test in a loop. Running it 50–100 times in a row quickly separates a stable test from a flaky one:

for i in $(seq 1 100); do npm test -- suspect.test.js || break; done

Randomize order in CI continuously, not just while debugging. This catches inter-test dependencies preventively.

Log every retry. If CI re-runs failed tests, those re-runs should be visible in the report, not hidden silently. Otherwise flakiness accumulates unnoticed.

Set up a quarantine. A chronically flaky test gets pulled out of the blocking run into a separate group — but stays in the backlog as a required fix. This beats both extremes: it doesn't block the team with false failures, and it doesn't lose coverage through silent deletion.

What NOT to do

Don't mask with retries. Auto-retrying up to three times turns red into green and hides the problem. As an emergency valve on a critical pipeline it's acceptable, but only paired with logging and follow-up. As a permanent solution it's a way to accumulate technical debt.

Don't add a bigger sleep. Increasing the delay doesn't cure a race condition; it just lowers the failure rate — until a machine turns out to be even slower.

Don't delete the test silently. Losing coverage without a trace is more dangerous than the flakiness itself: you stop verifying real functionality and don't even know it.

Wrapping up

A flaky test isn't randomness — it's an undiagnosed cause: timing, shared state, ordering, an external call, or data non-determinism. Fix the source, not the symptom. The real stake is trust: a test suite is worth exactly as much as the team's belief in a red run. One ignored flake costs less than the habit of ignoring red.

Share: