Flaky Test

You might say

Run the same commit five times: it sometimes passes and sometimes times out. Check waiting conditions and shared accounts first; do not hide red with endless retries.

A flaky test sometimes passes and sometimes fails when the code has not changed.The same CI commit can time out occasionally, or parallel tests can race for one account and alternate red and green. Unlike a defect that fails every time, flakiness makes the result untrustworthy. Retries can collect clues or temporarily reduce impact, but do not replace finding timing, shared-state, or environment causes.

Know first

Test Case

Flaky Test

Why five runs of one commit alternate red and green

TEST HISTORYlogin.e2e.tscommit 8fd21a

runcommitresulttime

#59128fd21a✓41s

#59138fd21a✓40s

#59148fd21a✓42s

#59158fd21a✓41s

Repeatedly run the same commit first to rule out changing code.

Flaky does not mean harmless occasional failure

Confirm unchanged code, changing result: Passing and failing on the same version and case points to an uncontrolled test, environment, or concurrency condition. A case that always fails is more like a reproducible defect and needs a different path.

Look for observable sources of variance: Check whether the test waits for a true completion signal, cases share an account or port, data remains, or CI differs from local. Record failure frequency and logs, then fix the cause.

Quick checkChoose the best answer

A login test on the same commit runs green, green, red, green, red, and every red is a timeout. What is the best first step?

Keep run evidence, inspect wait conditions, shared accounts, and CI differences, then make it reproduce stablySet retries to infinite until the pipeline turns greenTreat it directly like a defect that always reproduces, without comparing run conditions

You can say this to an AI Agent

This login test times out intermittently on the same commit. Run it repeatedly and retain each log, inspect wait conditions, shared accounts, and test data; find a reproducible source of variance before merely increasing retries.

Learn next

1Test Data and Fixture→2Monitoring→