View as Markdown

Flaky Test Detection

Set up automated flaky test detection to identify unreliable tests in your CI pipeline and improve test suite reliability.


Flaky test detection helps you identify tests that produce inconsistent results on the same code, allowing you to improve the reliability of your test suite and reduce false positives in your CI pipeline.

A flaky test is one whose outcome doesn’t follow from the code alone. The clearest signal is repeated runs of the same commit (SHA1) in the same pipeline and job that disagree. A test that fails on new commits now and then, with passing runs in between, is flaky too. A test that keeps failing, with nothing pointing to flakiness, is reported as broken rather than flaky. See Health Status for how a test moves between healthy, flaky, and broken.

Below is a simplified visualization of repeated runs on one commit: two executions of the same tests on the exact same commit (same SHA1) produce different results.

cluster_commit1 Commit abc123 cluster_commit2 Commit def456 cluster_commit3 Commit ghi789 commit1 test_something run1 Run #1 — tests pass commit1->run1  run 1 run2 Run #2 — tests fail commit1->run2  run 2 detector Flagged as flaky run1->detector run2->detector commit2 test_something run3 Run #1 — tests pass commit2->run3  run 1 run4 Run #2 — tests pass commit2->run4  run 2 stable Consistent not flaky run3->stable run4->stable commit3 test_something run5 Run #1 — tests fail commit3->run5  run 1 run6 Run #2 — tests fail commit3->run6  run 2 stable2 Consistent not flaky run5->stable2 run6->stable2 cluster_commit1 Commit abc123 cluster_commit2 Commit def456 cluster_commit3 Commit ghi789 commit1 test_something run1 Run #1 tests pass commit1->run1 run2 Run #2 tests fail commit1->run2 detector Flagged as flaky run1->detector run2->detector commit2 test_something run3 Run #1 tests pass commit2->run3 run4 Run #2 tests pass commit2->run4 stable Consistent not flaky run3->stable run4->stable commit3 test_something run5 Run #1 tests fail commit3->run5 run6 Run #2 tests fail commit3->run6 stable2 Consistent not flaky run5->stable2 run6->stable2
Two runs of one test on three commits: the runs that disagree flag the test as flaky

Mergify compares each uploaded test result against the results it already holds for the same test, on the same commit and branch, in the same pipeline and job, and records any disagreement. Any repeat execution counts, whether CI reruns a job or a job runs the suite several times in a row. It also watches for a test that keeps failing now and then on new commits, which catches flakiness whose runs sat on different commits.

Detection does not depend on the scheduled runs that the setup guides below describe. Running your suite several times on the same commit produces the evidence deliberately, on a schedule you choose, instead of waiting for a rerun to expose the problem. Keep the job name stable across those runs: a test’s history is tracked per branch, pipeline, and job, so a name that varies from one run to the next splits one test into several and leaves nothing to compare.

Setting up flaky test detection

Section titled Setting up flaky test detection

See CI-specific setup guides:

Test-level results live in Test Engine, not in CI Insights: its Detection page lists the flaky and broken tests found across your repositories and reports each one’s impact: the share of its executions that failed, as low, medium, or high.

Flakiness at the job level stays in CI Insights, on the Jobs page.

Was this page helpful?