Flaky Test Detection
Set up automated flaky test detection to identify unreliable tests in your CI pipeline and improve test suite reliability.
Flaky test detection helps you identify tests that produce inconsistent results on the same code, allowing you to improve the reliability of your test suite and reduce false positives in your CI pipeline.
Understanding flaky tests
Section titled Understanding flaky testsA flaky test is one whose outcome doesn’t follow from the code alone. The clearest signal is repeated runs of the same commit (SHA1) in the same pipeline and job that disagree. A test that fails on new commits now and then, with passing runs in between, is flaky too. A test that keeps failing, with nothing pointing to flakiness, is reported as broken rather than flaky. See Health Status for how a test moves between healthy, flaky, and broken.
Below is a simplified visualization of repeated runs on one commit: two executions of the same tests on the exact same commit (same SHA1) produce different results.
How flaky detection works
Section titled How flaky detection worksMergify compares each uploaded test result against the results it already holds for the same test, on the same commit and branch, in the same pipeline and job, and records any disagreement. Any repeat execution counts, whether CI reruns a job or a job runs the suite several times in a row. It also watches for a test that keeps failing now and then on new commits, which catches flakiness whose runs sat on different commits.
Detection does not depend on the scheduled runs that the setup guides below describe. Running your suite several times on the same commit produces the evidence deliberately, on a schedule you choose, instead of waiting for a rerun to expose the problem. Keep the job name stable across those runs: a test’s history is tracked per branch, pipeline, and job, so a name that varies from one run to the next splits one test into several and leaves nothing to compare.
Setting up flaky test detection
Section titled Setting up flaky test detectionSee CI-specific setup guides:
GitHub Actions
Configure scheduled looping runs and upload reports.
Jenkins
Configure scheduled looping runs and upload reports.
Buildkite
Configure scheduled looping runs and upload reports.
Where the results appear
Section titled Where the results appearTest-level results live in Test Engine, not in CI Insights: its Detection page lists the flaky and broken tests found across your repositories and reports each one’s impact: the share of its executions that failed, as low, medium, or high.
Flakiness at the job level stays in CI Insights, on the Jobs page.
Was this page helpful?
Thanks for your feedback!