Health Status
How Mergify decides whether a test is healthy, flaky, or broken, and what moves it from one state to another.
Every test is in one of three states: healthy, flaky, or broken. Mergify tracks a test per repository, branch, pipeline, job, and test name, so the same test can be healthy on one branch and flaky on another.
Mergify reads a test’s runs one at a time, in the order they finished, and ignores skipped runs. Each run can move the test at most one step, and a run that matches none of the arrows below leaves the test where it is. A test that stops running keeps its state for 90 days, then reads as healthy again, with no run left to back the verdict.
Flaky evidence
Section titled Flaky evidenceA failure alone does not make a test flaky. Mergify needs evidence that the outcome does not follow from the code, and any of these counts:
-
Same commit, different result: the test both passed and failed on the same commit.
-
Marked flaky: your test framework reported the run as flaky, even if it passed.
-
Intermittent failure: the test fails on a new commit after passing since its last failure, and that last failure is recent. This holds even if the test recovered in between, so a test that keeps failing now and then ends up flaky, not broken.
A clean pass is a passing run with no flaky evidence. Any failure, or any run with flaky evidence, restarts the streak of clean passes.
A flaky test takes longer to recover than a broken one. A broken test that starts passing has most likely been fixed, while a flaky test can pass for a while before failing again, so Mergify waits for a longer streak spread over time before calling it healthy.
Example
Section titled ExampleRelated pages
Section titled Related pages-
Detection lists your tests by state and helps you decide which ones to fix first.
-
Quarantine explains when Mergify quarantines a test based on its state.
Was this page helpful?
Thanks for your feedback!