View as Markdown

Health Status

How Mergify decides whether a test is healthy, flaky, or broken, and what moves it from one state to another.


Every test is in one of three states: healthy, flaky, or broken. Mergify tracks a test per repository, branch, pipeline, job, and test name, so the same test can be healthy on one branch and flaky on another.

Mergify reads a test’s runs one at a time, in the order they finished, and ignores skipped runs. Each run can move the test at most one step, and a run that matches none of the arrows below leaves the test where it is. A test that stops running keeps its state for 90 days, then reads as healthy again, with no run left to back the verdict.

failure failure few_clean a few clean passes broken Broken failure->broken evidence_from_healthy flaky evidence few_clean->broken long_clean a longer streak of clean passes over several days flaky Flaky evidence_from_healthy->flaky long_clean->flaky evidence_from_broken flaky evidence evidence_from_broken->flaky fail_streak fails on several commits in a row healthy Healthy healthy->failure healthy->few_clean healthy->evidence_from_healthy healthy->long_clean broken->evidence_from_broken broken->fail_streak flaky->fail_streak
Test health states: a failure makes a test unhealthy, a streak of clean passes brings it back

A failure alone does not make a test flaky. Mergify needs evidence that the outcome does not follow from the code, and any of these counts:

  • Same commit, different result: the test both passed and failed on the same commit.

  • Marked flaky: your test framework reported the run as flaky, even if it passed.

  • Intermittent failure: the test fails on a new commit after passing since its last failure, and that last failure is recent. This holds even if the test recovered in between, so a test that keeps failing now and then ends up flaky, not broken.

A clean pass is a passing run with no flaky evidence. Any failure, or any run with flaky evidence, restarts the streak of clean passes.

A flaky test takes longer to recover than a broken one. A broken test that starts passing has most likely been fixed, while a flaky test can pass for a while before failing again, so Mergify waits for a longer streak spread over time before calling it healthy.

2 days later several days later Runs Health `a1` a1 `e5` e5 `e5` e5 `e5` e5 broken broken healthy healthy flaky flaky healthy healthy 2 days later several days later Runs Health `a1` a1 `e5` e5 `e5` e5 `e5` e5 broken broken healthy healthy flaky flaky healthy healthy
A few clean passes end a broken stretch; a flaky one ends only after a longer streak spread over several days. Reruns of e5 keep it flaky: a pass and a failure on one commit is flaky evidence, not a recovery. In the runs, green passed and red failed.
  • Detection lists your tests by state and helps you decide which ones to fix first.

  • Quarantine explains when Mergify quarantines a test based on its state.

Was this page helpful?