View as Markdown

Health Status

How Mergify decides whether a test is healthy, flaky, or broken, and what moves it from one state to another.

Every test is in one of three states: healthy, flaky, or broken. Mergify tracks a test per repository, branch, pipeline, job, and test name, so the same test can be healthy on one branch and flaky on another.

Mergify reads a test’s runs one at a time, in the order they finished, and ignores skipped runs. Each run can move the test at most one step, and a run that matches none of the arrows below leaves the test where it is. Only runs move a test: a test that stops running keeps its state, and time alone never makes it healthy.

failure failure few_clean a few clean passes broken Broken failure->broken evidence_from_healthy flaky evidence few_clean->broken long_clean a longer streak of clean passes over several days flaky Flaky evidence_from_healthy->flaky long_clean->flaky evidence_from_broken flaky evidence evidence_from_broken->flaky fail_streak fails on several commits in a row healthy Healthy healthy->failure healthy->few_clean healthy->evidence_from_healthy healthy->long_clean broken->evidence_from_broken broken->fail_streak flaky->fail_streak
Test health states: a failure makes a test unhealthy, a streak of clean passes brings it back

Flaky evidence

Section titled Flaky evidence

A failure alone does not make a test flaky. Mergify needs evidence that the outcome does not follow from the code, and any of these counts:

  • Same commit, different result: the test both passed and failed on the same commit.

  • Marked flaky: your test framework reported the run as flaky, even if it passed.

  • Intermittent failure: the test fails on a new commit after passing since its last failure, and that last failure is recent. This holds even if the test recovered in between, so a test that keeps failing now and then ends up flaky, not broken.

A clean pass is a passing run with no flaky evidence. Any failure, or any run with flaky evidence, restarts the streak of clean passes.

A flaky test takes longer to recover than a broken one. A broken test that starts passing has most likely been fixed, while a flaky test can pass for a while before failing again, so Mergify waits for a longer streak spread over time before calling it healthy.

Example

Section titled Example

Here is one test, run by run:

  1. It fails on commit a1. Nothing points to flakiness, so it is broken.

  2. It passes cleanly on the next few commits and goes back to healthy.

  3. Two days later, it fails on commit e5. It passed since its recent failure on a1, so this failure is intermittent: the test is now flaky.

  4. A rerun of e5 fails again. Reruns of the same commit count once toward the failing streak that would make it broken, so the test stays flaky.

  5. Another rerun of e5 passes. Same commit, different result: still flaky, and the clean streak has not started.

  6. It then passes cleanly on new commits, many times over several days, and goes back to healthy.

Section titled Related pages

  • Detection lists your tests by state and helps you decide which ones to fix first.

  • Quarantine explains when Mergify quarantines a test based on its state.

Was this page helpful?