Detection

Even with prevention in place, tests can degrade over time. Detection surfaces all unhealthy tests (flaky and broken) across your repositories, so you can see the full picture and prioritize what to fix.

Detection reports on tests that ran on your repository’s default branch. Results uploaded from pull request branches are stored, but they do not contribute to the metrics shown here.

A repository whose CI has only ever run on pull requests shows an empty Detection page, even though uploads are working. Merge to the default branch and the tests appear after that run completes.

The same scope applies outside the dashboard: the mergify tests show command and the test search API return default-branch results.

Prevention covers the pull request side of Test Engine: it reports on tests running on pull request branches.

Mergify classifies a test from the results collected for it across CI runs:

• Flaky: The test’s outcome does not follow from the code alone, for example it passed and failed on the same commit.

• Broken: The test failed and nothing points to flakiness.

A test goes back to healthy once it passes cleanly for long enough, so a test you fixed stops being reported without waiting for its old failures to age out. See Health Status for every transition between states.

Detection lists tests under an Unhealthy or a Healthy tab. Flaky and broken tests share the unhealthy one, since both need attention, and the Health column on each row tells them apart.

Each test also has a State, shown next to its health:

• Active: the test’s failures count in your CI outcome.

• Quarantined: the test still runs, but its failures no longer block merges. See Quarantine for how it works.

Health and state are independent: a broken test can still be active, and a quarantined test can be healthy again. Select tests in the list to quarantine or unquarantine them.

When your subscription includes quarantine, a Test States card above the list shows how many tests are quarantined, how many failures quarantine kept from failing CI, how many pull requests it unblocked, and how quarantined tests have been doing over time.

Next to the tabs, Detection offers a Leftover Quarantines cleanup when it finds any: quarantines whose test has no recent runs on the default branch, often because the test was renamed or deleted. Detection cannot list a test that does not run, so these open in a dialog where you can remove the quarantines you no longer need. If such a test still runs on another branch, or runs again later, its failures count in your CI outcome once no quarantine covers it.

To find the quarantined tests that are healthy again, open the Healthy tab and filter on the Quarantined state.

Mergify can rerun unhealthy tests on the default branch automatically, to confirm a test really is flaky instead of waiting for one of its ordinary runs to contradict the last one. Reruns stay within a budget you control, which caps the extra CI time they add.

This is opt-in per repository and requires a test framework plugin. Once a plugin is installed, set a Detection Budget under Native Flaky Detection on the Configuration page to enable it.

The CI Impact column is the share of a test’s executions that failed, reported as low, medium, or high. A test that fails on most of its runs has a higher impact than one that fails once in a while, whatever the absolute number of failures.

Use impact to decide which tests to fix first: high-impact tests give you the most return on investment when fixed.

Filter on high impact to surface the tests causing the most CI disruption. These are the best candidates for immediate attention.

Use filters to focus on specific areas:

• Test: Search for a specific test or pattern
• Job: Focus on tests within a particular CI job
• Pipeline: Narrow to a specific CI pipeline
• CI Impact: Keep only tests at a given impact level
• State: Keep only active or only quarantined tests

Filter on the Quarantined state to see the tests your team already quarantined, or on Active to leave them out and focus on the tests that still block merges.

Detection requires test metrics collection through repeated CI runs. See the CI setup guides for your platform:

• GitHub Actions setup
• Buildkite setup
• Jenkins setup