Detection

Even with prevention in place, tests can degrade over time. Detection surfaces all unhealthy tests (flaky and broken) across your repositories, so you can see the full picture and prioritize what to fix.

Detection reports on tests that ran on your repository’s default branch. Results uploaded from pull request branches are stored, but they do not contribute to the metrics shown here.

A repository whose CI has only ever run on pull requests shows an empty Detection page, even though uploads are working. Merge to the default branch and the tests appear after that run completes.

The same scope applies outside the dashboard: the mergify tests show command and the test search API return default-branch results.

Prevention covers the pull request side of Test Engine: it reports on tests running on pull request branches.

Mergify classifies a test from the results collected for it across CI runs:

• Flaky: The test’s outcome does not follow from the code alone, for example it passed and failed on the same commit.

• Broken: The test failed and nothing points to flakiness.

A test goes back to healthy once it passes cleanly for long enough, so a test you fixed stops being reported without waiting for its old failures to age out. See Health Status for every transition between states.

Detection lists tests under an Unhealthy or a Healthy tab. Flaky and broken tests share the unhealthy one, since both need attention, and the Health column on each row tells them apart.

Mergify can rerun unhealthy tests on the default branch automatically, to confirm a test really is flaky instead of waiting for one of its ordinary runs to contradict the last one. Reruns stay within a budget you control, which caps the extra CI time they add.

This is opt-in per repository and requires a test framework plugin. Once a plugin is installed, set a Detection Budget under Native Flaky Detection on the Configuration page to enable it.

Impact is the share of a test’s executions that failed, reported as low, medium, or high. A test that fails on most of its runs has a higher impact than one that fails once in a while, whatever the absolute number of failures.

Use impact to decide which tests to fix first: high-impact tests give you the most return on investment when fixed.

Filter on high impact to surface the tests causing the most CI disruption. These are the best candidates for immediate attention.

Use filters to focus on specific areas:

• Test name: Search for a specific test or pattern
• Job name: Focus on tests within a particular CI job
• Pipeline name: Narrow to a specific CI pipeline
• Impact: Keep only tests at a given impact level

Tests that have already been quarantined are indicated in the health status. This helps you avoid spending time investigating tests that are already being managed through Mitigation.

Detection requires test metrics collection through repeated CI runs. See the CI setup guides for your platform:

• GitHub Actions setup
• Buildkite setup
• Jenkins setup