Prevention

Prevention monitors tests introduced or modified in pull requests. By rerunning tests on PRs, it detects flaky behavior before code merges, keeping your codebase reliable.

When a pull request runs tests, Mergify reruns them to check for consistency. Tests that produce different results on the same commit are flagged as flaky. This happens transparently as part of your existing CI pipeline, with no changes to your test code needed.

Tests caught as flaky on a PR are prevented from silently degrading your test suite. You can review their health status before deciding to merge.

Prevention provides key metrics to help you understand test reliability on pull requests:

The number of flaky tests detected during PR reruns. This is the core value of Prevention: every caught test is a reliability problem that didn’t make it into your codebase.

How many new tests reached your codebase, which tracks how the test suite grows. It counts tests, not their reliability: the health status and confidence Mergify derived from the reruns are on each pull request’s rows in the list below the metrics.

The total CI time spent on reruns. This metric helps teams understand the cost of flaky test prevention and make informed trade-offs between thoroughness and CI budget.

When a PR introduces or modifies tests, check the Prevention page to see their health status. Tests with a flaky or broken status should be investigated before merging.

Prevention opens on All PRs. Switch tabs to filter the list by pull request state:

• Open PRs: tests on pull requests still in review
• Merged PRs: tests on pull requests that have already been merged
• Closed PRs: tests on pull requests that were closed without merging

The confidence column reports how many runs support a test’s health status, counting the pull request’s own runs and the reruns Mergify added:

• High: enough runs to trust the health status.

• Medium: a limited number of runs, so the health status may still change.

• Low: very few runs, so treat the health status as preliminary.

A test that was only just added has barely run, so it starts at low confidence. Consider waiting for more runs before drawing conclusions about its reliability.

Prevention requires a test framework plugin that instruments test runs, and it is opt-in per repository. Once a plugin is installed, set a Prevention Budget under Native Flaky Detection on the Configuration page to enable it. Until then, Mergify collects test results but does not rerun tests.

See the test framework configuration for setup instructions specific to your framework (pytest-mergify, rspec-mergify, etc.).