Test Retry

With test retry, the Mergify plugin runs a failing test again, in the same job, right away. If a retry passes, the test passes and the build stays green, where a flaky failure on your default branch would otherwise turn it red until someone reruns the job. Every attempt is still recorded, so the test’s health status still shows it as flaky.

Test retry runs one test again inside the job. To rerun a whole CI job that failed, see Auto-Retry.

Test retry runs on pushes to your default branch only. Pull requests and merge queue runs never retry: there, a failure that goes away on a second try could be a test the pull request made flaky, and retrying it would let that test merge.

When a test fails, the plugin runs it again, until it passes or one of two limits stops it: the number of retries one test may get, and the job’s retry budget, the extra time the job may spend on retries.

The two limits work together:

• Retries per Test caps how many times one test runs again. A test that keeps failing stops there, so it cannot use up the budget on its own.

• Retry Budget is a share of how long the job’s tests usually take. Every retry in the job draws on it, in the order the failures happen. Once it is spent, the next failure stands. The plugin works out how long the tests usually take from the average duration of a test on your default branch times the number of tests it runs, and allows at least 4 seconds whatever that comes to.

For example, a job that runs 600 tests, each taking half a second on average, usually spends 300 seconds on them. At the default 5%, it may spend 15 more seconds on retries.

Some tests are never retried:

• tests Mergify classifies as broken on your default branch: they fail every time, so a retry would only cost time;

• quarantined tests: their failures already do not fail the job.

You can also keep a job or a test from retrying: see Turning it on.

The dashboard lets you set the budget from 1% to 50%, and the retries per test from 1 to 20.

Test retry is set per repository, in the Test Retry section of Test Engine > Configuration. Pick the repository at the top right, then turn on Retry Failing Tests: it starts from the defaults above. To tune it, change the budget or the retries per test and click Save Changes. Changing it requires write access to the repository.

Turning test retry off clears both settings: turning it on again starts from the defaults. View change history opens the activity log for every change to test retry.

Test retry is available on plans that include quarantine.

A job retries tests when it runs the latest release of the Mergify plugin for its test framework, set up with your MERGIFY_TOKEN as for the rest of Test Engine: pytest, RSpec, Minitest, Vitest, Playwright, Jest or Mocha. Jobs that only upload JUnit reports never retry. The Test Retry section lists the jobs whose plugin is too old or missing, and offers to install or update it.

To keep one job from retrying, set MERGIFY_TEST_RETRY_ENABLE to false in its environment. With pytest, you can also keep a single test from retrying with @pytest.mark.mergify(auto_retry=False).

The plugin says what test retry did at the end of the job log:

🔁 Test retry
Test retry: 2 tests failed, so Mergify retried them.
1 of them passed on a retry, so it does not fail this run.
Passed on a retry:
  - tests/test_checkout.py::test_coupon (attempt 3)
1 of them failed every attempt, so this run fails on it.
Still failing:
  - tests/test_checkout.py::test_po_box (6 attempts)
Test retry used 4.8 s of its 15.0 s budget.

Each framework retries a test its own way, and some cannot retry every test:

• pytest: no retry under pytest-xdist’s --dist each.

• Vitest: Vitest 4.1 or later, in Node workers. Each worker sizes the budget from the tests of the file it runs, and the workers spend it together.

• Playwright: the configuration needs withMergify. No retry for a test of a serial group, nor under failOnFlakyTests.

• Jest: the configuration needs withMergify and jest-circus. No retry for a test.concurrent test, nor for one Jest retries at the end of its block or with its whole describe block. The budget is sized from the tests of the file that failed, so it is often the 4-second minimum.

• Mocha: in a parallel run, each worker sizes the budget from the tests of the file it runs, and the workers spend it together.

A framework’s own retries, such as pytest-rerunfailures, rspec-retry or Playwright’s retries, run first. Test retry only runs a test again once they have all failed.