Fail First

When a batch fails, Mergify has to find the pull request that broke it, and it does so by retesting smaller splits of the batch. Fail First makes that search faster and cheaper: every split starts with the tests that already failed, and stops early when they fail again. On a suite that takes 40 minutes, a split holding the culprit still stops after the few tests that had failed. No test is ever skipped to turn a split green.

A batch of 8 pull requests failed, with 3 of its 96 tests red. Mergify split it in two rounds to find the culprit, and Fail First ran those 3 tests first in every split:

The splits without the culprit ran the whole suite and went green, as they would without Fail First. The splits holding it cost a few seconds each instead of a full run.

Fail First reorders the tests in a split:

If one of the tests that failed in the batch fails again, the run stops early: that split holds the culprit, and it gives its CI runner back without running the rest. If they all pass, the run goes on with the rest of the suite, so a green split has run every test.

Frameworks that run tests by file put first the files that hold those tests, so the other tests of those files run early too. In a sharded job, each shard starts with the failed tests it holds, and a shard that holds none runs in its usual order.

Pull request CI, the batch’s own run and Mergify’s retries keep your usual order.

Fail First is on by default. A job gets it when:

• your merge queue uses batches;

• the job runs the Mergify plugin for your test framework, version 2026.10.8 or later, set up with your MERGIFY_TOKEN as for the rest of Test Engine. There is nothing else to add to your jobs. Jobs that only upload JUnit reports keep their usual order.

To turn it off on a job, set MERGIFY_FAIL_FIRST_ENABLE to false there.

The Fail First section of Test Engine > Configuration says whether it runs on your repository and on which jobs. Over the last 30 days, it counts the splits that got it and the ones that stopped early, and estimates the CI time it saved. It also offers to install or update the plugin on the jobs that cannot use it yet.

The plugin says what Fail First did at the end of the job log, where a split is called a bisection step. A split that holds the culprit stops early, so its log lists the tests it did not run:

⏩ Fail First

3 tests failed in the previous bisection step, so they ran first.
3 of them failed again, so Mergify stopped the run early: the merge queue can
split this batch again without waiting.
93 tests did not run.
Failed again:
  - tests/test_taxes.py::test_taxes_adds_vat_to_net_price
  - tests/test_taxes.py::test_taxes_adds_vat_to_small_amounts
  - tests/test_taxes.py::test_taxes_adds_vat_to_order_total

Fail First moves the tests that failed in the batch, or the files that hold them, to the front of the suite; the others keep their order. A test that only passes once other tests have prepared something for it, such as a database row, a global setting or a file, can then fail just because it runs first, even though no pull request broke it. The split goes red, and the merge queue can remove the wrong pull request. If your suite has such tests, set MERGIFY_FAIL_FIRST_ENABLE to false on the jobs that run them.

Some CI setups add a summary job that depends on all the others, such as an “all checks passed” job, and make it the only check the merge conditions require. That job starts only once every other job has finished, so a split goes red when its slowest job ends, even if a test job stopped after a few seconds. Fail First still saves the CI time of the tests that did not run, but the culprit is found no sooner. To get both, require the checks of your test jobs themselves in your merge conditions: the split then goes red as soon as one of them fails.