View as Markdown

Fail First

When a batch fails, Fail First makes the search for the pull request that broke it faster and cheaper, without ever skipping a test.


When a batch fails, Mergify has to find the pull request that broke it, and it does so by retesting smaller splits of the batch. Fail First makes that search faster and cheaper: every split starts with the tests that already failed, and stops early when they fail again. On a suite that takes 40 minutes, a split holding the culprit still stops after the few tests that had failed. No test is ever skipped to turn a split green.

A batch of 8 pull requests failed, with 3 of its 96 tests red. Mergify split it in two rounds to find the culprit, and Fail First ran those 3 tests first in every split:

Tests round 1 round 2 PR1 to PR2 · 96 tests PR1 to PR2 runs all 96 tests and passes in 67 s. 67 s PR1 to PR4 · 96 tests PR1 to PR4 runs all 96 tests and passes in 72 s. 72 s PR1 to PR6 · 3 tests PR1 to PR6 holds PR5: it stops after the 3 tests that had failed, in 15 s. 15 s PR1 to PR7 · 3 tests PR1 to PR7 holds PR5: it stops after the 3 tests that had failed, in 17 s. 17 s PR1 to PR5 · 3 tests PR5, tested on top of the merged PR1 to PR4, stops after 3 tests in 14 s: PR5 broke the batch. 14 s Tests round 1 round 2 PR1 to PR2 · 96 tests PR1 to PR2 runs all 96 tests and passes in 67 s. 67 s PR1 to PR4 · 96 tests PR1 to PR4 runs all 96 tests and passes in 72 s. 72 s PR1 to PR6 · 3 tests PR1 to PR6 holds PR5: it stops after the 3 tests that had failed, in 15 s. 15 s PR1 to PR7 · 3 tests PR1 to PR7 holds PR5: it stops after the 3 tests that had failed, in 17 s. 17 s PR1 to PR5 · 3 tests PR5, tested on top of the merged PR1 to PR4, stops after 3 tests in 14 s: PR5 broke the batch. 14 s
With Fail First, the splits holding PR5 stopped after the 3 tests that had failed, in seconds, while the splits without it ran all 96 tests and passed. Round 2 starts once PR1 to PR4 have merged. Green passed, red failed.

Fail First reorders the tests in a split:

cluster_fail_first With Fail First cluster_usual Usual order f2 test 2 f5 test 5 f2->f5 f7 test 7 f5->f7 f1 test 1 f7->f1 f3 test 3 f1->f3 f4 test 4 f3->f4 f6 test 6 f4->f6 f8 test 8 f6->f8 u1 test 1 u2 test 2 u1->u2 u3 test 3 u2->u3 u4 test 4 u3->u4 u5 test 5 u4->u5 u6 test 6 u5->u6 u7 test 7 u6->u7 u8 test 8 u7->u8 cluster_usual Usual order cluster_fail_first With Fail First u1 test 1 u2 test 2 u1->u2 u3 test 3 u2->u3 u4 test 4 u3->u4 u5 test 5 u4->u5 u6 test 6 u5->u6 u7 test 7 u6->u7 u8 test 8 u7->u8 f2 test 2 f5 test 5 f2->f5 f7 test 7 f5->f7 f1 test 1 f7->f1 f3 test 3 f1->f3 f4 test 4 f3->f4 f6 test 6 f4->f6 f8 test 8 f6->f8
Tests 2, 5 and 7 failed in the batch: Fail First runs them first, the others follow in their usual order

The tests put first are the ones that failed in the run the split comes from: the batch in the first round, the failing split above it after that. If any of them fails again, that split holds the culprit: the run stops, and it gives its CI runner back without running the rest. If they all pass, the run goes on with the rest of the suite, so a green split has run every test.

Frameworks that run tests by file put first the files that hold those tests, so the other tests of those files run early too. In a sharded job, each shard starts with the failed tests it holds, and a shard that holds none runs in its usual order.

Pull request CI, the batch’s own run and Mergify’s retries keep your usual order.

Fail First is on by default. A job gets it when:

  • your merge queue uses batches;

  • the job runs the latest release of the Mergify plugin for your test framework, set up as for the rest of Test Engine. Jobs that only upload JUnit reports keep their order.

To turn it off on a job, set MERGIFY_FAIL_FIRST_ENABLE to false there.

The Fail First section of Test Engine > Configuration, also shown under Merge Queue > Configuration, says whether it runs on your repository and on which jobs. It offers to install or update the plugin on the jobs that cannot use it yet.

Fail First section marked Active, with the jobs it runs on and two jobs to update Fail First section marked Active, with the jobs it runs on and two jobs to update

The merge queue shows what Fail First did in each split and charts the time saved.

The plugin says what Fail First did at the end of the job log, where a split is called a bisection step. A split that holds the culprit stops early, so its log lists the tests it did not run:

⏩ Fail First
3 tests failed in the previous bisection step, so they ran first.
3 of them failed again, so Mergify stopped the run early: the merge queue can
split this batch again without waiting.
93 tests did not run.
Failed again:
- tests/test_taxes.py::test_taxes_adds_vat_to_net_price
- tests/test_taxes.py::test_taxes_adds_vat_to_small_amounts
- tests/test_taxes.py::test_taxes_adds_vat_to_order_total

Tests that depend on their order

Section titled Tests that depend on their order

Fail First moves the tests that failed in the run the split comes from, or the files that hold them, to the front; the others keep their order. A test that only passes once other tests have prepared something for it, such as a database row, a global setting or a file, can then fail just because it runs first, even though no pull request broke it. The split goes red, and the merge queue can remove the wrong pull request. If your suite has such tests, set MERGIFY_FAIL_FIRST_ENABLE to false on the jobs that run them.

A summary check that waits for every job

Section titled A summary check that waits for every job

Some CI setups add a summary job that depends on all the others, such as an “all checks passed” job, and make it the only check the branch requires. That job starts only once every other job has finished, so a split goes red when its slowest job ends, even if a test job stopped after a few seconds. Fail First still saves the CI time of the tests that did not run, but the culprit is found no sooner. To get both, make the checks of your test jobs themselves required on the branch: on GitHub, add them to the required status checks of the branch’s ruleset, which Mergify reads. The split then goes red as soon as one of them fails.

Was this page helpful?