---
title: Fail First
description: When a batch fails, Fail First makes the search for the pull request that broke it faster and cheaper, without ever skipping a test.
---

When a batch fails, Mergify [splits it](/merge-queue/batches/failures) to find
the pull request that broke it. Fail First makes that search faster and
cheaper: every split starts with the tests that already failed, and
stops early when they fail again. On a suite that takes 40 minutes, a split
holding the culprit still stops after the few tests that had failed. No test is
ever skipped to turn a split green.

## What it looks like

A batch of 8 pull requests failed, with 3 of its 96 tests red. Mergify split it
in two rounds to find the culprit, and Fail First ran those 3 tests first in every
split:

> **Figure:** With Fail First, the splits holding PR5 stopped after the 3 tests that had failed, in seconds, while the splits without it ran all 96 tests and passed. Round 2 starts once PR1 to PR4 have merged. Green passed, red failed.
>
> Timing diagram, to scale, of a real run: a batch of 8 pull requests, PR1 to PR8, failed with 3 of its 96 tests red, and Mergify split it to find the pull request that broke it. Round 1 starts four tests at once. PR1 to PR2 runs all 96 tests and passes in 67 seconds, PR1 to PR4 all 96 in 72 seconds. PR1 to PR6 and PR1 to PR7 hold PR5: each runs the 3 tests that had failed first, and stops after them, in 15 and 17 seconds. Once PR1 to PR4 have merged, at 72 seconds, round 2 tests PR1 to PR5: it stops after the same 3 tests, in 14 seconds, so PR5 broke the batch.

## How it works

Fail First reorders the tests in a split:

```dot class="queue" title="Tests 2, 5 and 7 failed in the batch: Fail First runs them first, the others follow in their usual order"
digraph {
  nodesep=0.15;
  node [shape=box, width=0.9];

  // Graphviz stacks the second row on top: the usual order is declared last
  // so that it reads first.
  subgraph cluster_fail_first {
    class="batch";
    label="With Fail First";
    f2 [label="test 2", class="failed"];
    f5 [label="test 5", class="failed"];
    f7 [label="test 7", class="failed"];
    f1 [label="test 1", class="config"];
    f3 [label="test 3", class="config"];
    f4 [label="test 4", class="config"];
    f6 [label="test 6", class="config"];
    f8 [label="test 8", class="config"];
    f2 -> f5 -> f7 -> f1 -> f3 -> f4 -> f6 -> f8 [class="chrome"];
  }

  subgraph cluster_usual {
    class="batch";
    label="Usual order";
    u1 [label="test 1", class="config"];
    u2 [label="test 2", class="failed"];
    u3 [label="test 3", class="config"];
    u4 [label="test 4", class="config"];
    u5 [label="test 5", class="failed"];
    u6 [label="test 6", class="config"];
    u7 [label="test 7", class="failed"];
    u8 [label="test 8", class="config"];
    u1 -> u2 -> u3 -> u4 -> u5 -> u6 -> u7 -> u8 [class="chrome"];
  }
}
```

```dot class="flow" narrow
digraph {
  nodesep=0.3;
  ranksep=0.25;
  node [shape=box, width=0.9];
  edge [class="chrome"];

  // Top to bottom, Graphviz sets the clusters left to right in the order they
  // are declared, so here the usual order comes first.

  subgraph cluster_usual {
    class="batch";
    label="Usual order";
    u1 [label="test 1", class="config"];
    u2 [label="test 2", class="failed"];
    u3 [label="test 3", class="config"];
    u4 [label="test 4", class="config"];
    u5 [label="test 5", class="failed"];
    u6 [label="test 6", class="config"];
    u7 [label="test 7", class="failed"];
    u8 [label="test 8", class="config"];
    u1 -> u2 -> u3 -> u4 -> u5 -> u6 -> u7 -> u8;
  }

  subgraph cluster_fail_first {
    class="batch";
    label="With Fail First";
    f2 [label="test 2", class="failed"];
    f5 [label="test 5", class="failed"];
    f7 [label="test 7", class="failed"];
    f1 [label="test 1", class="config"];
    f3 [label="test 3", class="config"];
    f4 [label="test 4", class="config"];
    f6 [label="test 6", class="config"];
    f8 [label="test 8", class="config"];
    f2 -> f5 -> f7 -> f1 -> f3 -> f4 -> f6 -> f8;
  }
}
```

The tests put first are the ones that failed in the run the split comes from:
the batch in the first round, the failing split above it after that. If any of
them fails again, that split holds the culprit: the run stops, and it gives its
CI runner back without running the rest. If they all pass, the run goes on with
the rest of the suite, so a green split has run every test.

Frameworks that run tests by file put first the files that hold those tests,
so the other tests of those files run early too. In a sharded job, each shard
starts with the failed tests it holds, and a shard that holds none runs in its
usual order.

Pull request CI, the batch's own run and Mergify's retries keep your usual
order.

## What you need

Fail First is on by default. A job gets it when:

- your merge queue uses [batches](/merge-queue/batches);

- the job runs the latest release of the Mergify plugin for your
  [test framework](/test-engine/test-frameworks). Jobs that only upload JUnit
  reports keep their order.

To turn it off on a job, set `MERGIFY_FAIL_FIRST_ENABLE` to `false` there.

## In the dashboard

The Fail First section of **Merge Queue → Configuration**, also shown under
**Test Engine → [Configuration](https://dashboard.mergify.com/test-engine/configuration)**,
says whether Fail First runs on your repository and on which jobs. It names the
jobs not set up for Fail First and offers to install or update the plugin on
them.

The merge queue shows what Fail First did in
[each split](/merge-queue/batches/failures#making-each-split-cheaper) and
[charts the time saved](/merge-queue/monitoring#queue-health).

## In your CI log

The plugin says what Fail First did at the end of the job log, where a split is
called a bisection step. A split that holds the culprit stops early, so its log
lists the tests it did not run:

```text
⏩ Fail First

3 tests failed in the previous bisection step, so they ran first.
3 of them failed again, so Mergify stopped the run early: the merge queue can
split this batch again without waiting.
93 tests did not run.
Failed again:
  - tests/test_taxes.py::test_taxes_adds_vat_to_net_price
  - tests/test_taxes.py::test_taxes_adds_vat_to_small_amounts
  - tests/test_taxes.py::test_taxes_adds_vat_to_order_total
```

## Limits

### Tests that depend on their order

Fail First moves the tests that failed in the run the split comes from, or the
files that hold them, to the front; the others keep their order. A test that
only passes once other tests have prepared something for it, such as a database row, a global setting or a file,
can then fail just because it runs first, even though no pull request broke it.
The split goes red, and the merge queue can remove the wrong pull request. If
your suite has such tests, set `MERGIFY_FAIL_FIRST_ENABLE` to `false` on the
jobs that run them.

### A summary check that waits for every job

Some CI setups add a summary job that depends on all the others, such as an
"all checks passed" job, and make it the only check the branch requires. That
job starts only once every other job has finished, so a split goes red when its
slowest job ends, even if a test job stopped after a few seconds. Fail First
still saves the CI time of the tests that did not run, but the culprit is found
no sooner. To get both, make the checks of your test jobs themselves required
on the branch: on GitHub, add them to the required status checks of the
branch's [ruleset](/merge-queue/github-rulesets), which Mergify reads. The split
then goes red as soon as one of them fails.
