---
title: When a Batch Fails
description: How Mergify finds the pull request that broke a batch, and how to bound that search or let a later pass skip a flaky failure.
---

When a batch fails, or its [checks time out](/merge-queue/lifecycle#checks-timeout), Mergify
splits it to find the pull request that broke it, removes that one, and merges the others. This
needs no setup.

## How a failed batch is split

Mergify cuts the batch into parts and tests them at once, each on top of the parts before it, as
many as your [parallel checks](/merge-queue/performance#parallel-checks) allow. The parts ahead of
the first one that fails merge, the ones after it go back to the queue, and the failing part is
split again. The last pull request of the batch is never retested: it already failed with the
others, so if every part passes, it is the culprit.

> **Figure:** A batch of 6 fails with 3 parallel checks, and PR4 broke it: two rounds of splitting find it, every other pull request merges, and PR4 leaves without ever being tested on its own. Green passed, red failed, dashed was cancelled. Not to scale.
>
> Timing diagram, not to scale, of a batch of 6 pull requests, PR1 to PR6, with max_parallel_checks set to 3, where PR4 is the pull request that broke it. The batch fails. Round 1 starts three tests at once: PR1 to PR2, PR1 to PR4 and PR1 to PR5. PR6 is in none of them. PR1 to PR2 passes, and PR1 and PR2 merge. PR1 to PR4 fails, so PR1 to PR5, tested on top of it, is cancelled. Round 2 splits the two pull requests that test added, PR3 and PR4: PR4 is not retested, and PR3 is tested on top of the merged PR1 and PR2. PR3 passes and merges, so PR4 is the culprit and leaves the queue. PR5 and PR6 go back to the queue, pass in a new batch and merge.

The batches tested on top of the failed one are cancelled, and their pull requests are tested
again. While the split runs, the batches report a [batch
status](/configuration/data-types#batch-status) such as `bisecting`,
`waiting_for_previous_batches` or `waiting_for_requeue` in [the dashboard and the
CLI](/merge-queue/monitoring). The event that starts a split counts the last pull request as a
sub-batch of its own, so its `number_of_sub_batches` is one more than the number of parts.

### How many checks a split gets

A split runs on `max_parallel_checks` checks, or on the [per-scope
capacity](/merge-queue/queue-modes#limiting-concurrency-per-scope) of its scope when that is
smaller. When the base branch requires branches to be up to date before merging and the queue does
not use [`fast-forward`](/merge-queue/merge-strategies#fast-forward), Mergify checks one batch at
a time, so a split gets one check. With `fast-forward`, the parts are tested one after the other.

In [serial mode](/merge-queue/queue-modes#serial-mode), and for a
[barrier](/configuration/scopes#barrier-behavior-in-the-queue) in parallel mode, the whole queue
waits on the split, so it takes every check, as does any split under
[`batch_max_failure_resolution_attempts`](#limiting-the-search). Otherwise, in
[parallel](/merge-queue/queue-modes#parallel-mode) and
[isolated](/merge-queue/queue-modes#isolated-mode) modes, a split takes the share of the checks
that matches the share of waiting pull requests it holds up, and leaves the rest to other scopes:

> **Figure:** In a parallel queue with 6 checks, a failed frontend batch of 13 takes the share of the checks matching the share of waiting pull requests it holds up, and leaves the rest to backend. With nothing else waiting, it stops at 3 parts: a wider cut would not find the culprit sooner. Ringed checks are the split's tests.
>
> Three columns, one per number of pull requests waiting in backend, each showing the 6 checks of a parallel queue with max_parallel_checks set to 6 while a failed frontend batch of 13 pull requests, PR1 to PR13, runs its first round of splitting. With 26 pull requests waiting in backend, the split runs 2 tests, PR1 to PR6 and PR1 to PR12, and the other 4 checks run backend batches. With 13 waiting in backend, it runs 3 tests, PR1 to PR4, PR1 to PR8 and PR1 to PR12, and the other 3 checks run backend batches. With nothing waiting in any other scope, it still runs 3 tests, PR1 to PR4, PR1 to PR8 and PR1 to PR12, and 3 checks stay free, because 4, 5 or 6 parts would not find the culprit in fewer rounds on average.

## Picking a batch size that splits cleanly

Each round of a split runs up to N tests at once, N being the checks it gets. Some batch sizes keep
all N busy in every round, whichever part fails:

| Parallel checks (N) | Sizes that keep every check busy |
| ------------------- | -------------------------------- |
| 2                   | 1, 3, 7, 15, 31                  |
| 3                   | 1, 4, 13, 40                     |
| 4                   | 1, 5, 21, 85                     |
| 5                   | 1, 6, 31                         |

> **Figure:** With 3 parallel checks, the worst case gains a round past 4, 13 and 40 pull requests, so a batch of 7 costs as many rounds as a batch of 13.
>
> Step chart of the most rounds of splitting it takes to find the pull request that broke a batch, against the number of pull requests in the failed batch, from 1 to 40, with 3 parallel checks. A batch of 1 needs no split. Batches of 2 to 4 need 1 round, 5 to 13 need 2 rounds, and 14 to 40 need 3 rounds. The points at 4, 13 and 40 mark the largest batch for each number of rounds.

So set `batch_size`, or the `max` of a [dynamic batch
size](/merge-queue/batches#dynamic-batch-sizing), to one of these sizes to bound the worst case:
with 3 parallel checks, `max: 13` never needs more than 2 rounds. A split that shares its checks
with other scopes can take more rounds.

## Limiting the search

`batch_max_failure_resolution_attempts` caps how many rounds of splitting a failure goes through.
When a part holding several pull requests fails in the last round, it is not split again: all its
pull requests leave the queue with the
[`batch-max-failure-resolution-attempts`](/configuration/data-types#queue-dequeue-reason) dequeue
reason. With `0`, a failed batch is never split, and all its pull requests leave the queue. The
cap counts the rounds behind each part: pull requests that go back to the queue start over in a
new batch, with a fresh count.

In a queue whose split gets one check, the last round tests only the first part. If it passes, it
merges, and every pull request after it leaves the queue untested with the
[`dropped-by-bisection-elimination`](/configuration/data-types#queue-dequeue-reason) dequeue
reason: the failure lies among them, and no round is left to tell them apart.

## Making each split cheaper

Your CI runs again on every part. With the Mergify plugin for your test framework, [Fail
First](/test-engine/fail-first) starts each part with the tests that failed in the batch it came
from, and stops the run as soon as one of them fails again, so a part that holds the culprit costs
a few tests instead of your whole suite. A part that passes has still run every test.

## Skipping a flaky failure

With `skip_intermediate_results`, a batch that passes stands in for the batches it was tested on
top of: they merge without waiting for their own checks, even one whose checks failed and that
Mergify is [retrying](/merge-queue/lifecycle#automatic-ci-retries). Leave it off where every batch
must pass its own checks.

```yaml
merge_queue:
  skip_intermediate_results: true
```

With [scopes](/configuration/scopes), a pass counts only in the scopes its batch was asked to test,
and an earlier batch merges once the passes on top of it cover every scope it touches. A
[barrier](/configuration/scopes#declaring-a-pull-request-impacts-every-scope) touches every scope.

> **Figure:** With skip_intermediate_results, batch 1 merges once the passes on top of it cover every scope it touches: batch 2's pass covers frontend only, and batch 3's adds api. Green passed, dashed was not waited for. Not to scale.
>
> Timing diagram, not to scale, of skip_intermediate_results with three batches, each tested on top of the ones before it: batch 1 touches frontend and api, batch 2 frontend, batch 3 api. Batch 2 passes first, but covers only frontend, so batch 1 keeps waiting on its own test, and batch 2 waits behind it. Batch 3 then passes on api. Between them, the two passes cover both scopes of batch 1, so batch 1 merges without waiting for its own test, and batches 2 and 3 merge right after it.

Inside a split, a part that passes stands in for the parts before it the same way. A batch whose
failure Mergify acts on, by splitting it or removing its pull request, cancels every batch behind
it, so no later pass stands in for it. On a
[`fast-forward`](/merge-queue/merge-strategies#fast-forward) or
[`merge-batch`](/merge-queue/merge-strategies#merge-batch) queue, a pass never stands in for a
batch or part whose checks failed. [Isolated mode](/merge-queue/queue-modes#isolated-mode) does
not support the option.

A check that diffs against [`checking_base_sha`](/merge-queue/batches/metadata#fields), such as
a linter on the changed files, reads only the passing batch's own pull requests, yet the batches
ahead still merge on its pass. To have such a check cover them, diff against the base branch
instead.

## Options

`batch_max_failure_resolution_attempts` and `skip_intermediate_results` are listed with the other
[batch options](/merge-queue/batches#options).

## See also

- [Batches](/merge-queue/batches)
- [Fail First](/test-engine/fail-first)
- [Parallel checks](/merge-queue/performance#parallel-checks)
