View as Markdown

When a Batch Fails

How Mergify finds the pull request that broke a batch, and how to bound that search or let a later pass skip a flaky failure.


When a batch fails, or its checks time out, Mergify splits it to find the pull request that broke it, removes that one, and merges the others. This needs no setup.

Mergify cuts the batch into parts and tests them at once, each on top of the parts before it, as many as your parallel checks allow. The parts ahead of the first one that fails merge, the ones after it go back to the queue, and the failing part is split again. The last pull request of the batch is never retested: it already failed with the others, so if every part passes, it is the culprit.

Tests round 1 round 2 new batch PR1 to PR6 The batch of 6 fails. fails PR1 to PR2 PR1 to PR2 passes: PR1 and PR2 merge. passes PR1 to PR4 PR1 to PR4 fails: the culprit is PR3 or PR4. fails PR1 to PR5 PR1 to PR5 was tested on top of a failure: cancelled. cancelled PR3 PR3 passes on top of PR1 and PR2: PR4 broke the batch. passes PR5 to PR6 PR5 and PR6, back in the queue, pass in a new batch. passes Pull requests PR1 merged PR2 merged PR3 merged PR4 PR4 leaves the queue without being tested on its own. PR4 leaves the queue without being tested on its own. removed PR5 merged PR6 merged Tests round 1 round 2 new batch PR1 to PR6 The batch of 6 fails. fails PR1 to PR2 PR1 to PR2 passes: PR1 and PR2 merge. passes PR1 to PR4 PR1 to PR4 fails: the culprit is PR3 or PR4. fails PR1 to PR5 PR1 to PR5 was tested on top of a failure: cancelled. cancelled PR3 PR3 passes on top of PR1 and PR2: PR4 broke the batch. passes PR5 to PR6 PR5 and PR6, back in the queue, pass in a new batch. passes Pull requests PR1 merged PR2 merged PR3 merged PR4 PR4 leaves the queue without being tested on its own. PR4 leaves the queue without being tested on its own. removed PR5 merged PR6 merged
A batch of 6 fails with 3 parallel checks, and PR4 broke it: two rounds of splitting find it, every other pull request merges, and PR4 leaves without ever being tested on its own. Green passed, red failed, dashed was cancelled. Not to scale.

The batches tested on top of the failed one are cancelled, and their pull requests are tested again. While the split runs, the batches report a batch status such as bisecting, waiting_for_previous_batches or waiting_for_requeue in the dashboard and the CLI. The event that starts a split counts the last pull request as a sub-batch of its own, so its number_of_sub_batches is one more than the number of parts.

A split runs on max_parallel_checks checks, or on the per-scope capacity of its scope when that is smaller. When the base branch requires branches to be up to date before merging and the queue does not use fast-forward, Mergify checks one batch at a time, so a split gets one check. With fast-forward, the parts are tested one after the other.

In serial mode, and for a barrier in parallel mode, the whole queue waits on the split, so it takes every check, as does any split under batch_max_failure_resolution_attempts. Otherwise, in parallel and isolated modes, a split takes the share of the checks that matches the share of waiting pull requests it holds up, and leaves the rest to other scopes:

checks running 26 in backend 13 in backend 0 in backend check 1 PR1–6 PR1–6 PR1–4 PR1–4 PR1–4 PR1–4 check 2 PR1–12 PR1–12 PR1–8 PR1–8 PR1–8 PR1–8 check 3 backend backend PR1–12 PR1–12 PR1–12 PR1–12 check 4 backend backend backend backend check 5 backend backend backend backend check 6 backend backend backend backend checks running 26 in backend 13 in backend 0 in backend check 1 PR1–6 PR1–6 PR1–4 PR1–4 PR1–4 PR1–4 check 2 PR1–12 PR1–12 PR1–8 PR1–8 PR1–8 PR1–8 check 3 backend backend PR1–12 PR1–12 PR1–12 PR1–12 check 4 backend backend backend backend check 5 backend backend backend backend check 6 backend backend backend backend
In a parallel queue with 6 checks, a failed frontend batch of 13 takes the share of the checks matching the share of waiting pull requests it holds up, and leaves the rest to backend. With nothing else waiting, it stops at 3 parts: a wider cut would not find the culprit sooner. Ringed checks are the split's tests.

Picking a batch size that splits cleanly

Section titled Picking a batch size that splits cleanly

Each round of a split runs up to N tests at once, N being the checks it gets. Some batch sizes keep all N busy in every round, whichever part fails:

Parallel checks (N) Sizes that keep every check busy
2 1, 3, 7, 15, 31
3 1, 4, 13, 40
4 1, 5, 21, 85
5 1, 6, 31
rounds to find the culprit, at most 0 1 2 3 4 1 4 13 40 pull requests in the failed batch 4: 1 round 4: 1 round 13: 2 rounds 13: 2 rounds 40: 3 rounds 40: 3 rounds rounds to find the culprit, at most 0 1 2 3 4 1 4 13 40 pull requests in the failed batch 4: 1 round 4: 1 round 13: 2 rounds 13: 2 rounds 40: 3 rounds 40: 3 rounds
With 3 parallel checks, the worst case gains a round past 4, 13 and 40 pull requests, so a batch of 7 costs as many rounds as a batch of 13.

So set batch_size, or the max of a dynamic batch size, to one of these sizes to bound the worst case: with 3 parallel checks, max: 13 never needs more than 2 rounds. A split that shares its checks with other scopes can take more rounds.

batch_max_failure_resolution_attempts caps how many rounds of splitting a failure goes through. When a part holding several pull requests fails in the last round, it is not split again: all its pull requests leave the queue with the batch-max-failure-resolution-attempts dequeue reason. With 0, a failed batch is never split, and all its pull requests leave the queue. The cap counts the rounds behind each part: pull requests that go back to the queue start over in a new batch, with a fresh count.

In a queue whose split gets one check, the last round tests only the first part. If it passes, it merges, and every pull request after it leaves the queue untested with the dropped-by-bisection-elimination dequeue reason: the failure lies among them, and no round is left to tell them apart.

Your CI runs again on every part. With the Mergify plugin for your test framework, Fail First starts each part with the tests that failed in the batch it came from, and stops the run as soon as one of them fails again, so a part that holds the culprit costs a few tests instead of your whole suite. A part that passes has still run every test.

With skip_intermediate_results, a batch that passes stands in for the batches it was tested on top of: they merge without waiting for their own checks, even one whose checks failed and that Mergify is retrying. Leave it off where every batch must pass its own checks.

.mergify.ymlyaml
merge_queue:
skip_intermediate_results: true

With scopes, a pass counts only in the scopes its batch was asked to test, and an earlier batch merges once the passes on top of it cover every scope it touches. A barrier touches every scope.

Tests batch 1: frontend, api Batch 1's own test is still running when batches 2 and 3 cover both its scopes. not waited for batch 2: frontend Batch 2 passes, but covers only frontend: batch 1 keeps waiting, and so does batch 2. passes batch 3: api Batch 3 passes on api: with batch 2, every scope of batch 1 is covered. passes Batches batch 1 merged batch 2 Batch 2 passed first, but merges only after batch 1. Batch 2 passed first, but merges only after batch 1. merged batch 3 merged Tests batch 1: frontend, api Batch 1's own test is still running when batches 2 and 3 cover both its scopes. not waited for batch 2: frontend Batch 2 passes, but covers only frontend: batch 1 keeps waiting, and so does batch 2. passes batch 3: api Batch 3 passes on api: with batch 2, every scope of batch 1 is covered. passes Batches batch 1 merged batch 2 Batch 2 passed first, but merges only after batch 1. Batch 2 passed first, but merges only after batch 1. merged batch 3 merged
With skip_intermediate_results, batch 1 merges once the passes on top of it cover every scope it touches: batch 2's pass covers frontend only, and batch 3's adds api. Green passed, dashed was not waited for. Not to scale.

Inside a split, a part that passes stands in for the parts before it the same way. A batch whose failure Mergify acts on, by splitting it or removing its pull request, cancels every batch behind it, so no later pass stands in for it. On a fast-forward or merge-batch queue, a pass never stands in for a batch or part whose checks failed. Isolated mode does not support the option.

A check that diffs against checking_base_sha, such as a linter on the changed files, reads only the passing batch’s own pull requests, yet the batches ahead still merge on its pass. To have such a check cover them, diff against the base branch instead.

batch_max_failure_resolution_attempts and skip_intermediate_results are listed with the other batch options.

Was this page helpful?