When a Batch Fails
How Mergify finds the pull request that broke a batch, and how to bound that search or let a later pass skip a flaky failure.
When a batch fails, or its checks time out, Mergify splits it to find the pull request that broke it, removes that one, and merges the others. This needs no setup.
How a failed batch is split
Section titled How a failed batch is splitMergify cuts the batch into parts and tests them at once, each on top of the parts before it, as many as your parallel checks allow. The parts ahead of the first one that fails merge, the ones after it go back to the queue, and the failing part is split again. The last pull request of the batch is never retested: it already failed with the others, so if every part passes, it is the culprit.
The batches tested on top of the failed one are cancelled, and their pull requests are tested
again. While the split runs, the batches report a batch
status such as bisecting,
waiting_for_previous_batches or waiting_for_requeue in the dashboard and the
CLI. The event that starts a split counts the last pull request as a
sub-batch of its own, so its number_of_sub_batches is one more than the number of parts.
How many checks a split gets
Section titled How many checks a split getsA split runs on max_parallel_checks checks, or on the per-scope
capacity of its scope when that is
smaller. When the base branch requires branches to be up to date before merging and the queue does
not use fast-forward, Mergify checks one batch at
a time, so a split gets one check. With fast-forward, the parts are tested one after the other.
In serial mode, and for a
barrier in parallel mode, the whole queue
waits on the split, so it takes every check, as does any split under
batch_max_failure_resolution_attempts. Otherwise, in
parallel and
isolated modes, a split takes the share of the checks
that matches the share of waiting pull requests it holds up, and leaves the rest to other scopes:
Picking a batch size that splits cleanly
Section titled Picking a batch size that splits cleanlyEach round of a split runs up to N tests at once, N being the checks it gets. Some batch sizes keep all N busy in every round, whichever part fails:
| Parallel checks (N) | Sizes that keep every check busy |
|---|---|
| 2 | 1, 3, 7, 15, 31 |
| 3 | 1, 4, 13, 40 |
| 4 | 1, 5, 21, 85 |
| 5 | 1, 6, 31 |
So set batch_size, or the max of a dynamic batch
size, to one of these sizes to bound the worst case:
with 3 parallel checks, max: 13 never needs more than 2 rounds. A split that shares its checks
with other scopes can take more rounds.
Limiting the search
Section titled Limiting the searchbatch_max_failure_resolution_attempts caps how many rounds of splitting a failure goes through.
When a part holding several pull requests fails in the last round, it is not split again: all its
pull requests leave the queue with the
batch-max-failure-resolution-attempts dequeue
reason. With 0, a failed batch is never split, and all its pull requests leave the queue. The
cap counts the rounds behind each part: pull requests that go back to the queue start over in a
new batch, with a fresh count.
In a queue whose split gets one check, the last round tests only the first part. If it passes, it
merges, and every pull request after it leaves the queue untested with the
dropped-by-bisection-elimination dequeue
reason: the failure lies among them, and no round is left to tell them apart.
Making each split cheaper
Section titled Making each split cheaperYour CI runs again on every part. With the Mergify plugin for your test framework, Fail First starts each part with the tests that failed in the batch it came from, and stops the run as soon as one of them fails again, so a part that holds the culprit costs a few tests instead of your whole suite. A part that passes has still run every test.
Skipping a flaky failure
Section titled Skipping a flaky failureWith skip_intermediate_results, a batch that passes stands in for the batches it was tested on
top of: they merge without waiting for their own checks, even one whose checks failed and that
Mergify is retrying. Leave it off where every batch
must pass its own checks.
merge_queue: skip_intermediate_results: trueWith scopes, a pass counts only in the scopes its batch was asked to test, and an earlier batch merges once the passes on top of it cover every scope it touches. A barrier touches every scope.
Inside a split, a part that passes stands in for the parts before it the same way. A batch whose
failure Mergify acts on, by splitting it or removing its pull request, cancels every batch behind
it, so no later pass stands in for it. On a
fast-forward or
merge-batch queue, a pass never stands in for a
batch or part whose checks failed. Isolated mode does
not support the option.
A check that diffs against checking_base_sha, such as
a linter on the changed files, reads only the passing batch’s own pull requests, yet the batches
ahead still merge on its pass. To have such a check cover them, diff against the base branch
instead.
Options
Section titled Optionsbatch_max_failure_resolution_attempts and skip_intermediate_results are listed with the other
batch options.
See also
Section titled See alsoWas this page helpful?
Thanks for your feedback!