Implementing a Two-Step CI with Mergify
Run essential tests on every PR and comprehensive tests before merging, optimizing CI time and resources.
Efficiency in CI resources is crucial for fast-moving projects. A two-step CI approach allows developers to prioritize and sequence tests based on their importance and the stage of the PR.
Running all tests on every PR might be exhaustive and time-consuming. It can also drain CI resources, especially if the full suite involves comprehensive, heavy, or time-intensive tests. Instead, separating tests into two distinct groups can be beneficial:
Preliminary Tests: These are the tests you run on every PR preliminary after they're created or updated, usually encompassing code linters and essential unit tests that are faster and critical to ensuring basic functionality.
Pre-Merge Tests: Before merging a pull request, a more comprehensive set of tests (including functional tests, integration tests, and perhaps some performance tests) are run to ensure robustness. These tests might be lengthier and more resource-intensive.
By structuring CI this way, you get preliminary feedback on the PR's essential aspects while ensuring a thorough check before the merge. This also aids in conserving CI resources, as not all tests are run at every PR iteration.
With Mergify's merge queue, you can further optimize this process. If you're batching multiple PRs for a merge, instead of running the pre-merge tests for each PR individually, you can run them once for the entire batch. This can result in significant savings, especially when the pre-merge tests are extensive.
To implement this with Mergify, you'll need to:
Configure your CI system to recognize the distinction between preliminary and pre-merge tests.
This is typically done by configuring the CI to run all the preliminary tests on every branches, while pre-merge tests would only be run on branches created by the merge queue. Those branches are prefixed by
mergify/merge-queue/by default (see
Configure Mergify to create a new branch for each pull request in the queue.
By default, Mergify has an optimized behavior that does not always create a new branch for every PR in the queue. You need to set
falsein your queue definition inside the
Adjust your Mergify configuration to act based on the outcomes of these test groups.
This is typically done by setting the right
merge_conditions. The former should include only the CI expected to enter the queue (the preliminary tests) while the later should list the pre-merge tests.
To help visualize this, here's an example using GitHub Actions:
name: CIon:pull_request:branches:- mainjobs:unit-tests:runs-on: ubuntu-lateststeps:- name: Checkout codeuses: actions/checkout@v2- name: Run Unit Testsrun: make unit-testspre-merge-tests:if: startsWith(github.head_ref, 'mergify/merge-queue/')runs-on: ubuntu-lateststeps:- name: Checkout codeuses: actions/checkout@v2- name: Run Pre-Merge Testsrun: make pre-merge-tests
In this setup,
unit-tests will run for every PR, while
only run for PRs that are in Mergify's merge queue.
Adjusting this approach to different CI systems will be based on their specific syntax and triggers, but the general principle remains the same.
The Mergify configuration should looks like:
queue_rules- name: defaultallow_inplace_checks: falsequeue_conditions:- check-success = unit-testsmerge_conditions:- check-success = pre-merge-tests
Implementing a two-step CI ensures that resources are optimized, developers get rapid feedback, and the code's robustness is assured before merging.