CI Insights Setup – Jenkins
Enable Mergify CI Insights and configure flaky test detection using Jenkins.
Enabling CI Insights for Jenkins
Section titled Enabling CI Insights for Jenkins-
Enable CI Insights for each repository you want it to cover, on the Repositories page of the dashboard (docs).
-
Install and configure the Mergify Jenkins plugin:
Installation:
- Go to
Manage Jenkins→Plugins - Select the
Available pluginstab - Search for
Mergifyand select the checkbox next to theMergifyentry - Install the plugin using one of the install buttons
- To verify installation, search for
Mergifyon theInstalled pluginstab
Configuration: After installation, you need to add a Mergify application key granted only CI Reporting for each GitHub organization you want to support. Go to
Manage Jenkins→Configure Systemand configure the Mergify plugin with your application keys.
Note: It’s highly recommended to set the GitHub project URL for each job if you’re not using the GitHub Branch Source plugin. This ensures proper correlation between Jenkins jobs and GitHub repositories.
- Go to
-
Configure your Jenkins pipeline job with the
MERGIFY_TOKENenvironment variable. First, create an application key granted only CI Reporting, as described in the API Usage documentation.Then add the key to Jenkins credentials and configure your pipeline to use it:
pipeline { agent any stages { stage('Test') { environment { MERGIFY_TOKEN = credentials('MERGIFY_TOKEN') // Jenkins does not set CI, and the test plugins only report when it is set. CI = 'true' } steps { sh 'pytest' } } } }Make sure to store your application key as a Jenkins credential (kind Secret text) with the ID
MERGIFY_TOKEN. Set these variables on the stage that runs the tests, so they do not leak into build or lint stages.CI: the test plugins (pytest, RSpec, Minitest, Vitest, Playwright, Jest, Mocha) report only whenCIis set, and Jenkins does not set it. If your tests behave differently whenCIis set, setMERGIFY_CI_ENABLE=trueinstead.Multibranch pipelines: every branch and every pull request of the project reports its tests under the same pipeline name and job name, the multibranch project (
my-repo, ormy-org/my-repoin an organization folder), not the branch’s or pull request’s own job (my-repo/PR-123).MERGIFY_TEST_JOB_NAME, when set, replaces the job name. Tests from a job outside a multibranch project are filed underJOB_NAME, unless the job is named after its ownBRANCH_NAME, as per-branch jobs made by a seed job are: those are grouped under their folder the same way.Job runs: the latest release of the Mergify Jenkins plugin reports a job’s runs under the same names as its tests: the multibranch project for every branch and pull request,
JOB_NAMEfor any other job, as both pipeline name and job name. Each run keeps its own branch, so you can filter them by branch. The plugin does not readMERGIFY_TEST_JOB_NAME, and a per-branch job whose tests are grouped under its folder (my-folder) reports its runs under its ownJOB_NAME(my-folder/main).Shell: Jenkins runs
shsteps with/bin/sh, which is oftendashand does not supportset -o pipefail. Start a script that uses it with#!/bin/bash. -
Click on
CI Insightsin the Mergify dashboard navigation. You should start seeing your Jenkins job runs appear:
Setting up flaky test detection
Section titled Setting up flaky test detectionTo effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.
Prerequisites
Section titled PrerequisitesBefore setting up flaky test detection, ensure you have:
-
Enabled CI Insights for your repository
-
Created an application key granted only CI Reporting and set it as
MERGIFY_TOKENin Jenkins credentials. -
Configured test integration (e.g.,
pytest-mergifyor JUnit report upload withmergify ci junit-process) for your test framework
Jenkins job setup
Section titled Jenkins job setupThe recommended approach is to use a scheduled job that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 repeated executions:
- Example with native integration (e.g.
pytest-mergify)
pipeline { agent any environment { MERGIFY_TOKEN = credentials('MERGIFY_TOKEN') CI = 'true' RUN_COUNT = '5' } triggers { cron('0 H/12 * * 1-5') // Every 12 hours, Monday to Friday }
stages { stage('Test') { steps { sh ''' # Run your test suite multiple times to detect flakiness set +e failed=0 for i in $(seq "$RUN_COUNT"); do pytest tests/ exit_code=$? if [ $exit_code -ne 0 ]; then failed=1 fi done exit $failed ''' } } }}-
Example with JUnit report upload via
mergify ci junit-processThis example calls the Mergify CLI, so the agent needs it on its
PATH. Install it as part of the pipeline or bake it into your agent image; see the CLI installation guide.
pipeline { agent any environment { MERGIFY_TOKEN = credentials('MERGIFY_TOKEN') RUN_COUNT = '5' } triggers { cron('0 H/12 * * 1-5') // Every 12 hours, Monday to Friday }
stages { stage('Test') { steps { sh ''' # Run your test suite multiple times to detect flakiness set +e for i in $(seq "$RUN_COUNT"); do # Write one JUnit report per run. With another framework, use its # JUnit option, e.g.: ./vendor/bin/phpunit --log-junit test-results-$i.xml pytest tests/ --junitxml=test-results-$i.xml done set -e # Quote the glob so Mergify expands it rather than the shell; # recommended when many reports match. mergify ci junit-process 'test-results-*.xml' ''' } } }}Configuration tips
Section titled Configuration tips-
Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage
-
Default Branch Only: Focus on the default branch where flaky tests have the most impact
-
Weekday Schedule: The example runs Monday to Friday (
1-5) to avoid running when no changes are made to the code -
Job name: Test Engine records the job a test ran in as part of that test’s identity, alongside the branch and the pipeline, so a job name that varies between runs splits one test into several and leaves nothing to compare. Keep it stable across the repeated runs. By default it comes from Jenkins’
JOB_NAME, or from the multibranch project in a multibranch pipeline. SetMERGIFY_TEST_JOB_NAMEin the pipeline’senvironmentblock to report a different one:environment { MERGIFY_TEST_JOB_NAME = 'unit-tests' }The pipeline name has the same default, and
MERGIFY_TEST_JOB_NAMEdoes not change it, so the variable cannot file the tests of two Jenkins pipelines together.
Key environment variables
Section titled Key environment variables| Variable | Purpose | Default |
|---|---|---|
MERGIFY_TEST_JOB_NAME |
Name the job's test results are reported under, instead of the CI job's own name. Set one per environment, never per shard: a name per shard splits a test's history. | |
MERGIFY_TOKEN |
The Mergify credential: an application key in CI. Without it the test reporters upload nothing. A |
Nothing in Mergify reads RUN_COUNT. The pipelines above declare it to drive
their own loop, so call it whatever you like, or drop it and repeat the suite
some other way.
Was this page helpful?
Thanks for your feedback!