View as Markdown

CI Insights Setup – Jenkins

Enable Mergify CI Insights and configure flaky test detection using Jenkins.


Enabling CI Insights for Jenkins

Section titled Enabling CI Insights for Jenkins
  1. Enable CI Insights for each repository you want it to cover, on the Repositories page of the dashboard (docs).

  2. Install and configure the Mergify Jenkins plugin:

    Installation:

    • Go to Manage Jenkins → Plugins
    • Select the Available plugins tab
    • Search for Mergify and select the checkbox next to the Mergify entry
    • Install the plugin using one of the install buttons
    • To verify installation, search for Mergify on the Installed plugins tab
    CI Insights Jenkins installation

    Configuration: After installation, you need to add a Mergify CI key for each GitHub organization you want to support. Go to Manage Jenkins → Configure System and configure the Mergify plugin with your CI keys.

    CI Insights Jenkins configuration

    Note: It’s highly recommended to set the GitHub project URL for each job if you’re not using the GitHub Branch Source plugin. This ensures proper correlation between Jenkins jobs and GitHub repositories.

  3. Configure your Jenkins pipeline job with the MERGIFY_TOKEN environment variable. First, create a CI key for Jenkins.

    Then add the key to Jenkins credentials and configure your pipeline to use it:

    pipeline {
        agent any
        stages {
            stage('Test') {
                environment {
                    MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
                    // Jenkins does not set CI, and the test plugins only report when it is set.
                    CI = 'true'
                }
                steps {
                    sh 'pytest'
                }
            }
        }
    }
    

    Make sure to store your application key as a Jenkins credential (kind Secret text) with the ID MERGIFY_TOKEN. Set these variables on the stage that runs the tests, so they do not leak into build or lint stages.

    CI: the test plugins (pytest, RSpec, Minitest, Vitest, Playwright, Jest, Mocha) report only when CI is set, and Jenkins does not set it. If your tests behave differently when CI is set, set MERGIFY_CI_ENABLE=true instead.

    Multibranch pipelines: every branch and every pull request of the project reports its tests under the same pipeline name and job name, the multibranch project (my-repo, or my-org/my-repo in an organization folder), not the branch’s or pull request’s own job (my-repo/PR-123). MERGIFY_TEST_JOB_NAME, when set, replaces the job name. Tests from a job outside a multibranch project are filed under JOB_NAME, unless the job is named after its own BRANCH_NAME, as per-branch jobs made by a seed job are: those are grouped under their folder the same way.

    Job runs: the latest release of the Mergify Jenkins plugin reports a job’s runs under the same names as its tests: the multibranch project for every branch and pull request, JOB_NAME for any other job, as both pipeline name and job name. Each run keeps its own branch, so you can filter them by branch. The plugin does not read MERGIFY_TEST_JOB_NAME, and a per-branch job whose tests are grouped under its folder (my-folder) reports its runs under its own JOB_NAME (my-folder/main).

    Shell: Jenkins runs sh steps with /bin/sh, which is often dash and does not support set -o pipefail. Start a script that uses it with #!/bin/bash.

  4. Click on CI Insights in the Mergify dashboard navigation. You should start seeing your Jenkins job runs appear:

    CI Insights Jobs CI Insights Jobs

Setting up flaky test detection

Section titled Setting up flaky test detection

To effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.

Before setting up flaky test detection, ensure you have:

  1. Enabled CI Insights for your repository

  2. Created a CI key and set it as MERGIFY_TOKEN in Jenkins credentials.

  3. Configured test integration (e.g., pytest-mergify or JUnit report upload with mergify ci junit-process) for your test framework

The recommended approach is to use a scheduled job that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 repeated executions:

  • Example with native integration (e.g. pytest-mergify)
groovy
pipeline {
agent any
environment {
MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
CI = 'true'
RUN_COUNT = '5'
}
triggers {
cron('0 H/12 * * 1-5') // Every 12 hours, Monday to Friday
}
stages {
stage('Test') {
steps {
sh '''
# Run your test suite multiple times to detect flakiness
set +e
failed=0
for i in $(seq "$RUN_COUNT"); do
pytest tests/
exit_code=$?
if [ $exit_code -ne 0 ]; then
failed=1
fi
done
exit $failed
'''
}
}
}
}
  • Example with JUnit report upload via mergify ci junit-process

    This example calls the Mergify CLI, so the agent needs it on its PATH. Install it as part of the pipeline or bake it into your agent image; see the CLI installation guide.

groovy
pipeline {
agent any
environment {
MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
RUN_COUNT = '5'
}
triggers {
cron('0 H/12 * * 1-5') // Every 12 hours, Monday to Friday
}
stages {
stage('Test') {
steps {
sh '''
# Run your test suite multiple times to detect flakiness
set +e
for i in $(seq "$RUN_COUNT"); do
# Write one JUnit report per run. With another framework, use its
# JUnit option, e.g.: ./vendor/bin/phpunit --log-junit test-results-$i.xml
pytest tests/ --junitxml=test-results-$i.xml
done
set -e
# Quote the glob so Mergify expands it rather than the shell;
# recommended when many reports match.
mergify ci junit-process 'test-results-*.xml'
'''
}
}
}
}
  • Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage

  • Default Branch Only: Focus on the default branch where flaky tests have the most impact

  • Weekday Schedule: The example runs Monday to Friday (1-5) to avoid running when no changes are made to the code

  • Job name: Test Engine records the job a test ran in as part of that test’s identity, alongside the branch and the pipeline, so a job name that varies between runs splits one test into several and leaves nothing to compare. Keep it stable across the repeated runs. By default it comes from Jenkins’ JOB_NAME, or from the multibranch project in a multibranch pipeline. Set MERGIFY_TEST_JOB_NAME in the pipeline’s environment block to report a different one:

    environment {
        MERGIFY_TEST_JOB_NAME = 'unit-tests'
    }
    

    The pipeline name has the same default, and MERGIFY_TEST_JOB_NAME does not change it, so the variable cannot file the tests of two Jenkins pipelines together.

VariablePurpose
MERGIFY_TEST_JOB_NAME

Name the job's test results are reported under, instead of the CI job's own name. Set one per environment, never per shard: a name per shard splits a test's history.

MERGIFY_TOKEN

The Mergify credential: an application key in CI. A GitHub Actions job granted id-token: write needs none: without it, the test reporters, mergify ci junit-process and mergify ci scopes-send exchange the job's ID token for a Mergify token, when a trust policy of the organization allows the job. Without either, the test reporters upload nothing. A mergify command takes it when --token is not given, over the credential mergify auth login stored, and mergify auth tells when it does. The mergify stack commands send it to GitHub as their GitHub token, ahead of GITHUB_TOKEN, unless it holds a Mergify user token (mut_…).

Nothing in Mergify reads RUN_COUNT. The pipelines above declare it to drive their own loop, so call it whatever you like, or drop it and repeat the suite some other way.

Was this page helpful?