CI Insights Setup – Jenkins

1. Enable CI Insights for each repository you want it to cover, on the Repositories page of the dashboard (docs).

2. Install and configure the Mergify Jenkins plugin:

   Installation:

   • Go to Manage Jenkins → Plugins
   • Select the Available plugins tab
   • Search for Mergify and select the checkbox next to the Mergify entry
   • Install the plugin using one of the install buttons
   • To verify installation, search for Mergify on the Installed plugins tab
   

   Configuration: After installation, you need to add a Mergify application key with ci scope for each GitHub organization you want to support. Go to Manage Jenkins → Configure System and configure the Mergify plugin with your application keys.

   

   Note: It’s highly recommended to set the GitHub project URL for each job if you’re not using the GitHub Branch Source plugin. This ensures proper correlation between Jenkins jobs and GitHub repositories.

3. Configure your Jenkins pipeline job with the MERGIFY_TOKEN environment variable. First, create an application key with ci scope as described in the API Usage documentation.

   Then add the key to Jenkins credentials and configure your pipeline to use it:

   pipeline {
       agent any
       stages {
           stage('Test') {
               environment {
                   MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
                   // Jenkins does not set CI, and the test plugins only report when it is set.
                   CI = 'true'
               }
               steps {
                   sh 'pytest'
               }
           }
       }
   }
   

   Make sure to store your application key as a Jenkins credential (kind Secret text) with the ID MERGIFY_TOKEN. Set these variables on the stage that runs the tests, so they do not leak into build or lint stages.

   CI: the test plugins (pytest, RSpec, Minitest, Vitest, Playwright, Jest, Mocha) report only when CI is set, and Jenkins does not set it. If your tests behave differently when CI is set, set MERGIFY_CI_ENABLE=true instead.

   Multibranch pipelines: each branch and each pull request keeps its own test history, and no setting changes that. The plugins file tests under JOB_NAME, which in a multibranch pipeline includes the branch or pull request (my-repo/PR-123), as both the pipeline name and the job name. MERGIFY_TEST_JOB_NAME replaces only the job name: do not set it for this purpose.

   Shell: Jenkins runs sh steps with /bin/sh, which is often dash and does not support set -o pipefail. Start a script that uses it with #!/bin/bash.

4. Click on CI Insights in the Mergify dashboard navigation. You should start seeing your Jenkins job runs appear:

   

To effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.

Before setting up flaky test detection, ensure you have:

1. Enabled CI Insights for your repository

2. Created an application key with ci scope and set it as MERGIFY_TOKEN in Jenkins credentials.

3. Configured test integration (e.g., pytest-mergify or JUnit report upload with mergify ci junit-process) for your test framework

The recommended approach is to use a scheduled job that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 repeated executions:

• Example with native integration (e.g. pytest-mergify)

pipeline {
  agent any
  environment {
      MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
      CI = 'true'
      RUN_COUNT = '5'
  }
  triggers {
    cron('0 H/12 * * 1-5')  // Every 12 hours, Monday to Friday
  }

  stages {
    stage('Test') {
      steps {
        sh '''
          # Run your test suite multiple times to detect flakiness
          set +e
          failed=0
          for i in $(seq "$RUN_COUNT"); do
            pytest tests/
            exit_code=$?
            if [ $exit_code -ne 0 ]; then
              failed=1
            fi
          done
          exit $failed
        '''
      }
    }
  }
}

• Example with JUnit report upload via mergify ci junit-process

  This example calls the Mergify CLI, so the agent needs it on its PATH. Install it as part of the pipeline or bake it into your agent image; see the CLI installation guide.

pipeline {
  agent any
  environment {
      MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
      RUN_COUNT = '5'
  }
  triggers {
    cron('0 H/12 * * 1-5')  // Every 12 hours, Monday to Friday
  }

  stages {
    stage('Test') {
      steps {
        sh '''
          # Run your test suite multiple times to detect flakiness
          set +e
          for i in $(seq "$RUN_COUNT"); do
            # Write one JUnit report per run. With another framework, use its
            # JUnit option, e.g.: ./vendor/bin/phpunit --log-junit test-results-$i.xml
            pytest tests/ --junitxml=test-results-$i.xml
          done
          set -e
          # Quote the glob so Mergify expands it rather than the shell;
          # recommended when many reports match.
          mergify ci junit-process 'test-results-*.xml'
        '''
      }
    }
  }
}

• Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage

• Default Branch Only: Focus on the default branch where flaky tests have the most impact

• Weekday Schedule: The example runs Monday to Friday (1-5) to avoid running when no changes are made to the code

• Job name: Test Engine records the job a test ran in as part of that test’s identity, alongside the branch and the pipeline, so a job name that varies between runs splits one test into several and leaves nothing to compare. Keep it stable across the repeated runs. By default it comes from Jenkins’ JOB_NAME. Set MERGIFY_TEST_JOB_NAME in the pipeline’s environment block to report a different one:

  environment {
      MERGIFY_TEST_JOB_NAME = 'unit-tests'
  }
  

  MERGIFY_TEST_JOB_NAME does not stabilize a multibranch pipeline whose JOB_NAME carries the branch. The pipeline name comes from JOB_NAME as well, and the variable replaces only the job name, so the branch still splits the history. The Multibranch pipelines note in Enabling CI Insights for Jenkins covers that case.

Nothing in Mergify reads RUN_COUNT. The pipelines above declare it to drive their own loop, so call it whatever you like, or drop it and repeat the suite some other way.