CI Insights Setup – Jenkins

1. Enable CI Insights on your repositories by visiting the GitHub Integration page (docs).

2. Install and configure the Mergify Jenkins plugin:

   Installation:

   • Go to Manage Jenkins → Manage Plugins
   • Select the Available tab
   • Search for Mergify and select the checkbox next to Mergify Plugin
   • Install the plugin using one of the install buttons at the bottom
   • To verify installation, search for Mergify Plugin on the Installed tab
   

   Configuration: After installation, you need to add the Mergify CI Insights API Key for each GitHub organization you want to support. Go to Manage Jenkins → Configure System and configure the Mergify plugin with your API keys.

   

   Note: It’s highly recommended to set the GitHub project URL for each job if you’re not using the GitHub Branch Source plugin. This ensures proper correlation between Jenkins jobs and GitHub repositories.

3. Configure your Jenkins pipeline job with the MERGIFY_TOKEN environment variable. First, get your token from the Mergify dashboard by going to Settings > CI Insights:

   

   Then add the token to Jenkins credentials and configure your pipeline to use it:

   pipeline {
       agent any
       environment {
           MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
       }
       ...
   }

   Make sure to store your Mergify token as a Jenkins credential with the ID MERGIFY_TOKEN.

4. Click on CI Insights in the Mergify dashboard navigation. You should start seeing your Jenkins job runs appear:

   

To effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.

Before setting up flaky test detection, ensure you have:

1. Enabled CI Insights for your repository

2. Set up your MERGIFY_TOKEN in Jenkins Credentials.

3. Configured test integration (e.g., pytest-mergify or mergify cli upload) for your test framework

The recommended approach is to use a scheduled job that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 parallel executions:

• Example with native integration (e.g. pytest-mergify)

pipeline {
  agent any
  environment {
      MERGIFY_TEST_FLAKY_DETECTION = "true"
      MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
      RUN_COUNT = '5'
  }
  triggers {
    cron('0 H/12 * * 1-5')  // Every 12 hours, Monday to Friday
  }

  stages {
    stage('Test') {
      steps {
        sh '''
          # Run your test suite multiple times to detect flakiness
          set +e
          failed=0
          for i in $(seq "$RUN_COUNT"); do
            pytest tests/
            exit_code=$?
            if [ $exit_code -ne 0 ]; then
              failed=1
            fi
          done
          exit $failed
        '''
      }
    }
  }
}

• Examples with mergify cli upload integration

pipeline {
  agent any
  environment {
      MERGIFY_TEST_FLAKY_DETECTION = "true"
      MERGIFY_TOKEN = credentials('MERGIFY_TOKEN')
      RUN_COUNT = '5'
  }
  triggers {
    cron('0 H/12 * * 1-5')  // Every 12 hours, Monday to Friday
  }

  stages {
    stage('Test') {
      steps {
        sh '''
          # Run your test suite multiple times to detect flakiness
          set +e
          for i in $(seq "$RUN_COUNT"); do
            pytest tests/
            # Run your test command with unique output file, e.g.:
            # npm test -- --reporters=default --reporters=jest-junit --outputFile=test-results-$i.xml
          done
          set -e
          mergify ci junit-process test-results-*.xml
        '''
      }
    }
  }
}

• Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage

• Default Branch Only: Focus on the default branch where flaky tests have the most impact

• Weekday Schedule: The example runs Monday to Friday (1-5) to avoid running when no changes are made on the code

• Job name: Ensure the job name is the same as the one running tests on your pull request, allowing CI-Insights to identify them. It can be overriden with MERGIFY_JOB_NAME if needed.

• MERGIFY_TEST_FLAKY_DETECTION: To tell CI Insights that flaky test detection is enabled
• MERGIFY_TOKEN: Required for uploading test results to CI Insights
• RUN_COUNT: Number of times to execute each test within a single job
• MERGIFY_JOB_NAME: The job name reported to CI Insights