CI Insights Setup – GitHub Actions

This page explains how to enable CI Insights for a repository using GitHub Actions and how to configure repeated test executions for flaky test detection.

1. Enable CI Insights on your repositories by visiting the GitHub Integration page (docs).

2. In order to upload your test results, you’ll need to configure your test. Before doing so, make sure you set your MERGIFY_TOKEN in GitHub Actions secrets.

   To find your token in the Mergify dashboard, go to Settings > CI Insights:

   

   Once copied, paste it into your GitHub Actions secrets:

   

   This token will be used to upload test reports.

3. Click on CI Insights in the Mergify dashboard navigation. You should start seeing your GitHub Actions job runs appear:

   

To effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.

Before setting up flaky test detection, ensure you have:

1. Enabled CI Insights for your repository

2. Set up your MERGIFY_TOKEN in GitHub Actions secrets

3. Configured test integration (e.g., pytest-mergify or mergify cli upload) for your test framework

The recommended approach is to use a scheduled workflow that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 parallel executions:

• Example with native integration (e.g. pytest-mergify)

name: Continuous Integration
on:
  pull_request:
  schedule:
    - cron: '0 */12 * * 1-5'  # Every 12 hours, Monday to Friday

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup your environment
        # Add your specific setup steps here (Python, Node.js, etc.)

      - name: Run Tests
        env:
          MERGIFY_TEST_FLAKY_DETECTION: "true"
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
          RUN_COUNT: 5  # Number of times to run each test
        run: |
          # Run your test suite multiple times to detect flakiness
          set +e
          failed=0
          for i in $(seq "$RUN_COUNT"); do
            pytest tests/
            exit_code=$?
            if [ $exit_code -ne 0 ]; then
              failed=1
            fi
          done
          exit $failed

• Examples with mergify cli upload integration

name: CI
on:
  pull_request:
  schedule:
    - cron: '0 */12 * * 1-5'  # Every 12 hours, Monday to Friday

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup your environment
        # Add your specific setup steps here (Python, Node.js, etc.)

      # Modified: Run tests multiple times for flaky detection
      - name: Run tests
        env:
          # Run tests 5 times on schedule, once on pull_request
          RUN_COUNT: ${{ github.event_name == 'schedule' && 5 || 1 }}
        run: |
          # Run the test suite based on RUN_COUNT
          for i in $(seq $RUN_COUNT); do
            echo "Running test suite - attempt $i of $RUN_COUNT"

            # Run your test command with unique output file, e.g.:
            # npm test -- --reporters=default --reporters=jest-junit --outputFile=test-results-$i.xml
          done

      # Upload all test results to CI Insights
      - name: Upload test results
        if: always()
        uses: mergifyio/gha-mergify-ci@v6
        with:
          flaky_test_detection: true
          token: ${{ secrets.MERGIFY_TOKEN }}
          report_path: test-results-*.xml

• Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage

• Default Branch Only: Focus on the default branch where flaky tests have the most impact

• Weekday Schedule: The example runs Monday to Friday (1-5) to avoid running when no changes are made on the code

• Job name: Ensure the job name is the same as the one running tests on your pull request, allowing CI-Insights to identify them. It can be overriden with MERGIFY_JOB_NAME if needed.

• MERGIFY_TEST_FLAKY_DETECTION: To tell CI Insights that flaky test detection is enabled
• MERGIFY_TOKEN: Required for uploading test results to CI Insights
• RUN_COUNT: Number of times to execute each test within a single job
• MERGIFY_JOB_NAME: The job name reported to CI Insights