View as Markdown

CI Insights Setup – GitHub Actions

Enable Mergify CI Insights and configure flaky test detection using GitHub Actions.


This page explains how to enable CI Insights for a repository using GitHub Actions and how to configure repeated test executions for flaky test detection.

Enabling CI Insights for GitHub Actions

Section titled Enabling CI Insights for GitHub Actions
  1. Enable CI Insights for each repository you want it to cover, on the Repositories page of the dashboard (docs).

  2. Let your CI reach Mergify, then configure your test framework to upload your test results.

  3. Click on CI Insights in the Mergify dashboard navigation. You should start seeing your GitHub Actions job runs appear:

    CI Insights Jobs CI Insights Jobs

The mergifyio/gha-mergify-ci action runs on Linux, macOS and Windows runners.

Grant id-token: write to the job, or to the whole workflow. The job then trades its GitHub-signed ID token for a Mergify OIDC CI token, and no MERGIFY_TOKEN is stored:

yaml
permissions:
id-token: write # lets the job ask GitHub for an ID token
contents: read # actions/checkout needs it

A permissions block sets every permission it does not list to none: list the ones your other steps need too.

Mergify refuses the ID token until a trust policy of your organization matches the job. A GitHub organization Owner or a CI Admin writes one under Settings → Security & Access → CI Authentication → Trust Policies on the dashboard. OIDC Authentication covers trust policies, pull requests from forks and refused exchanges. If your workflows store MERGIFY_TOKEN today, move them off it.

Other CIs authenticate with a ci application key, and so do the GitHub Actions jobs OIDC leaves out: runs for pull requests from forks when your organization sends them secrets, and jobs that send scopes from the base branch (pull_request_target, workflow_run, issue_comment). Store the key as the MERGIFY_TOKEN repository secret and set it on the job:

yaml
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}

A MERGIFY_TOKEN that is set wins over the ID token. For pull requests from forks, set it on those runs only and drop the fork condition from the upload step, as Jobs that keep a key shows. Store the key in your Dependabot secrets too, since runs for Dependabot pull requests read only those.

GitHub gives the runs of pull requests from forks no ID token, so they report nothing:

  • The test plugins (pytest, RSpec, Minitest, Vitest, Playwright, Jest, Mocha) upload nothing and the tests run as usual. Nothing to change.

  • The mergifyio/gha-mergify-ci upload step fails without a token. Skip it on those runs:

    - name: Mergify Test Engine Upload
      if: (success() || failure()) && (!github.event.pull_request || github.event.pull_request.head.repo.full_name == github.repository)
      uses: mergifyio/gha-mergify-ci@v30
      with:
        report_path: junit.xml
        test_step_outcome: ${{ steps.tests.outcome }}
    

    If the test step has continue-on-error: true (needed for quarantine), the job’s result comes from the upload step. When that step is skipped, a failing test run would leave the job green, so add a step that fails it:

    - name: Fail when tests failed and no report was uploaded
      if: steps.tests.outcome == 'failure' && github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository
      run: exit 1
    

A workflow called with uses: ./.github/workflows/<file>.yml gets no permission its caller does not grant: give id-token: write to the calling job. A job of the called workflow that declares its own permissions must list it too. With an application key, the caller passes the MERGIFY_TOKEN secret instead, with secrets: inherit or by a name the called workflow declares under workflow_call.secrets.

A called workflow reports under its caller’s workflow name. When two called workflows have a job with the same id (often test), their results collide: set a distinct MERGIFY_TEST_JOB_NAME in each.

Setting up flaky test detection

Section titled Setting up flaky test detection

To effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.

Before setting up flaky test detection, ensure you have:

  1. Enabled CI Insights for your repository

  2. Let your CI reach Mergify

  3. Configured test integration (e.g., pytest-mergify or JUnit report upload) for your test framework

The recommended approach is to use a scheduled workflow that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 repeated executions:

  • Example with native integration (e.g. pytest-mergify)
yaml
name: Continuous Integration
on:
pull_request:
schedule:
- cron: '0 */12 * * 1-5' # Every 12 hours, Monday to Friday
jobs:
test:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
# Add your specific setup steps here (Python, Node.js, etc.)
- name: Run Tests
env:
RUN_COUNT: 5 # Number of times to run each test
run: |
# Run your test suite multiple times to detect flakiness
set +e
failed=0
for i in $(seq "$RUN_COUNT"); do
pytest tests/
exit_code=$?
if [ $exit_code -ne 0 ]; then
failed=1
fi
done
exit $failed
  • Example with JUnit report upload via gha-mergify-ci
yaml
name: CI
on:
pull_request:
schedule:
- cron: '0 */12 * * 1-5' # Every 12 hours, Monday to Friday
jobs:
test:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
# Add your specific setup steps here (Python, Node.js, etc.)
# Modified: Run tests multiple times for flaky detection
- name: Run tests
env:
# Run tests 5 times on schedule, once on pull_request
RUN_COUNT: ${{ github.event_name == 'schedule' && 5 || 1 }}
run: |
# Run the test suite based on RUN_COUNT
for i in $(seq $RUN_COUNT); do
echo "Running test suite - attempt $i of $RUN_COUNT"
# Run your test command with unique output file, e.g.:
# ./vendor/bin/phpunit --log-junit test-results-$i.xml
done
# Upload all test results to Test Engine
- name: Upload test results
if: (success() || failure()) && (!github.event.pull_request || github.event.pull_request.head.repo.full_name == github.repository)
uses: mergifyio/gha-mergify-ci@v30
with:
report_path: test-results-*.xml
  • Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage

  • Default Branch Only: Focus on the default branch where flaky tests have the most impact

  • Weekday Schedule: The example runs Monday to Friday (1-5) to avoid running when no changes are made to the code

  • Job name: Test Engine records the job a test ran in as part of that test’s identity, alongside the branch and the pipeline, so a job name that varies between runs splits one test into several and leaves nothing to compare. Keep it stable across the repeated runs. By default it comes from GITHUB_JOB, the workflow job’s identifier. Set MERGIFY_TEST_JOB_NAME on the job to report a different one. Matrix builds need it, because every matrix job reports the same GITHUB_JOB:

    jobs:
      test:
        strategy:
          matrix:
            version: [10, 12, 14]
        env:
          MERGIFY_TEST_JOB_NAME: test (${{ matrix.version }})
    

    Set it on the job rather than on a single step, so every step in the job reports the same name. A step’s env applies to that step alone, so a name set on the step that runs your tests never reaches the upload step. The exception is a job that runs two test suites: give each its own name, on each test step for a plugin, or with each upload step’s job_name input.

    When the upload goes through gha-mergify-ci, the action’s job_name input sets the name reported with that step’s results:

    - uses: mergifyio/gha-mergify-ci@v30
      with:
        report_path: test-results-*.xml
        job_name: test (${{ matrix.version }})
    

    job_name takes precedence over MERGIFY_TEST_JOB_NAME when both are set. Leave it out and the name set on the job applies.

VariablePurpose
MERGIFY_TEST_JOB_NAME

Name the job's test results are reported under, instead of the CI job's own name. Set one per environment, never per shard: a name per shard splits a test's history.

Nothing in Mergify reads RUN_COUNT. The examples above declare it to drive their own loop, so call it whatever you like, or drop it and repeat the suite some other way.

Was this page helpful?