CI Insights Setup – GitHub Actions
Enable Mergify CI Insights and configure flaky test detection using GitHub Actions.
This page explains how to enable CI Insights for a repository using GitHub Actions and how to configure repeated test executions for flaky test detection.
Enabling CI Insights for GitHub Actions
Section titled Enabling CI Insights for GitHub Actions-
Enable CI Insights for each repository you want it to cover, on the Repositories page of the dashboard (docs).
-
Let your CI reach Mergify, then configure your test framework to upload your test results.
-
Click on
CI Insightsin the Mergify dashboard navigation. You should start seeing your GitHub Actions job runs appear:
The mergifyio/gha-mergify-ci action runs on Linux, macOS and Windows runners.
Authenticating your jobs
Section titled Authenticating your jobsLet your CI reach Mergify
Section titled Let your CI reach Mergify
Grant id-token: write to the job, or to the whole workflow. The job then trades its
GitHub-signed ID token for a Mergify OIDC CI token, and no MERGIFY_TOKEN is stored:
permissions: id-token: write # lets the job ask GitHub for an ID token contents: read # actions/checkout needs it
A permissions block sets every permission it does not list to none: list
the ones your other steps need too.
Mergify refuses the ID token until a trust policy of your organization matches the job. A
GitHub organization Owner or a CI Admin writes one under
Settings → Security & Access → CI Authentication → Trust Policies on the
dashboard. OIDC Authentication
covers trust policies, pull requests from forks and refused exchanges. If your workflows store
MERGIFY_TOKEN today,
move them off it.
Use an application key instead
Section titled Use an application key instead
Other CIs authenticate with a ci
application key, and so do the GitHub Actions
jobs OIDC leaves out: runs for pull requests from forks when your organization sends them
secrets, and jobs that send scopes from the base branch (pull_request_target,
workflow_run, issue_comment). Store the key as the
MERGIFY_TOKEN repository secret and set it on the job:
env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
A MERGIFY_TOKEN that is set wins over the ID token. For pull requests from forks,
set it on those runs only and drop the fork condition from the upload step, as
Jobs that keep a key shows. Store the key in
your
Dependabot secrets too, since runs for Dependabot pull requests read only those.
Pull requests from forks
Section titled Pull requests from forksGitHub gives the runs of pull requests from forks no ID token, so they report nothing:
-
The test plugins (pytest, RSpec, Minitest, Vitest, Playwright, Jest, Mocha) upload nothing and the tests run as usual. Nothing to change.
-
The
mergifyio/gha-mergify-ciupload step fails without a token. Skip it on those runs:- name: Mergify Test Engine Upload if: (success() || failure()) && (!github.event.pull_request || github.event.pull_request.head.repo.full_name == github.repository) uses: mergifyio/gha-mergify-ci@v30 with: report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }}If the test step has
continue-on-error: true(needed for quarantine), the job’s result comes from the upload step. When that step is skipped, a failing test run would leave the job green, so add a step that fails it:- name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository run: exit 1
Reusable workflows
Section titled Reusable workflowsA workflow called with uses: ./.github/workflows/<file>.yml gets no
permission its caller does not grant: give id-token: write to the calling
job. A job of the called workflow that declares its own permissions must
list it too. With an application key, the caller passes the MERGIFY_TOKEN
secret instead, with secrets: inherit or by a name the called workflow
declares under workflow_call.secrets.
A called workflow reports under its caller’s workflow name. When two called
workflows have a job with the same id (often test), their results collide:
set a distinct MERGIFY_TEST_JOB_NAME in each.
Setting up flaky test detection
Section titled Setting up flaky test detectionTo effectively detect flaky tests, you need to run the same tests multiple times on the same code (identified by the SHA1 of the repository). This section explains how to set up your CI to systematically detect flaky tests.
Prerequisites
Section titled PrerequisitesBefore setting up flaky test detection, ensure you have:
-
Enabled CI Insights for your repository
-
Configured test integration (e.g.,
pytest-mergifyor JUnit report upload) for your test framework
GitHub Actions setup
Section titled GitHub Actions setupThe recommended approach is to use a scheduled workflow that runs your test suite multiple times on the default branch. Here’s an example configuration that runs tests twice daily with 5 repeated executions:
- Example with native integration (e.g.
pytest-mergify)
name: Continuous Integrationon: pull_request: schedule: - cron: '0 */12 * * 1-5' # Every 12 hours, Monday to Friday
jobs: test: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - name: Checkout uses: actions/checkout@v4
# Add your specific setup steps here (Python, Node.js, etc.)
- name: Run Tests env: RUN_COUNT: 5 # Number of times to run each test run: | # Run your test suite multiple times to detect flakiness set +e failed=0 for i in $(seq "$RUN_COUNT"); do pytest tests/ exit_code=$? if [ $exit_code -ne 0 ]; then failed=1 fi done exit $failed- Example with JUnit report upload via
gha-mergify-ci
name: CIon: pull_request: schedule: - cron: '0 */12 * * 1-5' # Every 12 hours, Monday to Friday
jobs: test: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4
# Add your specific setup steps here (Python, Node.js, etc.)
# Modified: Run tests multiple times for flaky detection - name: Run tests env: # Run tests 5 times on schedule, once on pull_request RUN_COUNT: ${{ github.event_name == 'schedule' && 5 || 1 }} run: | # Run the test suite based on RUN_COUNT for i in $(seq $RUN_COUNT); do echo "Running test suite - attempt $i of $RUN_COUNT"
# Run your test command with unique output file, e.g.: # ./vendor/bin/phpunit --log-junit test-results-$i.xml done
# Upload all test results to Test Engine - name: Upload test results if: (success() || failure()) && (!github.event.pull_request || github.event.pull_request.head.repo.full_name == github.repository) uses: mergifyio/gha-mergify-ci@v30 with: report_path: test-results-*.xmlConfiguration tips
Section titled Configuration tips-
Frequency: Running twice daily (every 12 hours) provides a good balance between detection accuracy and resource usage
-
Default Branch Only: Focus on the default branch where flaky tests have the most impact
-
Weekday Schedule: The example runs Monday to Friday (
1-5) to avoid running when no changes are made to the code -
Job name: Test Engine records the job a test ran in as part of that test’s identity, alongside the branch and the pipeline, so a job name that varies between runs splits one test into several and leaves nothing to compare. Keep it stable across the repeated runs. By default it comes from
GITHUB_JOB, the workflow job’s identifier. SetMERGIFY_TEST_JOB_NAMEon the job to report a different one. Matrix builds need it, because every matrix job reports the sameGITHUB_JOB:jobs: test: strategy: matrix: version: [10, 12, 14] env: MERGIFY_TEST_JOB_NAME: test (${{ matrix.version }})Set it on the job rather than on a single step, so every step in the job reports the same name. A step’s
envapplies to that step alone, so a name set on the step that runs your tests never reaches the upload step. The exception is a job that runs two test suites: give each its own name, on each test step for a plugin, or with each upload step’sjob_nameinput.When the upload goes through
gha-mergify-ci, the action’sjob_nameinput sets the name reported with that step’s results:- uses: mergifyio/gha-mergify-ci@v30 with: report_path: test-results-*.xml job_name: test (${{ matrix.version }})job_nametakes precedence overMERGIFY_TEST_JOB_NAMEwhen both are set. Leave it out and the name set on the job applies.
Key environment variables
Section titled Key environment variables| Variable | Purpose |
|---|---|
MERGIFY_TEST_JOB_NAME |
Name the job's test results are reported under, instead of the CI job's own name. Set one per environment, never per shard: a name per shard splits a test's history. |
Nothing in Mergify reads RUN_COUNT. The examples above declare it to drive
their own loop, so call it whatever you like, or drop it and repeat the suite
some other way.
Was this page helpful?
Thanks for your feedback!