Test Engine Setup – CircleCI

CircleCI jobs can send their test results to Test Engine. Mergify does not collect CircleCI job runs for CI Insights’ job views: only test results reach it, through the test plugins or a JUnit report.

1. Enable CI Insights for the repository on the Repositories page of the dashboard (docs). Without it, uploads are refused.

2. Create a Mergify application key with the ci scope, as described in the API Usage documentation.

3. Store it as MERGIFY_TOKEN in CircleCI, never in .circleci/config.yml:

   • Project environment variable: Project Settings → Environment Variables (https://app.circleci.com/settings/project/github/<owner>/<repository>/environment-variables). Every job of the project receives it; nothing changes in the configuration.

   • Or a context, to share one key across projects: Organization Settings → Contexts. Jobs receive a context only when the workflow names it, so add context: [<your context>] to each test job under workflows.

CircleCI publishes no base branch for a pull request build, so Test Engine reads pull request builds like branch builds: Prevention, which works on pull requests, does not apply.

CircleCI does not pass project environment variables or contexts to builds of pull requests from forks unless the project setting Pass secrets to builds from forked pull requests is on. Those builds run without MERGIFY_TOKEN.

For pytest, RSpec, Vitest, Playwright, Jest and Mocha, install the plugin as its page describes. No step changes: CircleCI sets CI and the CIRCLE_* variables the plugin reads, and the job receives MERGIFY_TOKEN from the project or the context.

• Job names: each job reports under its name (CIRCLE_JOB). The legs of a matrix get their expanded names (such as test-3.12), so they are already distinct. Parallel containers of one job (parallelism) share the name, which is right: they run parts of the same suite.

• Tests inside Docker: the docker executor already runs your steps in the container, and the variables reach them. When a step starts the tests itself with docker run or Docker Compose, pass MERGIFY_TOKEN, CI, CIRCLECI and CIRCLE_* through to that container.

• Without a token (fork builds), the plugins upload nothing and your tests run as usual.

For any other framework, write a JUnit XML report from your test run, install the Mergify CLI in the job, and upload the report in the step that runs the tests:

Install the Mergify CLI in your pipeline and export MERGIFY_TOKEN. Run mergify ci junit-process after your tests:

set -o pipefail
exit_code=0
<your test command> || exit_code=$?
mergify ci junit-process \
  --test-exit-code "$exit_code" \
  junit.xml

Key Points:

• MERGIFY_TOKEN: Export this environment variable with your Mergify application key so junit-process can authenticate to the API.
• || exit_code=$?: CI shells run with -e, so a failing test command on its own line would end the script before the upload. This form keeps going and records the code. The script needs bash for set -o pipefail: on Jenkins, whose sh steps often run dash, start it with #!/bin/bash.
• --test-exit-code "$exit_code": Passes the test runner's exit code so Mergify can detect silent failures where the runner crashed but the JUnit report appears clean.
• junit-process exits last, so its status becomes the step's. Quarantine depends on that: it exits 0 when every failure it found is quarantined, and 1 when one is not. Re-exiting with the test runner's own code would fail the step whatever quarantine decided.
• The branch quarantine is evaluated against is detected from your CI provider, falling back to the checked-out branch. Pass --tests-target-branch <branch> only to override it.

CircleCI runs each step with bash -eo pipefail: keep the test command and the upload in the same run step, as above, so the upload happens even when tests fail. On a fork build, junit-process has no token and fails the step: skip it when MERGIFY_TOKEN is empty, for example if [ -n "${MERGIFY_TOKEN:-}" ]; then mergify ci junit-process …; else exit "$exit_code"; fi.