View as Markdown

Minitest Integration with Mergify

Report your test results from Minitest to Mergify


This guide explains how to integrate Minitest with Test Engine using the minitest-mergify gem, a Minitest plugin. The plugin uploads your test results to Test Engine when the run ends, with no JUnit file to write or upload, and applies quarantine and flaky detection inside the Minitest run. Rails test suites run on Minitest, so this page covers them too.

If your CI uploads a JUnit report from minitest-junit or minitest-reporters today, see Moving from a JUnit upload.

  • CRuby 3.1 to 4.0. JRuby, TruffleRuby, development builds of Ruby (ruby-head) and jobs that install gems with Bundler’s force_ruby_platform are not supported: the suite runs, but nothing reaches Test Engine, and the plugin says why after the run. Use a JUnit report upload for those jobs.

  • Ruby 3.1 or later with Minitest 5.19 or later, or Ruby 3.2 or later with Minitest 6.

  • Linux on x86_64 or aarch64 (glibc 2.28 or newer, or musl), macOS, or Windows on x86_64. The gem runs on mergify-ci-core, precompiled for those platforms, which Bundler installs alongside it at the same version. On any other platform your suite still runs, but nothing reaches Test Engine and the plugin says why after Minitest’s summary: use a JUnit report upload there too.

  • If your gems come from a mirror that only serves an allowlist, allow mergify-ci-core as well as minitest-mergify.

The plugin reports tests written in classes and in describe blocks, whether Minitest runs them in one process or in parallel, in its own threads (parallelize_me!) or in Rails’ parallel workers (parallelize).

Add the minitest-mergify gem to the test group of your Gemfile:

ruby
group :test do
gem 'minitest-mergify'
end

Then run:

Terminal windowbash
bundle install

Use the Bundler version your Gemfile.lock was written with (its BUNDLED WITH line), or Bundler rewrites that line.

Require the plugin where your suite sets Minitest up, before its tests are defined, such as in test/test_helper.rb:

test/test_helper.rbruby
require 'minitest/mergify'

Minitest 6 loads only the plugins a suite asks for, so without this line nothing reaches Test Engine. Minitest 5 loads the plugin by itself, but only once the tests are defined: keep the line there too, or a spec test with no body (it "does a thing" without a block) is reported without its line number, under the file of another test in the same describe.

The plugin works alongside the reporters you already use, minitest-reporters included. It prints what it did after Minitest’s summary, under # Mergify Test Engine:.

The plugin uploads results only when the CI environment variable is set, or when MERGIFY_CI_ENABLE is on. GitHub Actions, CircleCI and Buildkite set CI for you. Jenkins does not: set CI=true, or MERGIFY_CI_ENABLE=true, in the stage that runs your tests. If your tests behave differently when CI is set (some test suites skip or add tests on CI), set MERGIFY_CI_ENABLE=true instead of CI, so the tests that run stay the same.

CI counts as set unless it is empty or a false value (false, no, off, 0, f or n), so true, 1 and a CI system’s name (Woodpecker sets CI to woodpecker) all count. MERGIFY_CI_ENABLE is stricter: only true, yes, on, 1, t or y turn it on. Case does not matter for either.

Without a credential, the plugin uploads nothing and your tests run as usual, so pull requests from forks, which get neither an ID token nor secrets, report nothing and need no guard in the workflow.

Every test is identified by its name and the name of the CI job that ran it. By default the job name is the job’s identifier in your CI:

CI Default job name
GitHub Actions the job id (GITHUB_JOB), recorded with the workflow name (GITHUB_WORKFLOW)
CircleCI CIRCLE_JOB, the expanded job name of each matrix leg
Buildkite the step label (BUILDKITE_LABEL)
Jenkins JOB_NAME, also used as the pipeline name, except in a multibranch pipeline (below)

Set MERGIFY_TEST_JOB_NAME when that default is not one stable name per set of tests:

  • GitHub Actions matrix: every leg shares one job id. Set it per leg from the matrix values, for example MERGIFY_TEST_JOB_NAME: test (${{ matrix.python-version }}, ${{ matrix.os }}).

  • Two test runs in one job: set a distinct name on each step.

  • Reusable workflows: a called workflow reports under the caller’s workflow name, so two called workflows whose jobs share an id (often test) collide. Set a distinct name in each.

  • Shards of one test suite are the same tests split across machines: give all shards the same name. Never put a shard index in it.

In a Jenkins multibranch pipeline, every branch and every pull request of the project reports its tests under the same pipeline name and job name: the multibranch project (my-repo, not my-repo/PR-123). MERGIFY_TEST_JOB_NAME replaces the job name but never the pipeline name, so it cannot file the tests of two Jenkins pipelines together. Inside a container, pass BRANCH_NAME through as listed below, or the tests are filed under JOB_NAME.

Tests inside tox, Docker or Docker Compose

Section titled Tests inside tox, Docker or Docker Compose

tox, docker run and Docker Compose start the tests in an environment that does not inherit the CI job’s variables. Pass through CI, MERGIFY_TEST_JOB_NAME if you set it, the credential, and every variable the plugin reads to identify the run. On GitHub Actions the credential is ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN, which GitHub sets in a job granted id-token: write; with a key, it is MERGIFY_TOKEN. tox’s pass_env accepts globs (GITHUB_*); Docker Compose, docker run -e and the Buildkite Docker Compose plugin need each name:

GitHub Actions: GITHUB_ACTIONS GITHUB_BASE_REF GITHUB_EVENT_NAME GITHUB_EVENT_PATH
GITHUB_HEAD_REF GITHUB_JOB GITHUB_REF_NAME GITHUB_REF_TYPE GITHUB_REPOSITORY
GITHUB_REPOSITORY_ID GITHUB_RUN_ATTEMPT GITHUB_RUN_ID GITHUB_SERVER_URL
GITHUB_SHA GITHUB_WORKFLOW RUNNER_NAME
CircleCI: CIRCLECI CIRCLE_BRANCH CIRCLE_BUILD_URL CIRCLE_JOB
CIRCLE_REPOSITORY_URL CIRCLE_SHA1 CIRCLE_WORKFLOW_ID
Buildkite: BUILDKITE BUILDKITE_AGENT_NAME BUILDKITE_BRANCH BUILDKITE_BUILD_ID
BUILDKITE_BUILD_URL BUILDKITE_COMMIT BUILDKITE_LABEL BUILDKITE_PIPELINE_SLUG
BUILDKITE_PULL_REQUEST BUILDKITE_PULL_REQUEST_BASE_BRANCH BUILDKITE_REPO
BUILDKITE_RETRY_COUNT BUILDKITE_STEP_KEY
Jenkins: JENKINS_URL BUILD_ID BUILD_URL JOB_NAME BRANCH_NAME NODE_NAME GIT_URL
GIT_BRANCH GIT_COMMIT CHANGE_ID CHANGE_TARGET CHANGE_BRANCH CHANGE_FORK
TAG_NAME

Pass them by name, never by writing a value into a file. In a container on GitHub Actions, also mount the file GITHUB_EVENT_PATH points to, at the same path: without it the run is recorded against the merge commit instead of the pull request’s head commit.

With an application key, zizmor can report secrets-outside-env on the MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} line, depending on its version and persona. Run the repository’s own zizmor configuration on the changed workflow, and if it reports the finding, follow the convention the repository already uses for it: a deployment environment on the job, or an inline # zizmor: ignore[secrets-outside-env] comment.

The step that runs your tests needs nothing Mergify-specific:

yaml
- name: Run Minitest tests
run: bundle exec rake test

In a Rails application, keep the command you use today, such as bin/rails test.

Let your CI reach Mergify Preview

Section titled Let your CI reach Mergify

Grant id-token: write to the job, or to the whole workflow. The job then trades its GitHub-signed ID token for a Mergify token, and no MERGIFY_TOKEN is stored:

yaml
permissions:
id-token: write # lets the job ask GitHub for an ID token
contents: read # actions/checkout needs it

A permissions block sets every permission it does not list to none: list the ones your other steps need too.

Mergify refuses the ID token until a trust policy of your organization matches the job. A GitHub organization Owner or a CI Admin writes one under Settings → Security & Access → CI Authentication → Trust Policies on the dashboard. OIDC Authentication covers trust policies, pull requests from forks and refused exchanges. If your workflows store MERGIFY_TOKEN today, move them off it.

Other CIs authenticate with a ci application key, and so do the GitHub Actions jobs OIDC leaves out: runs for pull requests from forks when your organization sends them secrets, and jobs that send scopes from the base branch (pull_request_target, workflow_run, issue_comment). Store the key as the MERGIFY_TOKEN repository secret and set it on the job:

yaml
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}

A MERGIFY_TOKEN that is set wins over the ID token. For pull requests from forks, set it on those runs only and drop the fork condition from the upload step, as Jobs that keep a key shows. Store the key in your Dependabot secrets too, since runs for Dependabot pull requests read only those.

Set MERGIFY_TOKEN to a ci application key in the environment of the agents running your tests. The step itself then needs no Mergify-specific configuration:

yaml
steps:
- label: "Run Minitest tests"
command: bundle exec rake test

The plugin collects your test results and sends them to Test Engine.

Before any test runs, the plugin fetches the list of tests quarantined in Test Engine. A quarantined test still runs and reaches Test Engine with its real result, a failure included, but that failure does not fail the run: Minitest counts it as a skip, and so do your other reporters. Minitest’s verbose output (--verbose) gives the skip’s reason, Test is quarantined from Mergify Test Insights. This works the same whether your tests run in one process or in parallel.

A test quarantined while your repository uploaded JUnit reports stays quarantined: the plugin also matches each test under the name minitest-junit and minitest-reporters gave it, such as UserTest.test_valid.

The exit code of your test step already accounts for quarantine, so the step needs nothing more:

  • Do not add continue-on-error: true. The recipes that upload a JUnit report with the mergifyio/gha-mergify-ci action need it because the action decides the job’s result after the tests. Here nothing does, so it would let every real failure through.

  • There is no step id to set and no test_step_outcome to pass: both belong to that action, which this setup does not use.

A crash cannot pass for a green run either. Quarantine only covers the failure of a test: a test file that fails to load, or Minitest dying mid-run, still exits non-zero and fails the step. The plugin uploads results when the run ends, so a process killed outright before then, such as by the out-of-memory killer, sends nothing to Test Engine.

If the plugin cannot fetch the quarantine list, it quarantines nothing for that run, and a quarantined test that fails makes the step fail as usual.

The plugin reruns tests within the budgets set under Native Flaky Detection:

  • On a pull request, a test the target branch does not have yet is new. Once the suite is over, the plugin reruns the new tests within the Prevention Budget. If one of those runs fails, the test is flaky and the job fails, so the pull request does not merge. See Prevention.

  • On a branch with no target, such as a push to your default branch, the plugin reruns the tests Test Engine knows as unhealthy, within the Detection Budget, to report whether they still are. The job’s result does not change. See Rerunning unhealthy tests.

The reruns go through Minitest itself, so a test that ran in Minitest’s threads or in Rails’ parallel workers is rerun there. Minitest’s summary and your other reporters show only your run, so a new test the reruns found flaky still counts as passed there. The exit code and the # Mergify Test Engine: section say otherwise.

A test in a class is named as Minitest names it:

UserTest#test_valid

A test in a describe block is named by its file (relative to the directory Minitest runs in), the describe blocks around it and its text:

test/models/user_test.rb > User > when banned > cannot log in

Minitest numbers each it by its position in its describe (test_0001_cannot log in), so adding a test would rename every one below it. The name leaves that number out, except when two tests in the same describe share their text: those keep Minitest’s number, so they stay apart.

Test Engine tracks a test’s history under its name. For a test in a class, renaming the class or the method starts a new test. For a test in a describe block, renaming its file, a describe block or its text does. A test that reached Test Engine through a JUnit upload, as UserTest.test_valid, starts its history again under its new name, while its quarantine entry still applies.

The plugin uploads each run’s results under your CI job’s name. On GitHub Actions, that name is the job’s identifier (GITHUB_JOB, recorded with the workflow name), which every leg of a matrix shares. The job name is part of a test’s identity in Test Engine, so when the legs run the same suite on several Ruby versions, their results merge into one test.

Set MERGIFY_TEST_JOB_NAME on each leg and include the matrix value, so each leg reports under its own name:

yaml
jobs:
test:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
strategy:
matrix:
ruby-version: ['3.3', '3.4']
steps:
# ...
- name: Run Minitest tests
env:
MERGIFY_TEST_JOB_NAME: test (ruby ${{ matrix.ruby-version }})
run: bundle exec rake test

Name each leg after the environment it runs in, never after a shard of the suite: a test can move to another shard when tests are added or removed, so a name per shard would split that test’s history.

Setting MERGIFY_TEST_JOB_NAME on a job that already reports changes its name, so its tests start a new history in Test Engine.

Jobs the plugin cannot report from

Section titled Jobs the plugin cannot report from

On a job the plugin does not support (see Requirements), write a JUnit report with the minitest-reporters gem and upload it as described in Other frameworks. Add its JUnit reporter to your test helper:

test/test_helper.rbruby
require 'minitest/reporters'
Minitest::Reporters.use! [
Minitest::Reporters::DefaultReporter.new,
Minitest::Reporters::JUnitReporter.new,
]

The reporter writes one file per test class, named TEST-<class>.xml, into test/reports, so the report to upload is test/reports/*.xml. To write elsewhere, pass the reporter a directory, not a file name: JUnitReporter.new('junit.xml') creates a directory named junit.xml.

The minitest-junit gem writes a JUnit report too, but only with Minitest 5, and it does not install on JRuby. It writes the report only when you pass --junit, to report.xml unless --junit-filename names another file:

Terminal windowbash
bundle exec rake test TESTOPTS="--junit --junit-filename=junit.xml"

Both reports name a test like UserTest.test_valid, not the way the plugin does (see Test names), so its history starts again when its job switches between a JUnit report and the plugin.

If your CI writes a JUnit report with minitest-junit or minitest-reporters and uploads it with the mergifyio/gha-mergify-ci action, the mergifyio/mergify-ci Buildkite plugin or mergify ci junit-process, switch to minitest-mergify:

  1. Stop writing the JUnit report, unless another tool still reads the file:

    • With minitest-junit, remove the gem from your Gemfile and the require 'minitest/junit' line from your test helper. Then remove its options (-rminitest/junit, --junit, --junit-filename) and the JUNIT_OUTPUT_DIR variable wherever you set them: the test command, a Rake task or the test step’s env.

    • With minitest-reporters, remove Minitest::Reporters::JUnitReporter from your Minitest::Reporters.use! list. Keep the gem if you use its other reporters: the plugin works alongside them. Otherwise, also remove the gem, the require 'minitest/reporters' line and the Minitest::Reporters.use! call.

  2. Remove the upload: the mergifyio/gha-mergify-ci step, the mergifyio/mergify-ci plugin on the Buildkite step, or the mergify ci junit-process command. minitest-mergify sends the results itself. If you ran mergify ci junit-process, also drop || exit_code=$? from your test command, so its exit code is the step’s again.

  3. On GitHub Actions, remove continue-on-error: true from the test step. If you added a step for pull requests from forks that fails the job when no report was uploaded, remove it too.

  4. If your upload set a job_name, as a matrix setup does, move its value to MERGIFY_TEST_JOB_NAME on the step that runs the tests, as shown in Matrix runs, so your tests keep the job name they report under.

  5. Install minitest-mergify and require it, as shown in Installation, and grant the job id-token: write, or move the upload’s token to MERGIFY_TOKEN on the job, as shown in Update your CI workflow.

What changes in Test Engine once the plugin uploads:

  • Quarantine entries made from your JUnit uploads still apply: the plugin also matches each test under the name the upload gave it, such as UserTest.test_valid.

  • Your tests get new names, so their history starts again under those names.

Verify and review in Test Engine

Section titled Verify and review in Test Engine

After pushing these changes, your next CI run reports its Minitest results automatically.

You can then review your test results, including any failures or flaky tests, directly in the Test Engine dashboard.

VariablePurpose
MERGIFY_API_URL
defaulthttps://api.mergify.com

Base URL of the Mergify API. An API URL given on the command line or in a reporter's configuration takes precedence.

MERGIFY_CI_DEBUG
defaultfalse

Set to true to upload nothing and print the session verdict and the spans instead of sending them.

MERGIFY_CI_ENABLE
defaultfalse

Set to true to turn the reporter on outside CI. In CI, which the CI variable signals, it is on anyway.

MERGIFY_FAIL_FIRST_ENABLE
defaulttrue

Set to false to stop this job asking Mergify whether it is a merge-queue bisection step, and so never run first the tests it failed in the step above. On by default, and asked only in a build of a merge-queue batch: only a recognised no (false, no, off, 0) turns it off.

MERGIFY_TEST_JOB_NAME

Name the job's test results are reported under, instead of the CI job's own name. Set one per environment, never per shard: a name per shard splits a test's history.

MERGIFY_TEST_RETRY_ENABLE
defaulttrue

Set to false to have this job retry no failing test, whatever the repository's test retry configuration. On by default, and test retry only retries on a push to the default branch of a repository that configured it, or in a run its merge queue checks: only a recognised no (false, no, off, 0) turns it off.

MERGIFY_TOKEN

The Mergify credential: an application key in CI. A GitHub Actions job granted id-token: write needs none: without it, the test reporters, mergify ci junit-process and mergify ci scopes-send exchange the job's ID token for a Mergify token, when a trust policy of the organization allows the job. Without either, the test reporters upload nothing. A mergify command takes it when --token is not given, over the credential mergify auth login stored, and mergify auth tells when it does. The mergify stack commands send it to GitHub as their GitHub token, ahead of GITHUB_TOKEN, unless it holds a Mergify user token (mut_…).

MERGIFY_TRACEPARENT

W3C traceparent of a trace the job already started: the test session's spans join that trace instead of starting their own. A malformed value is ignored.

To check the setup without uploading anything, run your tests with MERGIFY_CI_ENABLE=true MERGIFY_CI_DEBUG=true: the plugin prints the spans it would send.

Was this page helpful?