RSpec Integration with Mergify
Report your test results from RSpec to Mergify
This guide explains how to integrate RSpec with Test Engine using the
rspec-mergify gem. Once installed, test results are automatically uploaded to
Test Engine without any extra workflow changes.
Requirements
Section titled Requirements-
CRuby 3.1 to 4.0. JRuby, TruffleRuby, development builds of Ruby (
ruby-head) and jobs that install gems with Bundler’sforce_ruby_platformare not supported: the suite runs, but nothing reaches Test Engine, and the plugin says why after the run. Use a JUnit report upload for those jobs. -
Ruby 3.1 or later, and
rspec-core3.12 or later. When some jobs of your CI matrix run an older Ruby, add the gem only for Ruby 3.1 and later (for examplegem 'rspec-mergify' if RUBY_VERSION >= '3.1'), require it fromspec/spec_helper.rbwith the same condition rather than from.rspec, and check that a frozenbundle installstill passes on the older Ruby. -
A platform the
mergify-ci-coregem ships a precompiled extension for. Bundler installs that gem alongsiderspec-mergify, and its platform gems cover Linux x86_64 and aarch64 (glibc 2.28 or later: Debian 10+, Ubuntu 20.04+, RHEL 8+ and its rebuilds, Amazon Linux 2023; or musl, such as Alpine), macOS arm64 and x86_64, and Windows x64 (UCRT). Anywhere else, including glibc older than 2.28 (Amazon Linux 2, CentOS 7, Ubuntu 18.04), the extension cannot load: the gem still installs, but nothing reaches Test Engine. On those systems, use a JUnit report upload, or pin the last pure-Ruby release (gem 'rspec-mergify', '0.1.4'), which needsMERGIFY_TOKEN: it cannot exchange a GitHub Actions job’s ID token.
Installation
Section titled InstallationYou need to install the
rspec-mergify gem to automatically
upload your test results to Test Engine.
Gemfile
Section titled GemfileAdd the gem to your Gemfile:
group :test do gem 'rspec-mergify'endThen run:
bundle installUse the Bundler version your Gemfile.lock was written with (its
BUNDLED WITH line), or Bundler rewrites that line.
Load the gem
Section titled Load the gemBundler does not load rspec-mergify by itself. The gem’s entry file is
rspec_mergify, which Bundler.require does not look for, and a plain
bundle exec rspec does not call Bundler.require at all. Require it in your
.rspec file:
--require rspec_mergifyOr add require 'rspec_mergify' at the top of spec/spec_helper.rb. Without
it, RSpec runs as usual and nothing reaches Test Engine.
gem install
Section titled gem installAlternatively, install it directly:
gem install rspec-mergifyRunning in CI
Section titled Running in CIWhen the plugin reports
Section titled When the plugin reportsThe plugin uploads results only when the CI environment variable is set, or
when MERGIFY_CI_ENABLE is on. GitHub Actions, CircleCI and Buildkite set CI
for you. Jenkins does not: set CI=true, or MERGIFY_CI_ENABLE=true, in
the stage that runs your tests. If your tests behave differently when CI is
set (some test suites skip or add tests on CI), set MERGIFY_CI_ENABLE=true
instead of CI, so the tests that run stay the same.
CI counts as set unless it is empty or a false value (false, no, off,
0, f or n), so true, 1 and a CI system’s name (Woodpecker sets CI
to woodpecker) all count. MERGIFY_CI_ENABLE is stricter: only true,
yes, on, 1, t or y turn it on. Case does not matter for either.
Without a credential, the plugin uploads nothing and your tests run as usual, so pull requests from forks, which get neither an ID token nor secrets, report nothing and need no guard in the workflow.
Job names
Section titled Job namesEvery test is identified by its name and the name of the CI job that ran it. By default the job name is the job’s identifier in your CI:
| CI | Default job name |
|---|---|
| GitHub Actions | the job id (GITHUB_JOB), recorded with the workflow name (GITHUB_WORKFLOW) |
| CircleCI | CIRCLE_JOB, the expanded job name of each matrix leg |
| Buildkite | the step label (BUILDKITE_LABEL) |
| Jenkins | JOB_NAME, also used as the pipeline name, except in a multibranch pipeline (below) |
Set MERGIFY_TEST_JOB_NAME when that default is not one stable name per set of
tests:
-
GitHub Actions matrix: every leg shares one job id. Set it per leg from the matrix values, for example
MERGIFY_TEST_JOB_NAME: test (${{ matrix.python-version }}, ${{ matrix.os }}). -
Two test runs in one job: set a distinct name on each step.
-
Reusable workflows: a called workflow reports under the caller’s workflow name, so two called workflows whose jobs share an id (often
test) collide. Set a distinct name in each. -
Shards of one test suite are the same tests split across machines: give all shards the same name. Never put a shard index in it.
In a Jenkins multibranch pipeline, every branch and every pull request of the
project reports its tests under the same pipeline name and job name: the
multibranch project (my-repo, not my-repo/PR-123). MERGIFY_TEST_JOB_NAME
replaces the job name but never the pipeline name, so it cannot file the tests
of two Jenkins pipelines together. Inside a container, pass BRANCH_NAME
through as listed below, or the tests are filed under JOB_NAME.
Tests inside tox, Docker or Docker Compose
Section titled Tests inside tox, Docker or Docker Composetox, docker run and Docker Compose start the tests in an environment that
does not inherit the CI job’s variables. Pass through CI,
MERGIFY_TEST_JOB_NAME if you set it, the credential, and every variable the
plugin reads to identify the run. On GitHub Actions the credential is
ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN, which
GitHub sets in a job granted id-token: write; with a key, it is
MERGIFY_TOKEN. tox’s pass_env accepts globs (GITHUB_*); Docker
Compose, docker run -e and the Buildkite Docker Compose plugin need each name:
GitHub Actions: GITHUB_ACTIONS GITHUB_BASE_REF GITHUB_EVENT_NAME GITHUB_EVENT_PATH GITHUB_HEAD_REF GITHUB_JOB GITHUB_REF_NAME GITHUB_REF_TYPE GITHUB_REPOSITORY GITHUB_REPOSITORY_ID GITHUB_RUN_ATTEMPT GITHUB_RUN_ID GITHUB_SERVER_URL GITHUB_SHA GITHUB_WORKFLOW RUNNER_NAMECircleCI: CIRCLECI CIRCLE_BRANCH CIRCLE_BUILD_URL CIRCLE_JOB CIRCLE_REPOSITORY_URL CIRCLE_SHA1 CIRCLE_WORKFLOW_IDBuildkite: BUILDKITE BUILDKITE_AGENT_NAME BUILDKITE_BRANCH BUILDKITE_BUILD_ID BUILDKITE_BUILD_URL BUILDKITE_COMMIT BUILDKITE_LABEL BUILDKITE_PIPELINE_SLUG BUILDKITE_PULL_REQUEST BUILDKITE_PULL_REQUEST_BASE_BRANCH BUILDKITE_REPO BUILDKITE_RETRY_COUNT BUILDKITE_STEP_KEYJenkins: JENKINS_URL BUILD_ID BUILD_URL JOB_NAME BRANCH_NAME NODE_NAME GIT_URL GIT_BRANCH GIT_COMMIT CHANGE_ID CHANGE_TARGET CHANGE_BRANCH CHANGE_FORK TAG_NAMEPass them by name, never by writing a value into a file. In a container on
GitHub Actions, also mount the file GITHUB_EVENT_PATH points to, at the same
path: without it the run is recorded against the merge commit instead of the
pull request’s head commit.
Workflow linters
Section titled Workflow lintersWith an application key, zizmor can report
secrets-outside-env on the MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} line,
depending on its version and persona. Run the repository’s own zizmor
configuration on the changed workflow, and if it reports the finding, follow the
convention the repository already uses for it: a deployment environment on the
job, or an inline # zizmor: ignore[secrets-outside-env] comment.
Update your CI workflow
Section titled Update your CI workflowGitHub Actions
Section titled GitHub ActionsThe step that runs your tests needs nothing Mergify-specific:
- name: Run RSpec Tests 🧪 run: bundle exec rspecLet your CI reach Mergify Preview
Section titled Let your CI reach Mergify
Grant id-token: write to the job, or to the whole workflow. The job then trades its
GitHub-signed ID token for a Mergify token, and no MERGIFY_TOKEN is stored:
permissions: id-token: write # lets the job ask GitHub for an ID token contents: read # actions/checkout needs it
A permissions block sets every permission it does not list to none: list
the ones your other steps need too.
Mergify refuses the ID token until a trust policy of your organization matches the job. A
GitHub organization Owner or a CI Admin writes one under
Settings → Security & Access → CI Authentication → Trust Policies on the
dashboard. OIDC Authentication
covers trust policies, pull requests from forks and refused exchanges. If your workflows store
MERGIFY_TOKEN today,
move them off it.
Use an application key instead
Section titled Use an application key instead
Other CIs authenticate with a ci
application key, and so do the GitHub Actions jobs OIDC
leaves out: runs for pull requests from forks when your organization sends them secrets, and jobs
that send scopes from the base branch (pull_request_target,
workflow_run, issue_comment). Store the key as the
MERGIFY_TOKEN repository secret and set it on the job:
env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
A MERGIFY_TOKEN that is set wins over the ID token. For pull requests from forks,
set it on those runs only and drop the fork condition from the upload step, as
Jobs that keep a key shows. Store the key in
your
Dependabot secrets too, since runs for Dependabot pull requests read only those.
Buildkite
Section titled BuildkiteSet MERGIFY_TOKEN to a ci application key in the
environment of the agents running your tests.
The step itself then needs no Mergify-specific configuration:
steps: - label: "Run RSpec Tests 🧪" command: bundle exec rspecThe gem automatically collects your test results and sends them to Test Engine.
Quarantine and crashed runs
Section titled Quarantine and crashed runsThe gem applies quarantine inside the RSpec run.
When a quarantined example fails, the gem marks it pending with the message
Test is quarantined from Mergify Test Insights, so it does not count as a
failure and does not change RSpec’s exit code. The exit code of your test step
already accounts for quarantine, so the step needs nothing more:
-
Do not add
continue-on-error: true. The recipes that upload a JUnit report with themergifyio/gha-mergify-ciaction need it because the action decides the job’s result after the tests. Here nothing does, so it would let every real failure through. -
There is no step
idto set and notest_step_outcometo pass: both belong to that action, which this setup does not use.
A crash cannot pass for a green run either. Quarantine only covers failures
inside an example: a spec file that fails to load, a failing before(:suite)
or after(:context) hook, or RSpec dying mid-run still exits non-zero and fails
the step. The gem uploads results when the run ends, so a process killed
outright before then, such as by the out-of-memory killer, sends nothing to
Test Engine.
If the gem cannot fetch the quarantine list, it quarantines nothing for that run, and a quarantined example that fails makes the step fail as usual.
Flaky detection
Section titled Flaky detectionThe gem reruns examples within the budgets set under Native Flaky Detection:
-
On a pull request, an example the target branch does not have yet is new. The gem runs it again right after its own run, inside the same RSpec run, as many times as fit in the Prevention Budget. When one of those runs fails, the example is flaky and RSpec reports it failed, so the pull request does not merge. The tries a retry plugin such as
rspec-retrymade inside the example’s first run count too: a failure among them makes the example flaky. See Prevention. -
On a push, the gem reruns the examples Test Engine knows as unhealthy, within the Detection Budget, to report whether they still are. The run’s result does not change. See Rerunning unhealthy tests.
Matrix runs
Section titled Matrix runsThe gem uploads each run’s results under your CI job’s name. On GitHub
Actions, that name is the job’s identifier (GITHUB_JOB), which every leg of
a matrix shares. The job name is
part of a test’s identity
in Test Engine, so when the legs run the same suite on several Ruby versions,
their results merge into one test.
Set MERGIFY_TEST_JOB_NAME on each leg and include the matrix value, so each
leg reports under its own name:
jobs: test: runs-on: ubuntu-latest permissions: id-token: write contents: read strategy: matrix: ruby-version: ['3.2', '3.3', '3.4'] steps: # ... - name: Run RSpec Tests 🧪 env: MERGIFY_TEST_JOB_NAME: test (ruby ${{ matrix.ruby-version }}) run: bundle exec rspecName each leg after the environment it runs in, never after a shard of the suite: a test can move to another shard when tests are added or removed, so a name per shard would split that test’s history.
Setting MERGIFY_TEST_JOB_NAME on a job that already reports changes its
name, so its tests start a new history in Test Engine.
Jobs the plugin cannot report from
Section titled Jobs the plugin cannot report fromOn a job the plugin does not support (see Requirements),
write a JUnit report with the rspec_junit_formatter gem and upload it as
described in Other frameworks.
That gem installs and runs on JRuby too. Add it to your Gemfile:
group :test do gem 'rspec_junit_formatter'endThen run RSpec with the formatter and the file to write:
bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xmlThe report to upload is rspec.xml. Keep --format progress (or the
formatter you already use): once a run names a formatter, RSpec drops its
default one, so with the JUnit formatter alone it prints nothing to the
console.
The report names an example from its spec file and its full description, like
spec.models.user_spec.User when banned cannot log in. The plugin names the
same example spec/models/user_spec.rb > User > when banned > cannot log in,
so Test Engine shows them as two tests. An example’s history starts again
when its job switches between a JUnit report and the plugin.
Its quarantine carries over, in that direction: the gem asks Mergify about
each example under the name a rspec_junit_formatter report would have given
it as well, so an example quarantined while the job uploaded a report is still
quarantined once the gem reports the suite. Going the other way, from the gem
back to a JUnit report upload, the quarantine has to be set again under the
report’s name.
Verify and review in Test Engine
Section titled Verify and review in Test EngineAfter pushing these changes, your next CI run reports its RSpec results automatically.
You can then review your test results, including any failures or flaky tests, directly in the Test Engine dashboard.
Environment variables
Section titled Environment variables| Variable | Purpose |
|---|---|
MERGIFY_API_URL |
Base URL of the Mergify API. An API URL given on the command line or in a reporter's configuration takes precedence. |
MERGIFY_CI_DEBUG |
Set to |
MERGIFY_CI_ENABLE |
Set to |
MERGIFY_FAIL_FIRST_ENABLE |
Set to |
MERGIFY_TEST_JOB_NAME |
Name the job's test results are reported under, instead of the CI job's own name. Set one per environment, never per shard: a name per shard splits a test's history. |
MERGIFY_TEST_RETRY_ENABLE |
Set to |
MERGIFY_TOKEN |
The Mergify credential: an application key in CI. Without it the test reporters upload nothing. A |
MERGIFY_TRACEPARENT |
W3C |
MERGIFY_TEST_SELECTION_ENABLE |
Deprecated name of |
RSPEC_MERGIFY_DEBUG |
Deprecated name of |
RSPEC_MERGIFY_ENABLE |
Deprecated name of |
To check the setup without uploading anything, run your tests with
MERGIFY_CI_ENABLE=true MERGIFY_CI_DEBUG=true: the gem prints its
Mergify CI: section.
Was this page helpful?
Thanks for your feedback!