Pytest Integration with Mergify

This guide explains how to integrate Pytest with Test Engine using the pytest-mergify plugin. Once installed, test results are automatically uploaded to Test Engine without any extra workflow changes.

• CPython only: Python 3.10 or later from release 2026.10.6, while 2026.10.5 still supports 3.8 and 3.9. pytest-mergify ships compiled wheels and no source distribution: Linux (x86_64 and aarch64, glibc and musl), macOS (x86_64 and arm64) and Windows (x86_64). There is no wheel for PyPy or for free-threaded CPython builds (3.13t, 3.14t), and pip then falls back to an old release that has to compile Rust.

• pytest 7.0 or later.

When some jobs of your CI matrix run PyPy, keep the plugin out of them with an environment marker on the requirement:

pytest-mergify; platform_python_implementation == "CPython"

No marker tells a free-threaded build apart: skip installing the plugin in those jobs from the workflow instead.

You need to install the pytest-mergify plugin to automatically upload your test results to Test Engine. This can be done in different ways depending on your Python dependency setup. Below are a few examples:

pip install pytest-mergify

Or add pytest-mergify to your requirements.txt.

setup(
    name="your-package",
    ...
    install_requires=[
        ...
    ],
    extras_require={
        "dev": ["pytest-mergify"]
    },
    ...
)

Make sure those dependencies are installed when running your tests.

[options.extras_require]
dev =
    pytest-mergify

Make sure those dependencies are installed when running your tests.

poetry add --group dev pytest-mergify

uv add --dev pytest-mergify

Use --group <name> instead of --dev when your tests install from another dependency group. If your pyproject.toml sets add-bounds = "major", add --bounds lower: pytest-mergify uses calendar versions, and a major bound would block next year’s releases. Run the uv version your project already uses: an older uv rewrites the revision line at the top of uv.lock.

The plugin uploads results only when the CI environment variable is set, or when MERGIFY_CI_ENABLE is on. GitHub Actions, CircleCI and Buildkite set CI for you. Jenkins does not: set CI=true, or MERGIFY_CI_ENABLE=true, in the stage that runs your tests. If your tests behave differently when CI is set (some test suites skip or add tests on CI), set MERGIFY_CI_ENABLE=true instead of CI, so the tests that run stay the same.

CI counts as set unless it is empty or a false value (false, no, off, 0, f or n), so true, 1 and a CI system’s name (Woodpecker sets CI to woodpecker) all count. MERGIFY_CI_ENABLE is stricter: only true, yes, on, 1, t or y turn it on. Case does not matter for either.

Without MERGIFY_TOKEN the plugin uploads nothing and your tests run as usual. Pull requests from forks and Dependabot pull requests on GitHub Actions get no repository secrets, so they report nothing: no guard is needed in the workflow. (The RSpec gem logs an authentication error when the variable is set but empty, as on those pull requests; the tests are not affected.) To have Dependabot pull requests report too, store the key as a Dependabot secret as well (gh secret set MERGIFY_TOKEN --app dependabot --repo <owner>/<repository>).

Every test is identified by its name and the name of the CI job that ran it. By default the job name is the job’s identifier in your CI:

Set MERGIFY_TEST_JOB_NAME when that default is not one stable name per set of tests:

• GitHub Actions matrix: every leg shares one job id. Set it per leg from the matrix values, for example MERGIFY_TEST_JOB_NAME: test (${{ matrix.python-version }}, ${{ matrix.os }}).

• Two test runs in one job: set a distinct name on each step.

• Reusable workflows: a called workflow reports under the caller’s workflow name, so two called workflows whose jobs share an id (often test) collide. Set a distinct name in each.

• Jenkins multibranch pipelines: each branch and each pull request keeps its own test history, and no setting changes that. JOB_NAME includes the branch or pull request (my-repo/PR-123) and is also the pipeline name tests are filed under; MERGIFY_TEST_JOB_NAME replaces only the job name. Do not set it for this purpose.

• Shards of one test suite are the same tests split across machines: give all shards the same name. Never put a shard index in it.

tox, docker run and Docker Compose start the tests in an environment that does not inherit the CI job’s variables. Pass through MERGIFY_TOKEN, CI, MERGIFY_TEST_JOB_NAME if you set it, and every variable the plugin reads to identify the run. tox’s pass_env accepts globs (GITHUB_*); Docker Compose, docker run -e and the Buildkite Docker Compose plugin need each name:

GitHub Actions: GITHUB_ACTIONS GITHUB_BASE_REF GITHUB_EVENT_NAME GITHUB_EVENT_PATH
  GITHUB_HEAD_REF GITHUB_JOB GITHUB_REF_NAME GITHUB_REF_TYPE GITHUB_REPOSITORY
  GITHUB_REPOSITORY_ID GITHUB_RUN_ATTEMPT GITHUB_RUN_ID GITHUB_SERVER_URL
  GITHUB_SHA GITHUB_WORKFLOW RUNNER_NAME
CircleCI: CIRCLECI CIRCLE_BRANCH CIRCLE_BUILD_URL CIRCLE_JOB
  CIRCLE_REPOSITORY_URL CIRCLE_SHA1 CIRCLE_WORKFLOW_ID
Buildkite: BUILDKITE BUILDKITE_AGENT_NAME BUILDKITE_BRANCH BUILDKITE_BUILD_ID
  BUILDKITE_BUILD_URL BUILDKITE_COMMIT BUILDKITE_LABEL BUILDKITE_PIPELINE_SLUG
  BUILDKITE_PULL_REQUEST BUILDKITE_PULL_REQUEST_BASE_BRANCH BUILDKITE_REPO
  BUILDKITE_RETRY_COUNT BUILDKITE_STEP_KEY
Jenkins: JENKINS_URL BUILD_ID BUILD_URL JOB_NAME NODE_NAME GIT_URL GIT_BRANCH
  GIT_COMMIT CHANGE_TARGET

Pass them by name, never by writing a value into a file. In a container on GitHub Actions, also mount the file GITHUB_EVENT_PATH points to, at the same path: without it the run is recorded against the merge commit instead of the pull request’s head commit.

Depending on its version and persona, zizmor can report secrets-outside-env on the MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} line. Run the repository’s own zizmor configuration on the changed workflow, and if it reports the finding, follow the convention the repository already uses for it: a deployment environment on the job, or an inline # zizmor: ignore[secrets-outside-env] comment.

Your workflow should run your tests as usual while exporting the secret MERGIFY_TOKEN as an environment variable.

Add the following to the GitHub Actions step running your tests:

env:
  MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}

For example:

- name: Run Tests 🧪
  env:
    MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
  run: pytest

Set MERGIFY_TOKEN in the environment of the agents running your tests. The step itself then needs no Mergify-specific configuration:

steps:
  - label: "Run Tests 🧪"
    command: pytest

The plugin collects your test results and sends them to Test Engine.

The plugin applies quarantine inside the pytest session. It marks each quarantined test as a non-strict xfail, so a quarantined test that fails is reported as xfailed and does not count against pytest’s exit code. The exit code of your test step already accounts for quarantine, so the step needs nothing more:

• Do not add continue-on-error: true. The recipes that upload a JUnit report with the mergifyio/gha-mergify-ci action need it because the action decides the job’s result after the tests. Here nothing does, so it would let every real failure through.

• There is no step id to set and no test_step_outcome to pass: both belong to that action, which this setup does not use.

A crash cannot pass for a green run either. If pytest dies mid-run or fails outside a test, such as a collection error, it exits non-zero and the step fails. The plugin uploads results when the session ends, so a process killed outright before then, such as by the out-of-memory killer, sends nothing to Test Engine. A run that stops early without the process being killed, such as an interrupted session or one cut short by --maxfail, still uploads the results collected up to that point.

If the plugin cannot fetch the quarantine list, it quarantines nothing for that run, and a quarantined test that fails makes the step fail as usual.

The plugin reruns tests within the budgets set under Native Flaky Detection:

• On a pull request, a test the target branch does not have yet is new. The plugin runs it again right after its own run, as many times as fit in the Prevention Budget. It holds back the fixture finalizers pytest would run between those runs, so a higher-scoped fixture is not torn down mid-rerun. When one of the runs fails, the test is flaky and pytest reports it failed, so the pull request does not merge. See Prevention.

• On a push, the plugin reruns the tests Test Engine knows as unhealthy, within the Detection Budget, to report whether they still are. The run’s result does not change. See Rerunning unhealthy tests.

To check the setup without uploading anything, run your tests with MERGIFY_CI_ENABLE=true MERGIFY_CI_DEBUG=true: the plugin prints its Mergify CI section and the spans it would send.

tox runs the tests in an environment that does not inherit the CI job’s variables. Add the plugin to the test environment’s dependencies and pass through what it needs:

[testenv]
deps =
    pytest
    pytest-mergify
pass_env =
    CI
    MERGIFY_*
    GITHUB_*
    RUNNER_*
commands = pytest

On another CI, replace GITHUB_* and RUNNER_* with that CI’s variables (see Tests inside tox, Docker or Docker Compose). tox does not pass CI by default.

In your CI workflow, give the token to the step that runs tox:

# GitHub Actions
- name: Run Tox Tests
  env:
    MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
    MERGIFY_TEST_JOB_NAME: tox (${{ matrix.python-version }})
  run: tox

On Buildkite the token comes from the agent’s environment, as above.

When one tox call runs several environments (for example py312-crypto and py312-nocrypto), each runs pytest with the same job name. Add the environment name to it, so their reports stay apart:

[testenv]
set_env =
    MERGIFY_TEST_JOB_NAME = {env:MERGIFY_TEST_JOB_NAME:tox} ({envname})

After pushing these changes, your next CI run reports its pytest results automatically.

You can then review your test results, including any failures or flaky tests, directly in the Test Engine dashboard.