---
title: Playwright Integration with Mergify
description: Report your test results from Playwright to Mergify
---
This guide explains how to integrate [Playwright](https://playwright.dev/) with
Test Engine using the `@mergifyio/playwright` reporter. Once installed, test
results are automatically uploaded to Test Engine without any extra workflow
changes.
## Installation
Install the
[`@mergifyio/playwright`](https://www.npmjs.com/package/@mergifyio/playwright)
package alongside `@playwright/test` to automatically upload your test results
to **Test Engine**.
### npm
```bash
npm install --save-dev @mergifyio/playwright
```
### yarn
```bash
yarn add --dev @mergifyio/playwright
```
### pnpm
```bash
pnpm add --save-dev @mergifyio/playwright
```
## Configuration
Setting up the reporter takes two steps.
### Wrap your Playwright config
Wrap your `playwright.config.ts` with `withMergify`:
```typescript
import { defineConfig } from '@playwright/test';
import { withMergify } from '@mergifyio/playwright';
export default withMergify(
defineConfig({
// ... your existing configuration
})
);
```
`withMergify` adds the Mergify reporter while preserving any reporters,
`globalSetup`, and `globalTeardown` you already have configured.
### Import `test` and `expect` from Mergify
In your test files, import `test` and `expect` from `@mergifyio/playwright`
instead of `@playwright/test`:
```typescript
import { test, expect } from '@mergifyio/playwright';
test('logs in', async ({ page }) => {
// ...
});
```
This import enables [test quarantine](/test-engine/quarantine): when a
quarantined test fails, its outcome is reported as passing so it doesn't block
your pipeline.
:::caution
If you wrap your config with `withMergify` but forget to update the `test`
import, the quarantine list is fetched but never applied, so quarantined
tests still fail your pipeline.
:::
## Update Your CI Workflow
Your workflow should run your tests as usual while exporting the secret
`MERGIFY_TOKEN` as an environment variable.
### GitHub Actions
Add the following to the GitHub Actions step running your tests:
```yaml
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
```
For example:
```yaml
- name: Run Tests 🧪
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
run: npx playwright test
```
### Buildkite
Set `MERGIFY_TOKEN` in the environment of the agents running your tests.
The step itself then needs no Mergify-specific configuration:
```yaml
steps:
- label: "Run Tests 🧪"
command: npx playwright test
```
The reporter automatically collects your test results and sends them to Test
Engine.
## Quarantine and Crashed Runs
The [`test` you import from `@mergifyio/playwright`](#import-test-and-expect-from-mergify)
applies [test quarantine](/test-engine/quarantine) inside the Playwright run.
When a quarantined test ends in any failing status, a timeout included, that
`test` sets the expected status to the outcome, so Playwright counts the result
as expected and the result does not change Playwright's exit code. The exit
code of your test step already accounts for quarantine, so the step needs
nothing more:
- Do not add `continue-on-error: true`. The recipes that upload a JUnit report
with the `mergifyio/gha-mergify-ci` action need it because the action decides
the job's result after the tests. Here nothing does, so it would let every
real failure through.
- There is no step `id` to set and no `test_step_outcome` to pass: both belong
to that action, which this setup does not use.
A crash cannot pass for a green run either. If Playwright dies mid-run, it
exits non-zero and the step fails. The reporter uploads results when the run
ends, so a process killed outright before then, such as by the out-of-memory
killer, sends nothing to Test Engine.
If the global setup that `withMergify` adds cannot fetch the quarantine list,
the run quarantines nothing, and a quarantined test that fails makes the step
fail as usual.
## Multi-Project (Cross-Browser) Runs
If your `playwright.config.ts` defines multiple
[projects](https://playwright.dev/docs/test-projects), such as one per browser
(`chromium`, `firefox`, `webkit`), the same test runs once per project. A test
is identified by its name within the job that ran it, and by default the
reporter leaves the project out of that name, so every project's copy of a test
shares one identity and is treated as a single test. A test that passes on `chromium`
but fails on `webkit` then looks flaky instead of consistently broken on one
browser.
To keep each project's tests separate, set
`PLAYWRIGHT_MERGIFY_INCLUDE_PROJECT_IN_TEST_NAME` to `true`. The reporter then
prefixes each test name with its project, for example
`[chromium] > login.spec.ts > logs in`, so Test Engine tracks flakiness and
quarantine per project:
```yaml
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
PLAYWRIGHT_MERGIFY_INCLUDE_PROJECT_IN_TEST_NAME: "true"
```
This option is opt-in (off by default) to preserve the history of tests that
already report without a project prefix.
## Matrix and Sharded Runs
The reporter uploads each run's results under your CI job's name. On GitHub
Actions, that name is the job's identifier (`GITHUB_JOB`), which every leg of
a matrix shares. The job name is
[part of a test's identity](/ci-insights/setup/github-actions#configuration-tips)
in Test Engine, so when the legs run the same tests in different
environments, their results merge into one test.
Set `MERGIFY_TEST_JOB_NAME` on each leg and include the matrix value, so each
leg reports under its own name:
```yaml
jobs:
e2e:
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
# ...
- name: Run Tests 🧪
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
MERGIFY_TEST_JOB_NAME: e2e (${{ matrix.os }})
run: npx playwright test
```
Legs that each run one `--shard` of the suite run different tests, so they can
share a name. Leave the shard index out of `MERGIFY_TEST_JOB_NAME`: adding or
removing tests can move a test to another shard, and a name per shard would
split that test's history. When you shard within an environment matrix, name
each leg after its environment only, as above.
Setting `MERGIFY_TEST_JOB_NAME` on a job that already reports changes its
name, so its tests start a new history in Test Engine.
## Verify and Review in Test Engine
After pushing these changes, your next CI run reports its Playwright results
automatically.
## Environment Variables
| Variable | Purpose | Default |
|----------|---------|---------|
| `MERGIFY_TOKEN` | API authentication token | **Required** |
| `MERGIFY_API_URL` | API endpoint location | `https://api.mergify.com` |
| `PLAYWRIGHT_MERGIFY_ENABLE` | Force-enable outside CI | `false` |
| `PLAYWRIGHT_MERGIFY_INCLUDE_PROJECT_IN_TEST_NAME` | Prefix the project name to tests in multi-project runs | `false` |
| `MERGIFY_CI_DEBUG` | Print spans to console instead of uploading | `false` |
| `MERGIFY_TRACEPARENT` | W3C distributed trace context | Optional |
| `MERGIFY_TEST_JOB_NAME` | Name this job reports under; set one per environment, never per shard | CI job name |
:::tip
The reporter auto-activates in CI environments (detected via the `CI`
environment variable). To enable it outside CI, set
`PLAYWRIGHT_MERGIFY_ENABLE=true`.
:::