---
title: ScalaTest Integration with Mergify
description: Report your test results from ScalaTest to Mergify
---

This guide shows how to upload the JUnit reports of your
[ScalaTest](https://www.scalatest.org/) suites, run with
[sbt](https://www.scala-sbt.org/), to **Test Engine** from your CI workflow.

## Generate a JUnit report with ScalaTest

sbt writes a JUnit report for every suite its `test` task runs, with no
setting to add. Run your tests:

```bash
sbt test
```

Each project of the build writes its reports in its own `test-reports`
directory, one file per suite, such as `TEST-com.example.CartSuite.xml`. Where
that directory sits depends on your sbt version:

- sbt 1 writes them under the project's own `target/test-reports/`, such as
  `core/target/test-reports/` for a `core` project.

- sbt 2 writes them under `target/out/jvm/scala-<version>/<project>/test-reports/`,
  such as `target/out/jvm/scala-3.3.7/core/test-reports/`. Run `sbt testFull`
  rather than `sbt test` on sbt 2: `test` reuses cached results, which leaves
  some reports missing or out of date.

The pattern `**/test-reports/*.xml` finds them with either version, in every
project of the build.

## Update your CI workflow

The examples below run `sbt test`. On sbt 2, use `sbt testFull` instead.

### GitHub Actions

> **Note:** your Mergify application key granted only [CI Reporting](/api/usage#ci-reporting) must be available to your CI as `MERGIFY_TOKEN`, as explained in the [CI setup guides](/ci-insights#ci-setup-guides). On GitHub Actions, a job can trade its ID token for a Mergify token instead: see [OIDC Authentication](/integrations/gha/oidc).

Run the tests in a step with an `id`, and let the upload step decide the
job's result:

```yaml
- name: Run ScalaTest Suites
  id: tests
  continue-on-error: true
  run: sbt test
```

```yaml
- name: Mergify CI Upload
  if: success() || failure()
  uses: mergifyio/gha-mergify-ci@v30
  with:
    token: ${{ secrets.MERGIFY_TOKEN }}
    report_path: "**/test-reports/*.xml"
    test_step_outcome: ${{ steps.tests.outcome }}
```

Key points:

- `if: success() || failure()`: runs the upload step even when the tests fail, so **Test Engine** gets the full report.
- `report_path: "**/test-reports/*.xml"`: where your test run writes the JUnit file. It must match the path your test configuration uses.
- `test_step_outcome: ${{ steps.tests.outcome }}`: passes the test step's outcome, so Mergify detects a runner that crashed while the JUnit report looks clean. Give your test step an `id` (here `tests`) and use it in `steps.<id>.outcome`.
- On pull requests from forks and from Dependabot, `MERGIFY_TOKEN` is empty and this step fails: skip it as shown in [Pull Requests Without the Secret](/ci-insights/setup/github-actions#pull-requests-without-the-secret).
- `token`: your `ci` application key. A job granted `id-token: write` can leave it out and authenticate with its ID token: see [OIDC Authentication](/integrations/gha/oidc).

If you use a job matrix in your workflow (e.g., to test across multiple versions), every leg reports the same job name, because GitHub Actions gives them all one `GITHUB_JOB`. Set the `job_name` input so **Test Engine** can tell the reports apart:

```yaml
jobs:
  example_matrix:
    strategy:
      matrix:
        version: [10, 12, 14]
```

Your upload step should look like:

```yaml
- name: Mergify CI Upload
  if: success() || failure()
  uses: mergifyio/gha-mergify-ci@v30
  with:
    job_name: example_matrix (${{ matrix.version }})
    token: ${{ secrets.MERGIFY_TOKEN }}
    report_path: "**/test-reports/*.xml"
    test_step_outcome: ${{ steps.tests.outcome }}
```

> **Note:** the job name is part of a test's identity, so a scheduled workflow has to report the same name as the job running those tests on your pull requests.

To benefit from Test Engine quarantine, you need to add `continue-on-error: true`
in your GitHub Actions step that executes your tests and generates the JUnit file.
The step running the `gha-mergify-ci` action then decides whether the job passes or fails,
taking quarantined tests into account.

You should also pass `test_step_outcome: ${{ steps.tests.outcome }}` to the step that runs
`mergifyio/gha-mergify-ci` (where `tests` is the `id` of your test runner step) to detect
silent failures where the test runner crashed but the JUnit report appears clean. Without
this input, a crash that produces a partial or empty report could be mistakenly treated as a
success.

With `continue-on-error: true`, the job's result comes from the upload step. On
pull requests from forks and from Dependabot, `MERGIFY_TOKEN` is empty and the
upload step has to be skipped: add the step that fails the job when the tests
failed and nothing was uploaded, as shown in
[pull requests without the secret](/ci-insights/setup/github-actions#pull-requests-without-the-secret).
A job using [OIDC](/integrations/gha/oidc) skips the upload on pull requests from
forks only, as that page shows.

### Buildkite

```yaml
steps:
  - label: "Run tests"
    command: <your test command>
    plugins:
      - mergifyio/mergify-ci#v15:
          action: junit-process
          report_path: "**/test-reports/*.xml"
```

Key points:

- The plugin runs in the `post-command` hook, so it uploads results **after** your tests finish, even if they fail.
- `report_path: "**/test-reports/*.xml"`: where your test run writes the JUnit file. It must match the path your test configuration uses.
- Silent failure detection is automatic: the plugin reads the step's exit code to detect a runner that crashed while the JUnit report looks clean.

If you use a build matrix in your pipeline, each variation reports its own step label as its job name, so the variations are already distinct. Set the `job_name` plugin property when the label is not the name you want **Test Engine** to file the reports under:

```yaml
steps:
  - label: "Flaky detection ({{matrix}})"
    matrix:
      - "3.10"
      - "3.11"
      - "3.12"
    command: <your test command>
    plugins:
      - mergifyio/mergify-ci#v15:
          action: junit-process
          job_name: "Tests ({{matrix}})"
          report_path: "**/test-reports/*.xml"
```

> **Note:** the job name is part of a test's identity, so a scheduled pipeline has to report the same name as the step running those tests on your pull requests.

> **Caution: keep the token out of the pipeline you upload.** Set `MERGIFY_TOKEN` in the environment of the agents running your steps: [Buildkite secrets](https://buildkite.com/docs/pipelines/security/secrets/buildkite-secrets), a secrets plugin backed by your own secrets store, or an agent `environment` hook that exports it. Do not put the token in the pipeline itself: `buildkite-agent pipeline upload` interpolates `${MERGIFY_TOKEN}` before the pipeline is stored, so a token written into an `env:` block is uploaded in plain text and stays readable in the Buildkite UI and in build history. The plugin's `token:` property carries it the same way.

When using the Buildkite plugin for Test Engine quarantine, the plugin automatically
detects test failures using the step's exit code. No extra configuration is needed.

The `mergifyio/mergify-ci` plugin runs in the `post-command` hook and reads
`BUILDKITE_COMMAND_EXIT_STATUS` to detect silent failures where the test runner
crashed but the JUnit report appears clean.

### Any CI (Mergify CLI)

Quote the pattern, so that `mergify ci junit-process` expands it rather than
the shell: without `globstar`, bash reads `**` as `*` and misses the reports
at other depths.

Install the [Mergify CLI](/cli/usage) in your pipeline and export `MERGIFY_TOKEN`. Run `mergify ci junit-process` after your tests:

```bash
set -o pipefail
exit_code=0
sbt test || exit_code=$?
mergify ci junit-process \
  --test-exit-code "$exit_code" \
  "**/test-reports/*.xml"
```

Key points:

- `MERGIFY_TOKEN`: export this environment variable with your Mergify application key so `junit-process` can authenticate to the API.
- `|| exit_code=$?`: CI shells run with `-e`, so a failing test command on its own line would end the script before the upload. This form keeps going and records the code.
- The script needs bash for `set -o pipefail`: on Jenkins, whose `sh` steps often run `dash`, start it with `#!/bin/bash`.
- `--test-exit-code "$exit_code"`: passes the test runner's exit code, so Mergify detects a runner that crashed while the JUnit report looks clean.
- `junit-process` runs last, so its status becomes the step's: `0` when every failure it found is quarantined, `1` when one is not.
- The branch quarantine is evaluated against is detected from your CI provider, falling back to the checked-out branch. Pass `--tests-target-branch <branch>` only to override it.

## Test names

Test Engine names each test by its suite class's full name, then a dot and the
test's full text. In a `FunSuite`, `test("adds an item")` in
`com.example.CartSuite` is reported as `com.example.CartSuite.adds an item`.
In styles that nest, the text joins every level with spaces:
`"An account" when { "banned" should { "refuse a login" in ... } }` is
reported as `com.example.AccountSpec.An account when banned should refuse a login`.

Renaming a suite, or changing a test's text at any level, starts a new history
for the tests it holds. Ignored tests (`ignore`) and pending tests (`pending`)
are reported as skipped.

## Verify and review in Test Engine

After pushing these changes:

1. Your CI workflow runs your ScalaTest suites with sbt.
2. sbt writes a report per suite in each project's `test-reports` directory.
3. Mergify uploads the reports to Test Engine.

You can then review your test results, including any failures or flaky tests,
directly in the [Test Engine
dashboard](https://dashboard.mergify.com/test-engine/detection).

## Troubleshooting tips

<ul>
  <li>
    Tests missing from Test Engine with sbt 2: `test` skips the suites that passed in a cached
    run. Run `testFull`.
  </li>

  <li>The CLI provides information about the upload. Check the logs in GitHub Actions.</li>
<li>File paths: Double-check that the output file matches the path used in `report_path`.</li>
<li>Permissions: Make sure the `MERGIFY_TOKEN` is valid and setup in your GitHub Actions secrets [as explained in the docs](/ci-insights/setup/github-actions#enabling-ci-insights-for-github-actions).</li>
<li>Workflow conditions: If your step is not running, confirm the if condition is actually triggered in your job.</li>
</ul>
