--- title: Flutter Integration with Mergify description: Report your test results from Flutter to Mergify --- This guide shows how to generate a JUnit report from the tests of your [Flutter](https://flutter.dev/) app, run with `flutter test`, and upload it to **Test Engine** from your CI workflow. For a Dart package without Flutter, see the [Dart guide](/test-engine/test-frameworks/dart). ## Generate a JUnit report with Flutter `flutter test` has no JUnit output of its own. It writes its results as JSON, which the [`junitreport`](https://pub.dev/packages/junitreport) package converts to JUnit XML. Install the converter once on the machine that runs your tests: ```bash dart pub global activate junitreport ``` Run your tests with a JSON file reporter. `--file-reporter` writes the JSON to a file and keeps the usual output in your terminal: ```bash flutter test --file-reporter json:test-results.json ``` Then convert the JSON to JUnit XML, passing your checkout's path as `--base`: ```bash dart pub global run junitreport:tojunit --base "$PWD/" --input test-results.json --output junit.xml ``` `flutter test` records each test file by its absolute path. `--base` strips your checkout's path from it, so test names do not change with the directory your CI checks the repository out to. Run the conversion even when tests fail: `flutter test` exits with a non-zero status then, and the conversion is what writes the report you upload. ## Update your CI workflow ### GitHub Actions > **Note:** your Mergify application key with the `ci` scope must be available to your CI as `MERGIFY_TOKEN`, as explained in the [CI setup guides](/ci-insights#ci-setup-guides). Run the tests in a step with an `id`, convert their results, and let the upload step decide the job's result: ```yaml - name: Install junitreport run: dart pub global activate junitreport - name: Run Flutter Tests id: tests continue-on-error: true run: flutter test --file-reporter json:test-results.json - name: Convert the results to JUnit run: dart pub global run junitreport:tojunit --base "$PWD/" --input test-results.json --output junit.xml ``` ```yaml - name: Mergify CI Upload if: success() || failure() uses: mergifyio/gha-mergify-ci@v27 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} ``` Key points: - `if: success() || failure()`: runs the upload step even when the tests fail, so **Test Engine** gets the full report. - `report_path: junit.xml`: where your test run writes the JUnit file. It must match the path your test configuration uses. - `test_step_outcome: ${{ steps.tests.outcome }}`: passes the test step's outcome, so Mergify detects a runner that crashed while the JUnit report looks clean. Give your test step an `id` (here `tests`) and use it in `steps..outcome`. - On pull requests from forks and from Dependabot, `MERGIFY_TOKEN` is empty and this step fails: skip it as shown in [Pull Requests Without the Secret](/ci-insights/setup/github-actions#pull-requests-without-the-secret). If you use a job matrix in your workflow (e.g., to test across multiple versions), every leg reports the same job name, because GitHub Actions gives them all one `GITHUB_JOB`. Set the `job_name` input so **Test Engine** can tell the reports apart: ```yaml jobs: example_matrix: strategy: matrix: version: [10, 12, 14] ``` Your upload step should look like: ```yaml - name: Mergify CI Upload if: success() || failure() uses: mergifyio/gha-mergify-ci@v27 with: job_name: example_matrix (${{ matrix.version }}) token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} ``` > **Note:** the job name is part of a test's identity, so a scheduled workflow has to report the same name as the job running those tests on your pull requests. To benefit from Test Engine quarantine, you need to add `continue-on-error: true` in your GitHub Actions step that executes your tests and generates the JUnit file. The step running the `gha-mergify-ci` action then decides whether the job passes or fails, taking quarantined tests into account. You should also pass `test_step_outcome: ${{ steps.tests.outcome }}` to the step that runs `mergifyio/gha-mergify-ci` (where `tests` is the `id` of your test runner step) to detect silent failures where the test runner crashed but the JUnit report appears clean. Without this input, a crash that produces a partial or empty report could be mistakenly treated as a success. With `continue-on-error: true`, the job's result comes from the upload step. On pull requests from forks and from Dependabot, `MERGIFY_TOKEN` is empty and the upload step has to be skipped: add the step that fails the job when the tests failed and nothing was uploaded, as shown in [pull requests without the secret](/ci-insights/setup/github-actions#pull-requests-without-the-secret). ### Buildkite Run the conversion in your step's `command`, after the tests, so it runs whatever their outcome, and exit with the tests' status: ```bash dart pub global activate junitreport status=0 flutter test --file-reporter json:test-results.json || status=$? dart pub global run junitreport:tojunit --base "$PWD/" --input test-results.json --output junit.xml exit $status ``` ```yaml steps: - label: "Run tests" command: plugins: - mergifyio/mergify-ci#v8: action: junit-process report_path: junit.xml ``` Key points: - The plugin runs in the `post-command` hook, so it uploads results **after** your tests finish, even if they fail. - `report_path: junit.xml`: where your test run writes the JUnit file. It must match the path your test configuration uses. - Silent failure detection is automatic: the plugin reads the step's exit code to detect a runner that crashed while the JUnit report looks clean. If you use a build matrix in your pipeline, each variation reports its own step label as its job name, so the variations are already distinct. Set the `job_name` plugin property when the label is not the name you want **Test Engine** to file the reports under: ```yaml steps: - label: "Flaky detection ({{matrix}})" matrix: - "3.10" - "3.11" - "3.12" command: plugins: - mergifyio/mergify-ci#v8: action: junit-process job_name: "Tests ({{matrix}})" report_path: junit.xml ``` > **Note:** the job name is part of a test's identity, so a scheduled pipeline has to report the same name as the step running those tests on your pull requests. > **Caution: keep the token out of the pipeline you upload.** Set `MERGIFY_TOKEN` in the environment of the agents running your steps: [Buildkite secrets](https://buildkite.com/docs/pipelines/security/secrets/buildkite-secrets), a secrets plugin backed by your own secrets store, or an agent `environment` hook that exports it. Do not put the token in the pipeline itself: `buildkite-agent pipeline upload` interpolates `${MERGIFY_TOKEN}` before the pipeline is stored, so a token written into an `env:` block is uploaded in plain text and stays readable in the Buildkite UI and in build history. The plugin's `token:` property carries it the same way. When using the Buildkite plugin for Test Engine quarantine, the plugin automatically detects test failures using the step's exit code. No extra configuration is needed. The `mergifyio/mergify-ci` plugin runs in the `post-command` hook and reads `BUILDKITE_COMMAND_EXIT_STATUS` to detect silent failures where the test runner crashed but the JUnit report appears clean. ### Any CI (Mergify CLI) Install the [Mergify CLI](/cli/usage) in your pipeline and export `MERGIFY_TOKEN`. Run the conversion between your tests and `mergify ci junit-process`: ```bash set -o pipefail exit_code=0 flutter test --file-reporter json:test-results.json || exit_code=$? dart pub global run junitreport:tojunit --base "$PWD/" --input test-results.json --output junit.xml mergify ci junit-process \ --test-exit-code "$exit_code" \ junit.xml ``` `--test-exit-code` passes the tests' status to Mergify, which uses it to catch a run that crashed but left a clean report behind. `junit-process` exits last, so its status, which accounts for quarantined tests, becomes the step's. ## Test names Test Engine names each test by its file's path without `_test.dart`, dotted, then a dot and its groups and description, joined with spaces: the test `'adds an item'` inside `group('add()')` inside `group('Cart')` of `test/cart_test.dart` is reported as `test.cart.Cart add() adds an item`. Moving or renaming the file, a group or the test starts a new history for it. Widget tests are named the same way: `testWidgets('Counter increments smoke test')` in `test/widget_test.dart` is reported as `test.widget.Counter increments smoke test`. ## Verify and review in Test Engine After pushing these changes: 1. Your CI workflow runs your Flutter tests. 2. `flutter test` writes its results to `test-results.json`, and `tojunit` converts them to `junit.xml`. 3. Mergify uploads the report to Test Engine. You can then review your test results, including any failures or flaky tests, directly in the [Test Engine dashboard](https://dashboard.mergify.com/test-engine/detection). ## Troubleshooting tips
  • Test names hold a path such as `.home.runner.work`: pass `--base "$PWD/"` to `tojunit`, from the root of your checkout.
  • Report not found when tests fail: check that the conversion runs after a failed test run, as in the examples above.
  • The CLI provides information about the upload. Check the logs in GitHub Actions.
  • File paths: Double-check that the output file matches the path used in `report_path`.
  • Permissions: Make sure the `MERGIFY_TOKEN` is valid and setup in your GitHub Actions secrets [as explained in the docs](/ci-insights/setup/github-actions#enabling-ci-insights-for-github-actions).
  • Workflow conditions: If your step is not running, confirm the if condition is actually triggered in your job.