View as Markdown

Test Engine

Monitor, detect, and manage unreliable tests across your repositories.


Test Engine (formerly Test Insights) helps you manage test reliability across the full lifecycle. It catches flaky tests on pull requests before they merge, surfaces unhealthy tests across your repositories, and lets you quarantine problematic tests so they don’t block your CI pipeline.

Test Engine is organized into three phases that follow the natural lifecycle of a test reliability problem:

  1. Prevention: Catch flaky and broken tests on pull requests before they reach your codebase. Mergify reruns tests on PRs to detect inconsistent behavior early.

  2. Detection: Identify and prioritize unhealthy tests across your repositories. See which tests are flaky or broken, and focus on the ones with the most impact.

  3. Quarantine: Unblock CI without removing problematic tests. Tests keep running, but their failures no longer block merges. You quarantine and review them from Detection.

In the merge queue, Fail First finds the pull request that broke a batch sooner: when the queue splits a failed batch, each split starts with the tests that already failed and stops early if they fail again.

On your default branch, test retry runs a failing test again, so a flaky failure no longer fails the build.

You can also search test health directly from the terminal or from AI coding agents with the tests CLI reference. This is useful when an agent needs to decide whether a failing test is a real regression or known flakiness.

  • Flaky test: A test whose outcome does not follow from the code alone, for example one that passed and failed on the same commit.

  • Broken test: A test that fails, with nothing pointing to flakiness.

  • Health status: A test’s reliability classification: healthy, flaky, or broken. Based on the test’s CI runs over time. See Health Status for how a test moves between them.

  • Confidence: How many runs support a test’s health status, reported as low, medium, or high. Prevention reports it for the tests a pull request adds. Low confidence means the status could still change as the test runs again.

  • Quarantine: Isolating a test so its failures are ignored for merge decisions. The test still runs and results are still collected, preserving full visibility.

  • Tracked branches: A test’s health comes from the runs on each tracked branch. A repository tracks its default branch until you choose others. Pull request runs feed Prevention instead. See which runs Detection reports on.

Test Engine is powered by the same CI integration as CI Insights. To get started, configure your CI system and test framework:

To let your coding agent do the whole setup, give it the mergify-test-engine-setup skill.

Then set the rerun budgets and turn on auto-quarantine for each repository on the Configuration page.

Test Engine captures test-level data from your test framework. Each framework needs a short setup so its results reach Test Engine: a Mergify plugin where one exists, or a JUnit XML report uploaded after the tests run.

Test framework setup lists every supported framework, grouped by language, with the setup each one uses.

Was this page helpful?