Test Engine

Test Engine (formerly Test Insights) helps you manage test reliability across the full lifecycle. It catches flaky tests on pull requests before they merge, surfaces unhealthy tests across your repositories, and lets you quarantine problematic tests so they don’t block your CI pipeline.

Test Engine is organized into three phases that follow the natural lifecycle of a test reliability problem:

1. Prevention: Catch flaky and broken tests on pull requests before they reach your codebase. Mergify reruns tests on PRs to detect inconsistent behavior early.

2. Detection: Identify and prioritize unhealthy tests across your repositories. See which tests are flaky or broken, and focus on the ones with the most impact.

3. Quarantine: Unblock CI without removing problematic tests. Tests keep running, but their failures no longer block merges. You quarantine and review them from Detection.

You can also search test health directly from the terminal or from AI coding agents with the tests CLI reference. This is useful when an agent needs to decide whether a failing test is a real regression or known flakiness.

• Flaky test: A test whose outcome does not follow from the code alone, for example one that passed and failed on the same commit.

• Broken test: A test that fails, with nothing pointing to flakiness.

• Health status: A test’s reliability classification: healthy, flaky, or broken. Based on the test’s CI runs over time. See Health Status for how a test moves between them.

• Confidence: How much rerun data backs a test’s health status, reported as low, medium, or high. Prevention reports it for the tests a pull request adds. Low confidence means the status could still change as the test is rerun.

• Quarantine: Isolating a test so its failures are ignored for merge decisions. The test still runs and results are still collected, preserving full visibility.

• Default branch scope: Test health across repositories comes from runs on the default branch. Pull request runs feed Prevention instead. See which runs Detection reports on.

Test Engine is powered by the same CI integration as CI Insights. To get started, configure your CI system and test framework:

• GitHub Actions setup
• Buildkite setup
• Jenkins setup
• CircleCI setup
• Test framework setup

Then set the rerun budgets and turn on auto-quarantine for each repository on the Configuration page.

Test Engine captures test-level data from your test framework. Each framework needs a short setup so its results reach Test Engine: a Mergify plugin where one exists, or a JUnit XML report uploaded after the tests run.

Test framework setup lists every supported framework, grouped by language, with the setup each one uses.