Test Engine

Test Engine (formerly Test Insights) helps you manage test reliability across the full lifecycle. It catches flaky tests on pull requests before they merge, surfaces unhealthy tests across your repositories, and lets you quarantine problematic tests so they don’t block your CI pipeline.

Test Engine is organized into three phases that follow the natural lifecycle of a test reliability problem:

1. Prevention: Catch flaky and broken tests on pull requests before they reach your codebase. Mergify reruns tests on PRs to detect inconsistent behavior early.

2. Detection: Identify and prioritize unhealthy tests across your repositories. See which tests are flaky or broken, and focus on the ones with the most impact.

3. Mitigation: Quarantine problematic tests to unblock CI without removing them. Tests keep running, but their failures no longer block merges.

You can also search test health directly from the terminal or from AI coding agents with the tests CLI reference. This is useful when an agent needs to decide whether a failing test is a real regression or known flakiness.

• Flaky test: A test that passed and failed close together on the same branch, pipeline, and job, so its outcome does not follow from the code alone. A test whose results differ without ever failing stays healthy.

• Broken test: A test that fails, with no passing run close in time to contradict those failures.

• Health status: A test’s reliability classification: healthy, flaky, or broken. Based on results from multiple CI runs.

• Confidence: How much rerun data backs a test’s health status, reported as low, medium, or high. Prevention reports it for the tests a pull request adds. Low confidence means the status could still change as the test is rerun.

• Quarantine: Isolating a test so its failures are ignored for merge decisions. The test still runs and results are still collected, preserving full visibility.

• Default branch scope: Test health across repositories comes from runs on the default branch. Pull request runs feed Prevention instead. See which runs Detection reports on.

Test Engine is powered by the same CI integration as CI Insights. To get started, configure your CI system and test framework:

• GitHub Actions setup
• Buildkite setup
• Jenkins setup
• Test framework configuration

Test Engine supports many test frameworks and captures test-level data from them. Each framework requires a quick setup to ensure test results are parsed and uploaded properly.

Refer to the dedicated page for your specific framework to configure and verify your test output.