Test Flakiness Detector

Detect unreliable tests by running them multiple times and tracking failure rates.

Overview

Test Flakiness Detector helps identify non-deterministic tests by running your test suite multiple times and analyzing the results. It's framework-agnostic and works with any test command.

Status: Production Ready (v0.1.0)

Language: TypeScript

Repository: tuulbelt/test-flakiness-detector

Features

Framework Agnostic

Works with any test command - Jest, Vitest, Pytest, Cargo, Go tests, or any other test framework. Just provide the test command and run count.

Repeated Execution Analysis

Detects flaky tests by running your test suite multiple times and tracking which tests pass sometimes and fail sometimes.

Comprehensive JSON Reports

Generates detailed JSON reports with failure rates, individual run results, timestamps, and execution duration for analysis.

Zero Runtime Dependencies

Uses only Node.js built-ins. No npm install required in production.

Quick Start

# Clone the repository
git clone https://github.com/tuulbelt/test-flakiness-detector.git
cd test-flakiness-detector

# Install dev dependencies (for TypeScript)
npm install

# Detect flaky tests
flaky --test "npm test" --runs 10

Use Cases

• CI/CD Validation: Catch unreliable tests before they merge
• Test Suite Health: Identify which tests need fixing
• Pre-Release Checks: Ensure test stability before shipping
• Root Cause Analysis: Understand why tests fail intermittently
• Quality Metrics: Track test reliability over time

Why Flakiness Detection?

Flaky tests undermine confidence in your test suite:

1. False Positives: Tests fail even when code is correct
2. Wasted Time: Developers re-run tests or investigate non-issues
3. Reduced Trust: Teams start ignoring test failures
4. Hidden Bugs: Real failures get dismissed as "just flaky"
5. CI/CD Friction: Builds fail randomly, blocking deployments

This tool helps you identify and fix flaky tests systematically.

Demo

See the tool in action:

▶ View interactive recording on asciinema.org

Try it online:

Run the detector directly in your browser with zero setup. Test different commands, adjust run counts, and explore flakiness detection patterns.

Demos are automatically generated via GitHub Actions when demo scripts are updated.

Next Steps

• Getting Started Guide - Installation and setup
• CLI Usage - Command-line interface
• Library Usage - TypeScript/JavaScript API
• Examples - Real-world usage patterns
• API Reference - Complete API documentation

License

MIT License - see repository for details.