CLI Usage

Command-line interface for Test Flakiness Detector.

Basic Syntax

flaky [options]

Options

--test <command>

Required. The test command to execute repeatedly.

flaky --test "npm test"

--runs <number>

Optional. Number of times to run the tests. Default: 10, Max: 1000

flaky --test "npm test" --runs 20

--verbose

Optional. Enable verbose output showing each run's result. Default: false

flaky --test "npm test" --verbose

--help

Display help message with all available options.

flaky --help

Examples

Basic Flakiness Detection

flaky --test "npm test" --runs 10

With Verbose Output

flaky --test "npm test" --runs 20 --verbose

Different Test Frameworks

Jest:

flaky --test "npm run test:jest" --runs 15

Pytest:

flaky --test "pytest tests/" --runs 20

Cargo (Rust):

flaky --test "cargo test" --runs 10

Go:

flaky --test "go test ./..." --runs 15

Vitest:

flaky --test "vitest run" --runs 20

Output Format

The tool outputs JSON to flakiness-report.json:

{
  "success": true,
  "totalRuns": 10,
  "passedRuns": 8,
  "failedRuns": 2,
  "flakyTests": [
    {
      "testName": "Test Suite",
      "passed": 8,
      "failed": 2,
      "totalRuns": 10,
      "failureRate": 20.0
    }
  ],
  "runs": [
    {
      "success": true,
      "exitCode": 0,
      "stdout": "...",
      "stderr": ""
    },
    {
      "success": false,
      "exitCode": 1,
      "stdout": "...",
      "stderr": "Error: test failed"
    }
    // ... more runs
  ]
}

Exit Codes

• 0 — All tests passed consistently (no flakiness detected)
• 1 — Flaky tests detected, or all tests failed consistently
• 2 — Invalid arguments or execution error

Integration with CI/CD

GitHub Actions

name: Flakiness Detection

on: [push, pull_request]

jobs:
  detect-flakiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Run flakiness detection
        run: |
          cd test-flakiness-detector
          flaky --test "npm test" --runs 20

      - name: Upload report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: flakiness-report
          path: test-flakiness-detector/flakiness-report.json

GitLab CI

test:flakiness:
  script:
    - cd test-flakiness-detector
    - npm install
    - flaky --test "npm test" --runs 20
  artifacts:
    paths:
      - test-flakiness-detector/flakiness-report.json
    when: always

Jenkins

stage('Flakiness Detection') {
  steps {
    dir('test-flakiness-detector') {
      sh 'npm install'
      sh 'flaky --test "npm test" --runs 20'
      archiveArtifacts artifacts: 'flakiness-report.json'
    }
  }
}

Tips & Best Practices

Choosing Run Count

• Highly flaky tests: 5-10 runs sufficient
• Occasionally flaky: 20-50 runs recommended
• Rarely flaky: 100+ runs may be needed
• Production validation: 50-100 runs for confidence

Use Verbose Mode for Debugging

flaky --test "npm test" --runs 10 --verbose

This shows the output of each individual run, helpful for diagnosing why tests are flaky.

Analyzing Results

# Run detection
flaky --test "npm test" --runs 20

# Check the report
cat flakiness-report.json | jq '.summary'

# Count flaky runs
cat flakiness-report.json | jq '.runs | map(select(.success == false)) | length'

Performance Considerations

For slow test suites, use fewer runs:

# For suites that take > 1 minute
flaky --test "npm test" --runs 5

# For fast suites (< 10 seconds)
flaky --test "npm test" --runs 50

See Also

• Getting Started — Installation and setup
• Examples — Real-world usage examples
• API Reference — Programmatic API
• Library Usage — TypeScript/JavaScript integration