A Large-Scale Longitudinal Study of Flaky Tests
summary
Empirical study applies iDFlakies and NonDex plus Isolation and One-By-One to 68 Maven Java projects, detecting 684 flaky tests and evaluating 245 at their Test-Introducing commit. Results show 75% of flaky tests are flaky when added, running detectors on added or directly modified tests finds up to 85%, and remaining cases often appear a median of 144 commits later. Public dataset of flakiness-introducing commits and labels is available.
AI-written summary · about
chapters (9)
- 0:00Intro
- 1:57Flaky test - Example in different orders
- 3:45Decreasing detector cost
- 4:41Our contributions
- 5:20Study setup: Overview
- 6:48Study setup: Step 1
- 8:41Study setup: Steps 2-4
- 10:05Study setup: Flaky tests
- 12:03RQ3: When should one run detectors?
related talks
28mTest flakiness; methods for dealing with flaky tests
28mThe State of Art in Tackling Flaky Tests
46mThe state of the Art in Tackling Flaky Tests
31mCI and the Great Flakiness Adventure
41m0Flake - Reaching reliable non-flaky tests
47mTesting on Thin Ice: Chipping Away at Test Unpredictability
01h 09mFlaky Tests: The Method
42mTesting on Thin Ice: Chipping Away at Test Unpredictability