Understanding and Improving Flaky Test Classification
summary
Shows that prior flaky-test classifiers leak data and use unrealistic mixes so reported F1 falls from 81.82% to 56.62% after fixes; builds a realistic dataset FlakeBench with ~3% flaky tests and project-wise splits; fine-tunes a new model FlakyLens using focal loss and L2 regularization to raise F1 to 65.79%; uses integrated gradients to find high-attribution tokens and performs semantic-preserving perturbations that change accuracy by up to -18.37pp, exposing token reliance rather than true semantics.
AI-written summary · about
chapters (10)
- 0:00Flaky test classification intro
- 1:06What makes tests flaky
- 2:30Limitations of classifiers
- 4:36FlakeBench dataset
- 5:58FlakyLens model design
- 7:02FlakyLens performance
- 7:45Model decisions with token attribution
- 9:10Token importance validation
- 10:58Contributions and future work
- 11:20Q&A
related talks
16mA Large-Scale Longitudinal Study of Flaky Tests
26m"Tests as Classifiers"
41m0Flake - Reaching reliable non-flaky tests
22mTest Confidence: Types of Flake & How to Remediate
28mTest flakiness; methods for dealing with flaky tests
28mThe State of Art in Tackling Flaky Tests
46mThe state of the Art in Tackling Flaky Tests
42mFlaky tests are a pain