Testing on Thin Ice: Chipping Away at Test Unpredictability
summary
Claims that flaky tests cost $36 billion wasted yearly, shows cases where fixed sleeps, unique-constraint collisions, CI load and parallel runs cause nondeterministic failures, and demonstrates concrete fixes: run tests in parallel to expose contention, use retry failing test count or fail-on-flake Maven/Gradle flags, quarantine with searchable tags, adopt Testcontainers, JUnit5 before/after setup, deep links for E2E, and team rituals like nightly reruns or a “flaky test day” to remove or rewrite unstable tests.
AI-written summary · about
chapters (8)
- 0:00Introduction to flaky tests and their impact
- 1:36Defining what constitutes a flaky test
- 2:04Common causes of test flakiness
- 14:11Strategies for identifying flaky tests
- 19:40How to prevent and manage flaky tests
- 24:48Quarantining and fixing unstable tests
- 30:53Techniques for writing stable, isolated tests
- 40:23Addressing the cultural aspect of test reliability
related talks
47mTesting on Thin Ice: Chipping Away at Test Unpredictability
28mTest flakiness; methods for dealing with flaky tests
42mDon't Let Your Tests Flake Out
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
46mThe state of the Art in Tackling Flaky Tests
41m0Flake - Reaching reliable non-flaky tests
15mHow to (not) write a flaky test
42mFlaky tests are a pain