Testing on Thin Ice: Chipping Away at Test Unpredictability
summary
Flaky tests cost an estimated 36 billion USD yearly and the talk shows concrete fixes: use Maven/Gradle retry plugin to detect flakes, push results into a DevVelocity-style dashboard, quarantine failing tests with a linked issue, run tests in parallel to reveal order/resource issues, prefer wait-for-condition over fixed sleeps, create/cleanup data with before/after hooks or testcontainers, mock external APIs in early environments, use ephemeral ports and deep links, and schedule “flaky-test days” to remove or rewrite persistent failures.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:49What is a flaky test
- 10:17Why flaky tests matter
- 11:00Testing pyramid revisited
- 16:18Finding flaky tests
- 19:50Detecting flaky tests in CI
- 22:17Preventing flaky tests
- 26:50Quarantining flaky tests
- 31:44Debugging CI-only failures
- 32:04Fixing test data issues
- 33:48Isolating tests
- 37:58Reducing flakiness in E2E tests
- 40:31Cultural shift needed
- 41:33Q&A
related talks
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
46mThe state of the Art in Tackling Flaky Tests
28mTest flakiness; methods for dealing with flaky tests
28mThe State of Art in Tackling Flaky Tests
42mDon't Let Your Tests Flake Out
41m0Flake - Reaching reliable non-flaky tests
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
15mHow to (not) write a flaky test