CI and the Great Flakiness Adventure
summary
Defines intrarun vs interrun flakiness and a four-step lifecycle: detect assign resolve verify. Recommends automated cron jobs that pull historical results from Datadog/Velocity, quarantine flaky tests via a repo block list in repo (dots skip tests), and assign a single owner via CODEOWNERS/git blame. Notes trade-offs: retries inflate CI time, running new tests repeatedly (meta-style gate) prevents regressions, and a 6% failure rate example guides thresholds for treating a test as flaky.
AI-written summary · about
chapters (20)
- 0:00CI flakiness introduction
- 1:57Defining flaky tests
- 4:11Causes of test flakiness
- 6:51Flaky tests as signals
- 8:00Trust erosion in CI
- 8:21Desired CI test properties
- 9:09Detecting flaky tests
- 11:26Ownership of flaky tests
- 13:20Quarantining flaky tests
- 14:51Preventing flaky tests
- 17:04Reducing flakiness impact
- 19:21Optimizing test selection
- 21:15Test pyramid trade-offs
- 22:22Rewriting flaky tests
- 23:48Verifying flaky tests
- 24:57Validating test environments
- 25:44Summary
- 26:05Q&A: AI for flakes
- 27:11Q&A: Flakiness thresholds
- 28:51Q&A: Flakiness root causes
related talks
28mThe State of Art in Tackling Flaky Tests
41m0Flake - Reaching reliable non-flaky tests
46mThe state of the Art in Tackling Flaky Tests
36mFixing Flaky Tests Like a Detective
15mHow to (not) write a flaky test
28mTest flakiness; methods for dealing with flaky tests
01h 15mTaking Control Of Your Test Environment
16mA Large-Scale Longitudinal Study of Flaky Tests