Fixing Flaky Tests Like a Detective
summary
Describes a concrete debugging recipe: gather CI failures into a bug tracker, classify flakes into the five usual suspects—async code (Capybara, have_content, save_screenshot), order dependency (DatabaseCleaner, RSpec bisect and randomized runs), time issues (Timecop or Timecop.travel), unordered collections, and randomness (use explicit values not Faker, control Kernel.srand). Shows fixes and replication techniques, recommends assigning owners, logging extra state, and tracking a target pass rate (example target: 90% master build pass rate).
AI-written summary · about
chapters (23)
- 0:00Fixing Flaky Tests Like a Detective
- 4:37Types of Evidence
- 7:06Async code in system/feature tests
- 9:45Another example: async code
- 11:17Identifying async code flakes
- 12:20Preventing async flakes
- 14:05Potential areas of shared state
- 14:28Database cleaning
- 16:47Know your database cleaner
- 17:17Example: order dependency
- 18:25Other sources of order dependency
- 19:16Identifying order dependency
- 20:13Preventing order dependency
- 20:58Example: Time
- 22:40Identifying time-based flakes
- 23:09Preventing time flakes
- 24:25Example: Unordered Collections
- 25:55Example: Randomness
- 27:07Identifying randomness-based
- 27:47Preventing randomness based
- 28:28Strategy tips
- 29:55If you're stuck
- 33:37Fixing flaky tests as a team
related talks
29mCommon Mistakes that make test cases flaky and how to avoid them
34mEliminating Inconsistent Test Failures
41mDeterministic Solutions to Intermittent Failures
21mTaming Flaky Specs: Diagnosing Test Failures in Ruby
47mHow to Un-Flake Flaky Tests- a New Hire's Toolkit
01h 15mTaking Control Of Your Test Environment
15mHow to (not) write a flaky test
46mBlink and You’ll Miss It: Stories From Discovering Most Annoying Bugs and Test Flakes