Deterministic Solutions to Intermittent Failures
summary
Claims a repeatable workflow that turns “flaky” tests into deterministic failures, with concrete tools and steps: run the failing test alone, re-run the CI test group with the reported RSpec --seed, use RSpec’s --bisect to find the minimal reproducer, loop a test 100.times, and inspect data pollution, unordered queries, caching, timezones, and async timeouts. Cites Timecop/Timecop‑rspec, SitePrism load validation, parallel CI pain, and months of work fixing thousands of tests to make builds stable.
AI-written summary · about
chapters (22)
- 0:00Intro
- 2:30Source Code Examples
- 3:47Tests Are Software Too
- 10:32Common Reproducible Failures
- 13:12RSpec Test Group
- 14:03Re-running Test Group With Seed
- 16:23Data Pollution
- 16:43Defensive Testing
- 18:52Class/Singleton Caching
- 20:33Mutated Constants
- 22:14Mutated (Test) Constants
- 24:07Running Tests in a Loop
- 25:12Unordered Queries
- 26:42Frozen Time
- 27:48Randomized Test Data
- 29:18Date and Time
- 30:24UTC vs Local Date/Time
- 31:23SQL Date Comparisons
- 32:46Timeouts and Asynchronous Javascript
- 35:19Environmental Differences
- 37:02Strategies for Unreproducible Failures
- 39:56Takeaways
related talks
34mEliminating Inconsistent Test Failures
29mCommon Mistakes that make test cases flaky and how to avoid them
36mFixing Flaky Tests Like a Detective
24mHow to solve intermittent issues, aka Random Failures
21mTaming Flaky Specs: Diagnosing Test Failures in Ruby
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
46mThe Road to Reliable Automated Tests
45mThe Secret Ingredient: How To Understand and Resolve Just...