Taming Flaky Specs: Diagnosing Test Failures in Ruby
summary
Checklist and concrete fixes for flaky Ruby specs: identifies that about 50% of flakiness stems from inconsiderate assertions (count-based job checks, exact-time equality), shows how unstubbed network calls reveal themselves by turning off the internet, explains Time.zone vs local time problems and when to use time freezing or Rails time helpers, warns about mixing transactional and non-transactional tests, demonstrates how mis-set Sidekiq modes break order-dependent runs, and provides a small script + randomized CI log replay to reproduce and isolate failures.
AI-written summary · about
chapters (10)
- 0:00Introduction to flaky specs
- 1:42Why flaky specs are interesting
- 3:08Goal of debugging flaky specs
- 3:29Inconsiderate assertions explained
- 5:20Unstubbed network calls issue
- 6:47Time zone confusion in tests
- 11:06Aftereffects of non-transactional tests
- 12:54Misuse of test environment helpers
- 16:13Object pollution in Ruby
- 19:40Summary and conclusion
related talks
29mCommon Mistakes that make test cases flaky and how to avoid them
36mFixing Flaky Tests Like a Detective
42mDon't Let Your Tests Flake Out
34mEliminating Inconsistent Test Failures
41mDeterministic Solutions to Intermittent Failures
45mThe Secret Ingredient: How To Understand and Resolve Just...
15mHow to (not) write a flaky test
31mGoing on a Testing Anti-Pattern Safari