Randomized testing: Gotta Catch ‘Em All
summary
Shows that adding randomness finds bugs missed by hand: uses the Carrot Search randomized runner (Carrot Search) with JUnit4 to vary locales, time zones, cluster size (1–20 nodes), shards 1–30, replicas up to nodes−1, bulk sizes (500–1000) and execution order; seed values make failures reproducible and tests can be repeated (10 iterations example). Finds real bugs — an AZ locale null-pointer in FSColor, leaked threads, and flaky ordering issues — and warns JUnit5 lacks built-in support.
AI-written summary · about
chapters (18)
- 0:00Welcome
- 4:40Ariane 5 failure lesson
- 7:13Speaker and project
- 11:08Randomized testing framework
- 18:16Randomness with seeds
- 21:45Locales and time zones
- 23:55Bulk operations testing
- 25:22Ignoring tests
- 27:08Skipping long tests
- 28:33Thread leaks detection
- 30:24Test execution order
- 31:48AZ locale bug in Elasticsearch
- 35:42Upgrading libraries
- 36:24Framework limitations
- 37:27Failing tests benefits
- 38:10Q&A: Convincing teams
- 40:40Q&A: Random vs deterministic
- 41:43Q&A: Measuring impact
related talks
14mChance Makes Good Tests
42mMake your tests fail
29mRandomness In Tests (Don't Do It)
59m.NET Testing Techniques You Didn’t Know You Needed
56mRandom Testing
20mWhat's the fuzz all about? Randomized data generation for robust unit testing
28mDetecting the unexpected in React applications
42mTesting on Thin Ice: Chipping Away at Test Unpredictability