Some Problems with Properties: A Study on Property-Based Testing in Industry
summary
A preliminary interview study with seven interviews finds engineers struggle to write generators (failing to satisfy nontrivial preconditions, bad distributions, awkward generator DSLs) and to find meaningful properties beyond “does not crash.” An expanded study with 29 of 30 developers at Jane Street (OCaml users) confirms generator pain, reveals heavy use of automation, frequent model-based testing, and friction around CI nondeterminism (teams often set seeds in CI). Ongoing work aims to build evaluation tools, prototypes, surveys, and improved docs.
AI-written summary · about
chapters (16)
- 0:00Property-based testing study
- 1:49What is property-based testing
- 3:12Real-world impact focus
- 4:17Interview study findings
- 6:45Generator design challenges
- 8:08Property design issues
- 9:31Model-based testing in practice
- 10:36Workflow struggles
- 11:21Education barriers
- 12:48Bootstrapping property-based testing
- 13:31Jane Street developer study
- 15:16Refining study themes
- 17:03Generator improvements
- 17:46Evaluating effectiveness
- 18:31Future research directions
- 19:35Q&A
related talks
28mLanguage Support for Property-Based Testing
01h 24mIntroduction to Property Based Testing
20mUnlock The Power Of Property-Based Testing
23mProperty based testing is a mindset
58mProperty Based Testing: Concepts and Examples
54mAn Introduction to Property-based Testing
45mProperty based tests for the masses
26mThe missing piece of testing