An Empirical Evaluation of Property-Based Testing in Python
summary
Corpus study of 426 Python projects using Hypothesis classifies PBTs into 12 dataflow-based categories, runs mutation testing on 40 projects, and reports an odds ratio of 52: property-based tests find ~50× as many mutations as unit tests. Tests that assert exception raising, collection inclusion, or types are most effective (over 19× more effective), detection is largely disjoint from unit tests, and a parameter sweep shows 76% within 20 inputs and many mutants found with single inputs.
AI-written summary · about
chapters (8)
- 0:00Property-based testing in Python
- 2:31PBT effectiveness questions
- 3:56Test effectiveness with mutation analysis
- 5:43Computational cost of PBT
- 6:45Recommended PBT properties
- 9:13PBT properties by data flow
- 10:36PBT recommendations and limits
- 12:01Q&A: unit testing vs PBT
related talks
27mHypothesis: Simpler Powerful Test Cases
41mProperty Based Testing in Python with Hypothesis
01h 24mIntroduction to Property Based Testing
26mProperty Based Testing the Python Way
01h 31mIntroduction to Property-Based Testing
41mProperty Based Testing with Hypothesis: Stronger Tests, Less Work
27mTesting Beyond Examples: Property Based Testing with Hypothesis
50mProperty based testing with Hypothesis