Tuning Random Generators: Property-Based Testing as Probabilistic Programming
summary
Generators become probabilistic programs with symbolic weights and are tuned by computing a symbolic distribution, compiling it into compressed BDD-based representations, and optimizing weights by gradient descent with differentiable inference and REINFORCE-style estimators. The implementation Loaded Dice supports learning and parameterized generators; the paper introduces specification entropy to balance validity and diversity and uses knowledge compilation to scale. Evaluation on Etna benchmarks shows tuned generators find bugs about 3–7× faster.
AI-written summary · about
chapters (8)
- 0:00Property-based testing intro
- 1:03Fixed generator weights problem
- 2:51Generators as programs
- 4:59Loaded Dice system
- 6:25Scaling tuning challenges
- 8:11Conflicting tuning objectives
- 10:19Bug-finding speed evaluation
- 11:24Q&A
related talks
46mGenerative AI: Say Goodbye to Deterministic Testing?
27mReflecting on Random Generation
23mProperty based testing is a mindset
56mUsing, Generating and Testing with Pseudo-Random Numbers
56mRandom Testing
19mCovering All the Bases: Type-Based Verification of Test Input Generators
09mVerifying Sampling Algorithms via Distributional Invariants
28mLanguage Support for Property-Based Testing