Grammar-Based Testing for Little Languages: An Experience Report with Student Compilers
summary
Evaluates grammar-based test generation on a small teaching language called Ample and claims generated suites reach parity with manual suites for crash testing but not for coverage or parsing-error detection. Uses 61 final submissions from a student compiler course, compares systematic coverage algorithms vs random derivations, shows very large random sets (about 6000 tests / 0.5M tokens) match manual crash-finding, and adds lightweight semantic rendering using dcl and ref markup to inject and detect scoping and typing faults.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:21The $64,000 Question of Compiler Testing
- 1:14Experimental Setup: A Modest Programming Language S Starting point: example language used in second-year computer architecture course
- 4:55Random test suites outperform systematic test suites of comparable size but trail behind the manual test suite.
- 9:41Breaking the rules We can inject specific errors by overwriting mark-up mediator mappings
- 11:00Encoding typing rules - The basic idea . add type annotations to rules
- 12:47Results of semantic testing
- 13:32Conclusions
related talks
05mVision: Bias in Systematic Grammar-Based Test Suite Construction Algorithms
28mLanguage Support for Property-Based Testing
38mTesting with two failure seeking missiles: fuzzing and property based testing
19mCovering All the Bases: Type-Based Verification of Test Input Generators
26mThe missing piece of testing
43m(Programs), Proofs and Refutations (and Tests and Mutants)
47mTDD & Generative AI: A Perfect Pairing
42mSurprise yourself with Property Based Testing