“Synthesizing Input Grammars”: A Replication Study
summary
Replication reimplements the PLDI 2017 GLADE algorithm and finds its published claims don’t hold up: the provided replication packages use a custom serialized format that hides mined grammars, reimplementation yields enormous outputs (about 1500 nonterminals and ~120,000 rules with ~100 alternatives each), and measured F1 scores fall well below the paper’s 0.9+ claims. The GLADE-generated grammars still drive a fuzzer that increases coverage like AFL, and independent ARVADA work similarly observes large, low-F1 grammars, confirming the replication.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:27Why Synthesize Grammars?
- 3:02Evaluating Grammars
- 4:40Mining Input Grammars from Samples
- 6:53GLADE Grammars
- 7:56GLADE Replication Package
- 8:43Reimplementing GLADE
- 10:56What next?
- 14:01Takeaways
- 15:27Alternatives for Mining Input Languages
related talks
15mGrammar-Based Testing for Little Languages: An Experience Report with Student Compilers
59mAutomatic Grammar Repair
12mAutomatic Grammar Repair
01h 04mWhat are we thinking in the age of AI?
05mVision: Bias in Systematic Grammar-Based Test Suite Construction Algorithms
04mFTFY: Research advances in automatic bug repair
20mSemantic Reification: A New Paradigm for Random Program Generation
14mSynthesizing Contracts Correct Modulo a Test Generator