Feature-Sensitive Coverage for Conformance Testing of Programming Language Implementations
summary
Presents a new coverage refinement that distinguishes identical CFG nodes by their innermost language features and by feature-call paths (with k-limiting), implemented inside a coverage-guided conformance generator for the mechanized JavaScript spec. feature-sensitive (FS) coverage and feature-call-path-sensitive (FCPS) turn node/branch TRs into feature-aware triples so fuzzing keeps semantically different tests. The tool synthesizes 237,981 tests in 50 hours, finds 143 distinct conformance bugs (42 engines, 101 transpilers), 85 confirmed and 83 newly discovered.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:23Conformance testing background
- 1:49Feature-sensitive coverage proposal
- 3:57Coverage limits in JavaScript
- 6:46Node coverage explained
- 8:13Call-path sensitivity example
- 10:21Call-path coverage details
- 11:31Tool implementation setup
- 12:15Bugs detected in JavaScript engines
- 13:41k-feature coverage effectiveness
- 14:03Call-path coverage effectiveness
- 14:45Summary of contributions
- 15:09Q&A
- 15:31Traditional metrics limitations
- 16:13Higher k value evaluation
- 16:56Applying to C compilers
- 18:44Closing remarks
related talks
14mCompatible Branch Coverage Driven Symbolic Execution for Efficient Bug Finding
20mConditional Execution of Transpiler Passes Based on Per-Script Feature Detection
52mStructure aware fuzzing
47mProperty-Based Testing
40mAdvanced Test Coverage Criteria: Specify and Measure, Cover and Unmask
26mDifferential Testing with Foundry
58mCoverage, quality, and Goodhart's Law
03mTest coverage