Catching More Bugs with Fewer False Alarms
summary
Argues that combining dynamic analyses with real tests finds more real defects while cutting false alarms: Vroom isolates tests in-memory to avoid ~1.5s JVM restart overhead and yields large speedups on suites with many millisecond tests; Phosphor implements whole-JVM dynamic taint tracking; Rivulet replays and mutates existing functional tests to confirm injection bugs, producing a single false negative versus a static tool’s nine false negatives and ~100 false positives, and it uncovers six new vulnerabilities (and reproduces the Struts/Equifax bug).
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:16BUGS ARE BAD
- 2:07VULNERABILITIES ARE EVEN WORSE
- 2:43WHAT WENT WRONG?
- 3:27HIRE MORE TESTERS?
- 4:43SUPPORTING OPEN SOURCE CREATING OPEN SOURCE TOOLS FOR USE BY ALL, BUT ESPECIALLY FOR THE OPEN SOURCE COMMUNITY
- 5:11SUPPORTING OSS SUPPORTS INDUSTRY PROPRIETARY CODE
- 6:22MY WORK
- 8:55TEST INDEPENDENCE
- 10:30A TALE OF TWO TESTS
- 11:37JUNIT TEST EXECUTION
- 13:06VMVM: UNIT TEST VIRTUALIZATION
- 13:53VMVM'S REINITIALIZATION
- 15:27PERFORMANCE EVALUATION
- 17:48VMVM + ELECTRICCLOUD
- 18:52VMVM: 6 YEARS LATER ICSE 2014
- 22:11TEST ISOLATION: ANOTHER STORY (2009)
- 29:282012: DIVERSION FROM IN-VIVO TESTING
- 31:57PHOSPHOR
- 32:22IN-VIVO TESTING: NOT DEAD YET
- 34:01TAINT TRACKING FOR VULNERABILITY DETECTION
- 35:13RIVULET LEVERAGES EXISTING TESTS
- 36:24RIVULET: DYNAMIC TAINT TRACKING + TESTS
- 38:30RIVULET: BENCHMARK EVALUATION Number of Test Cases RIVULET Benchmark Suite Type True Alarm False Alarm TP FP TN EN Time
related talks
40mCatching More Bugs with Fewer False Alarms
41mPragmatic Testing
44mTest Intelligence: Finding More Bugs in Less Time with AI
01h 05mMicrotesting
50mFix The Damned Software
44mStatic analysis as an additional barrier against bugs
10mBeyond “try harder”: Effective strategies to tackle bugs
42mTesting on Thin Ice: Chipping Away at Test Unpredictability