Coverage, quality, and Goodhart's Law
summary
Measures collapse under Goodhart but coverage used well finds real problems: using GCC 15 and zcov on a borrowed binary search shows 15/24 paths covered, a 5,000-input brute force only adds one path, refactoring removes a redundant check and raises coverage to 17/18 then 18/18 with two added tests; MCDC on satellite code uncovers spec and error‑handling faults; techniques include dependency injection, allocation‑failure injection via operator new override and link‑time wrapping, and prioritizing MCDC and path/edge coverage over line percentage.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:11Measuring coverage with Zcov
- 5:26Fixing poor path coverage in binary search
- 9:06Coverage as software tool
- 12:18Testing beyond coverage
- 13:21Line coverage flaws
- 15:33Block and edge coverage
- 16:59Goodhart's Law on coverage
- 19:51Coverage improves quality
- 24:45Code complexity and prime paths
- 28:01Coverage-guided programming
- 29:34Case study: Leap day detection
- 33:56Fixing leap day function with time injection
- 39:19Restructuring code for coverage
- 40:45Rust and GCC insights
- 42:11Duff's device complexity
- 44:23Exception handling challenges
- 48:04Testing exception paths
- 50:17Forcing exception paths
- 51:41Dependency injection for errors
- 54:13Final thoughts on measurement
- 55:39Link-time function wrapping
related talks
01h 23mModified Condition and Decision Coverage in Gcc: Intro, Motivation & Tooling
03mTest coverage
19mTesting Your Tests With Code Coverage
26mImproving Coverage Analysis
39mHow to Talk about Coverage
01h 40mTesting, Preconditions, Coverage and Templates in Safety-Critical C++ Code
40mAdvanced Test Coverage Criteria: Specify and Measure, Cover and Unmask
10m99% is not enough - full test coverage and why it's awesome