Improving Coverage Analysis
summary
Claim: test coverage reports give a false positive. Tools examined: Ruby's Coverage API, SimpleCov, and a newly released gem minitest-coverage that records a baseline of implementation, loads all code, and measures coverage per class under test. A client example drops from 83% to 51% when baseline adds unloaded files, then rises to 62% after fixing naming; the gem helps expose macro, micro, and omission errors but admits mapping heuristics and parallel/edge cases remain unresolved.
AI-written summary · about
chapters (35)
- 0:00Intro
- 0:16Improving Coverage Analysis
- 2:23Introduction
- 2:33Huge/Messy Implementations
- 3:50Without Tests?
- 4:00How to Improve Tests?
- 4:11What is Code Coverage Analysis?
- 4:28The Measure of the Amount of the Code that the Tests execute
- 5:17Function Coverage
- 6:53Parameter Edge Case Coverage
- 7:20Path Coverage
- 7:38State Coverage
- 8:09Because Fuck Scott Adams
- 8:27Leading to Gaming the System
- 8:47Proof of Good Tests
- 9:38Red Green Refactor
- 9:51Natural Fix
- 10:08Nobody Knows About It
- 11:02Hooks in VM
- 11:21tests run code
- 12:46Types of Errors
- 12:56False Positive
- 13:47Numerator Errors
- 14:03": Error by Omission
- 14:21Denominator Error
- 14:36How Coverage Sucks
- 16:50New Strategy
- 17:07Baseline?
- 17:56Map Test Class to Implementation
- 18:42Biased Towards False Negatives
- 19:04+ Baseline: 51%
- 19:15Fixed Type 2s: 62%
- 20:20Hook into simplecov's report generator
- 20:46Better error handling: type 2 errors on name mismatches
- 21:52Need Tools? Need Help? I'm Available!
related talks
03mTest coverage
43mAn introduction and future of Ruby coverage library
07mHow code coverage can be used and abused to guide testing.
35mRisk Based Code Coverage Analysis
46mCode coverage MythBusters
48mTest Coverage Myth Busted: Test-to-Code Mapping and Test Impact Analytics
24mWhat it Means to Have Good Test Covearage...
51mDebunking Myths Around Test Coverage