"Tests as Classifiers"
summary
Proposes treating tests as binary classifiers that label a code change buggy or not buggy and evaluates suites with precision and recall combined into F and F‑beta scores. Recommends concrete measurements: run main repeatedly to find flaky tests, track test edits and whether reported bugs produced new tests, and use mutation testing plus CI/tracing to estimate missing alarms. Uses these numbers as a lagging guide to prioritize fixing flakes, tightening assertions, or increasing coverage.
AI-written summary · about
chapters (31)
- 0:00Intro
- 2:13Acknowledgement of Country
- 2:25What is a classifier ?
- 2:49Classifier example: chihuahua or muffin
- 3:26Classifier failure: False alarm
- 4:05Classifier failure: Missing alarm
- 4:42Test (suites) as classifiers
- 7:07Always alarm: a test
- 7:37Never alarm: a test suite
- 8:17Why not simple classifiers?
- 10:30Balancing Precision and Recall: F score
- 11:40Balancing Precision and Recall: F beta score
- 12:27Balancing Precision and Recall: Who is beta?
- 13:39Beta: a Meaning
- 14:21Tests and the F score
- 15:07False Alarm: Flakey test
- 15:41False Alarm: Implementation test
- 16:26Missing Alarm: Non-covered code
- 16:56Missing Alarm: Loose assertions
- 17:45Estimating F score: Flakey tests
- 18:17Estimating F score: Implementation tests
- 18:59Estimating F score: Mutation testing
- 19:21Estimating F score: Check reported bugs
- 20:14Estimating F score: True alarms
- 21:15Estimating F score: Lagging indicator
- 22:00F score cautions
- 24:54Improving tests
- 25:18Improve: Flakey tests
- 25:37Summary: Goal
- 25:47Summary: Decide
- 25:58Summary: Improve
related talks
31mMeasure it or it didnt happen
15mUnderstanding and Improving Flaky Test Classification
28mTest flakiness; methods for dealing with flaky tests
48mTesting is Confidence - A Developer's Perspective
01h 24mPragmatic Not Dogmatic
51mProperty-Based Testing
42mHow Can I Trust My Test Suite?
42mEffective Developer Testing: taking it to the next level