Quest for Adequate Autotest Coverage
summary
The talk describes a testing crisis where thousands of brittle UI tests give only ~50% code coverage, test runs take four hours, and flakiness is ~16–20%; the target is a 20-minute test run. The team built a three-level test framework plus custom rerun tools (costing about $7 per change) and experimented with a feature-test model that reduces UI tests by moving checks into unit and integration layers to prevent manual regressions, though this alone did not automatically increase measured coverage.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:48Automation challenges
- 3:14Legacy code issues
- 5:19Test strategy goals
- 8:12Coverage vs speed
- 10:17Feature testing rules
- 13:12App demo and tests
- 17:54Experiment results
- 18:56New approach issues
- 20:25Quality improvement plans
- 21:10Q&A
related talks
26mKeeping Your Tests Lean
21mLowering End to End Testing Barriers
56mThe End Game
38mScaling Automated Testing & Architecture from Startup to Enterprise
37mUnderstanding the successes and the pain points of different testing strategies
47mEradicate Flaky Tests
50mAsaf Shochet Avida, Eitan Noy: Why My Amazing Test Automation Project Crumbled
52mHow to win with automation and influence people