High Cost Tests and High Value Tests

on YouTube ↗42m

" summary

Built a small Ruby on Rails inventory app with capybara end-to-end tests and measured costs: 22 system tests versus 119 unit tests, system tests take ~13s total while unit tests run under 2s, and system tests are ~12% of tests but ~75% of run time. Larger projects hit ~1,800 tests and ~12 minute suites; CI medians around 12–16 minutes per build. Recommendation: prefer focused fast tests and reserve slow end-to-end tests.

AI-written summary · about

" chapters (41)

1. 0:00Start
2. 3:46I could Outside-In TDD
3. 5:15I'm making a decision
4. 5:35What does worth it mean?
5. 6:40How can you measure cost and value?
6. 8:09How do tests cost time?
7. 8:36The test runs , A lot
8. 9:47fixed
9. 10:12Writing the test improves code design
10. 11:18Bugs
11. 13:18Strategy Tactics
12. 15:58System tests are 12% of the tests and 75% of the run time
13. 16:10The slowest 4 tests are 40% of the run time
14. 16:57Another project
15. 17:55Balance 1 time spent
16. 18:48As you write similar tests, costs go down
17. 19:28Short term cost is not related to test type
18. 19:50Long term savings come from focus
19. 20:11Big payoff in avoiding the slowest tests
20. 21:19No individual test causes a slow suite
21. 22:32Is a slow test suite inevitable?
22. 25:06Test suites get longer as the code gets more complicated.
23. 25:48Testing is great
24. 26:10Testing is slow
25. 26:45That's hard. Let's throw CI at it.
26. 27:11Giving up causes you to lose the value of tests in improving code
27. 29:47System Test Workflow Test Unit test
28. 30:16Unit test A partial test of the workflow Or a model test Or a logic object
29. 31:12What if you don't like unit tests?
30. 31:30TDD is Dead
31. 31:44Argument against unit tests: Unit tests cos too much to write
32. 32:07True if framework doesn't handle tests
33. 33:05bad unit tests
34. 34:02A lot of unrelated assertions
35. 34:59Lack of unit tests
36. 35:13diagnose
37. 35:32fail?
38. 36:18Create the minimum amount of objects needed to trigger the failure
39. 36:33Use multiple test failures as an opportunity
40. 36:50Sometimes you can delete tests
41. 37:20Use integration tests to save development time

" related talks

• 41mHigh Cost Tests and High Value TestsNoel Rappin · RubyConf 2017
• 31mReal World Ruby TestingRein Henrichs · GoGaRuCo 2010
• 38mThere Are No TestsJeff Casimir · Rocky Mountain Ruby 2011
• 48mThere is Such a Thing as Too Much TestingJake Scruggs · RubyConf 2009
• 41mTest Automation: Do the Same but BetterIgor Khrol · TestCon Vilnius 2016
• 27mTest Effective Development - End 2 (What) End?Shai Reznik · XtremeJS 2022
• 23mAdaptive TestingBeccy Stafford, Preet Sandhu · The Lead Developer London 2015
• 47mThe Valley of SuccessTitus Fortner · Selenium Camp 2020