Split Testing for Product Discovery

on YouTube ↗35m

" summary

Split tests move decisions from opinion to measurements: small cosmetic tweaks yield typical lifts of 1–3% conversion, anecdotal claims show up to 200–300% but can fail; feature experiments that are actually built include a speed date feature and a “surprise me” autofill which increase posting and messages, a forced-post modal that surprisingly raises engagement, a weekend price tripling that loses money, and regionally free markets that do not grow. Implementation uses hashing for stable buckets, binomial significance, cohort dashboards and daily experiment emails.

AI-written summary · about

" chapters (34)

1. 0:00Intro
2. 0:17SPLIT TESTING FOR PRODUCT DISCOVERY
3. 1:53WEB BUSINESS
4. 2:41LICENSE TO PRINT MONEY
5. 3:24GOING NOWHERE FAST
6. 4:57BASIC A/B TESTING
7. 6:28SKEPTICAL? YOU MIGHT BE SURPRISED.
8. 8:14WHEN TO IMPLEMENT?
9. 8:39LOW RISK, LOW COST, POSSIBLY HIGH REWARD
10. 9:01DEEPER INTEGRATION
11. 9:28WHAT FEATURES DO YOUR CUSTOMERS WANT?
12. 10:12WHAT ARE YOUR CUSTOMERS WILLING TO PAY FOR?
13. 11:16ARE YOU OVER-SATURATING YOUR USERS OR ARE YOU MISSING POSSIBLE SALES OPPORTUNITIES?
14. 12:32TEST EVERY NEW USER-FACING FEATURE
15. 12:46EXAMPLES
16. 17:16SUFFICIENT DATA OBVIATES BEST PRACTICES
17. 18:18YOUR WHOLE BUSINESS IS A CONVERSION
18. 18:56TECHNICAL IMPLEMENTATION
19. 21:39USER PARTITIONING
20. 21:50START/END EXPERIMENTS FROM ADMIN DASHBOARD
21. 23:10TECHNICAL DEBT
22. 23:45AUTOMATED TESTING IS ESSENTIAL
23. 24:30TEST IN ISOLATION
24. 25:00KILL DEAD CODE IMMEDIATELY
25. 25:30REVERTABILITY
26. 26:02EMOTIONALLY
27. 26:30MEASURING SUCCESS
28. 26:45BINOMIAL DISTRIBUTION
29. 27:16COHORT METRICS
30. 28:57WHAT IF THERE'S A TIE?
31. 30:15UNEXPECTED BENEFITS
32. 31:55A DEFENSE AGAINST CONFLICTING CUSTOMER FEEDBACK
33. 33:14RECAP
34. 33:55THERE ARE TOOLS AT YOUR DISPOSAL

" related talks

• 36mStep 1) Hack, Step 2) ?, Step 3) ProfitBradley Herman · RailsConf 2016
• 34m SaaS Lessons LearnedTom Rossi · Friendly.rb 2024
• 26mOne does not simply... rebuild a productPrakriti Mateti · LDX3 London 2025
• 23mDeveloping multiple personalities to bootstrap your first productGarren Smith · Rubyfuza 2013
• 35mSurviving Black Friday: Tales from an e-Commerce EngineerAaron Suggs · PAYMENTSfn 2019
• 40mMake Money and Enjoy Freedom Creating a Software BusinessRyan MacInnes · Rails Pacific 2016
• 31mTesting in ProductionAja Hammerly · RailsConf 2018
• 38mSmall Details, Big ImpactYehuda Katz, Liz Baillie · RailsConf 2016