Open Source Warehouse Native A/B Testing.
summary
A claim that companies can run production A/B testing directly on their data warehouse using GrowthBook is demonstrated with a Docker container that connects to BigQuery, autodiscovers metrics, exposes all SQL for auditing, and includes feature flags for variation assignment. Industry experiments win about 33% of the time, so the talk warns against self-building due to one-statistics-bug risk and recommends AA tests, sample ratio checks, guarding against p‑hacking, and making experimentation a continuous part of product development. The platform is open source.
AI-written summary · about
chapters (7)
- 0:00Introduction to A/B testing
- 4:27Why we A/B test
- 7:57Growth Book platform demo
- 13:40Semmelweis reflex in testing
- 18:17Experimentation frequency
- 21:49Common testing mistakes
- 24:39Conclusion and key takeaways
related talks
13mData for product
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
32mHow open source is re-shaping the cloud data warehouse landscape
45mA/B Testing From The Ground Up
23mThe Best Data Warehouse is a Lakehouse
34mA\B Test Best Practices and Fun Case Studies from Deliveroo
26mDevOps in a Data Warehouse: Inside Out
28mRethinking Status Quo: Open Source and the Cloud Data Warehouse