Hidden Biases In Data: Moving Beyond Black Box Models
summary
Built a behavior model that simulates a full user population (not an average) to reproduce cohort retention and session patterns, sampling sessions with a negative binomial (mean 4.5) and attributing an engagement parameter per user. The simulation logs every event, matches chest‑open metrics almost one‑to‑one with live data, and uncovers selection bias behind high payer retention. Implementation starts in Python, then rewrote in Julia for ~10× speed, and the framework gives causal knobs and actionable baselines for design changes.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:37Markos background
- 2:28Big Live Game
- 3:42Game in Development
- 5:41Cohort Metrics
- 6:47User Behavior
- 9:43Core Loop
- 11:22Virtual Economy
- 13:30Live Date
- 16:26Pair Retention
- 17:57Thought Experiment
- 21:39Other Benefits
- 22:18Thank You
- 22:44Audience Question
- 25:45Tools
- 26:35Results
- 27:13Why Julia
- 28:38How long did it take
- 29:18Business decision
- 30:50Top 11 user data
- 32:27Conclusion
related talks
30mDetecting Hidden Bias in Datasets Before Models Fail
41mKeynote: Personalization, Bandits, and Causal inference
41mA/B Testing with Style
47mAlgorithmic bias is everywhere (especially at Breeze) - what can we do about it?
30mBias, Explainability, and Accountability: The Data Scientist's Burden
18mThe Importance of Modeling Data Collection
15mBeyond Efficiency
10mIs Bigger Really Better? Predictive Analytics with Fine-grained Behavior Data