Data Labeling as Religious Experience
summary
Describes building a spam classifier for Slack invites that fails when using invite-acceptance as the label and reveals the comical result that VP of engineering was top spammer. Replaces brittle rules with Snorkel’s weak supervision and auto-generates heuristics with Snuba to produce probabilistic labels, trains a model, and finds labeling quality — not embeddings — drives performance. Recommends data programming plus dataset versioning (example: dolt) and notes BERT-style embeddings help but cannot substitute for high-quality labels.
AI-written summary · about
chapters (15)
- 0:00About Me
- 1:53How Does It Feel?
- 2:28What's Next?
- 3:04Talk Outline
- 3:49Let's Talk About My Startup
- 7:02Remembrance of Things Past
- 8:08Search Problems: A Comparison
- 10:10The Social Answer
- 11:59How Did Google Make Search Good?
- 13:40The Elephant In The Room
- 14:17Feedback Is Everything.
- 18:54How Do We Get Good Labeled Data?
- 19:24Snorkel And The Rise of Weak Supervision
- 21:14From Snorkel to Snuba
- 24:03Focus.
related talks
51mHandling the problem of hand-labeled training data with data programming...
33mWeak supervision: a new paradigm for unreliable data
30mEfficient data labelling with weak supervision
36mProgrammatically creating and managing training data
20mGreenfield vs. Brownfield Data Labeling
29mLabeling against all odds
02mDBTB INT Omar Alonso
42mClean Labels for ML