Alessandro Magnani, Data Scientist, WalmartLabs
summary
Walmart catalog labels get stored with a recorded sampling probability for each item so accuracy estimates use a self‑normalized importance sampling estimator (weight = 1/pi) that produces unbiased, reuseable metrics as the product set drifts. A label store plus a sampler that minimizes variance under a labeling budget chooses where to spend extra labels; sampling is adjusted toward low‑pi regions and can incorporate model posteriors or active learning. System scales to millions of items and cuts wasted labels.
AI-written summary · about
chapters (4)
- 0:00Classification Model Performance
- 2:36Practical challenges
- 4:28A motivating example
- 11:59Evaluation framework
related talks
27mI have no clue what my model is doing...
23mMachine Learning with a Deep Dive into Automated Forecasting System
33mWeak supervision: a new paradigm for unreliable data
19mOperational Challenges in Machine Learning Model Life Cycle
01h 02mSampling for Linear Algebra, Statistics, and Optimization II
39mMallikharjuna MV of Walmart Labs at Machine Learning Developers Summit 2019
21mAWS On Air ft. Amazon SageMaker Ground Truth Plus
29mLabeling against all odds