Gotta catch'em all: recognizing sloppy work in crowdsourcing tasks
summary
Trains a random forest on hand‑labeled clicks keystrokes and network logs from tens of thousands of tests to flag lazy workers with about 90% accuracy. Uses a JavaScript recorder for events, feature histograms, explicit balancing (sub/oversampling), and model persistence packaged with metadata instead of raw pickles — serialized with joblib and served as a microservice behind an HTTP API using Flask. Training runs offline weekly; production decisioning returns probabilistic scores.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:04Crowdsourcing problem
- 2:51Detecting dishonest testers
- 4:36Capturing tester behavior
- 6:23Labeling data for ML
- 7:51Imbalanced datasets
- 10:41Training classification model
- 12:48Deploying model
- 14:59Serializing models
- 17:53Model versioning
- 18:58Step-by-step limitations
- 19:39Future work
- 20:27Question and answer session
related talks
29mGotta catch'em all recognizing sloppy work in crowdsourcing tasks
39mThe practice of acquiring good labels
50mEffective test scrubbing
37mOptimal on Paper, Broken in Reality
42mDo Your Homework! Writing tests for Data Science and Stochastic Code
02h 03mFull stack AI design and development
21mFrom 0 to a crowd sourced neural network in a jiffy
38mReal Life Customer Stories with AI