Gotta catch'em all recognizing sloppy work in crowdsourcing tasks
summary
Claim: a machine learning classifier reliably flags lazy crowdsourced QA so fewer repeats and cheaper checks. Uses captured mouse and keystroke traces plus network events, labeled by developers (roughly 10 labels/week) with an imbalanced training set ~100:1 good to bad. Model is a Random Forest classifier per step plus a job-level aggregator, persisted as a scikit learn estimator (joblib serialization) and served behind a simple HTTP microservice; works well but needs careful labeling and feature/version management.
AI-written summary · about
chapters (18)
- 0:00Crowdsourcing challenges intro
- 1:46Data collection context
- 3:09Worker mistake types
- 5:18Worker activity capture
- 7:35Labeling for model training
- 9:23Unbalanced training data
- 10:45Lazy-work classifier
- 12:10Model as microservice
- 14:17Model versioning challenges
- 16:46Empathy in workflows
- 18:49Training workers
- 20:15Worker community feedback
- 21:43Limitations and future
- 23:13Open-source potential
- 24:44Worker reputation fairness
- 26:16Engaging worker communities
- 27:23Payment strategies
- 28:06Q&A
related talks
25mGotta catch'em all: recognizing sloppy work in crowdsourcing tasks
39mThe practice of acquiring good labels
29mLabeling Tools are Great, but What About Quality Checks?
38mBuilding robust machine learning systems
30mEfficient data labelling with weak supervision
32mTesting Machine Learning Systems: What it is and why you should care
38mReal Life Customer Stories with AI
36mMachine Learning Sucks