Crowd-Kit: A Scikit-Learn for Crowdsourced Annotations
summary
Open-source Crowd-Kit claims to be a scikit-learn–style quality control library for crowdsourced labels, implemented as pandas-friendly estimators with Scikit-learn style fit/predict API and uses pandas dataframes. It actually implements aggregation, uncertainty and agreement metrics across categorical, textual, image and pairwise data, demonstrates a pairwise text-complexity dataset that recovers ground-truth rankings, assumes examples labeled by ~3 annotators and reports worker accuracies as low as 19% up to ~50%, and the library is platform-agnostic and working.
AI-written summary · about
chapters (10)
- 0:00Crowd-Kit intro
- 2:34Why crowdsourcing needed
- 7:06Crowd-Kit as scikit-learn
- 12:25API design mirroring
- 14:30Live demo pairwise labels
- 16:18Features and models
- 17:23Conclusion and purpose
- 18:27Q&A: Contributing
- 21:15Q&A: Deep learning compat
- 23:22Q&A: Subjective results
related talks
29mAnnotating data the right way
29mGotta catch'em all recognizing sloppy work in crowdsourcing tasks
36mGold standard data: lessons from the trenches
29mLabeling Tools are Great, but What About Quality Checks?
42mClean Labels for ML
25mGotta catch'em all: recognizing sloppy work in crowdsourcing tasks
33mBulk Labelling Techniques
39mActive learning in the interactive python environment