Conceptualizing Disagreement in Qualitative Coding
summary
Defines disagreement as divergence versus diversity in label distributions and builds two tree-based ranking metrics—a post-traversal metric (top-down) and an n-ary tree metric (bottom-up, aka energy/analytic). Evaluates both with 50 MTurk participants and a small expert PhD sample, using tables and visualizations; finds the n-ary metric matches human judgments better. Notes concrete pain: eight coders spent 8–10 weeks and still reached only ~5% resolution, motivating automated ranking.
AI-written summary · about
chapters (10)
- 0:00Introduction to disagreement in coding
- 1:53What is disagreement in qualitative coding
- 3:22Limitations of current coding tools
- 4:49Human-centered approach to disagreement
- 6:12Top-down disagreement decomposition
- 8:42Bottom-up disagreement metric
- 10:28Comparing the two metrics
- 11:55Expert coder strategies
- 13:42Contributions and future work
- 15:30Q&A session
related talks
29mAnnotating data the right way
35mThe Code Quality Advantage How Empirical Data Shatters the Speed vs Quality Myth
13mDxHF: Providing High-Quality Human Feedback for LLM Alignment with Interactive Decomposition
09mRubrics cubed: Attempting to quantify the qualitative in the interview process
09mThe Art of Giving and Receiving Code Reviews Gracefully
36mWhy It's Worth Arguing for an Hour About Naming in Code Review
36mThe Code Quality Advantage: How Empirical Data Shatters the Speed vs Quality Myth
08mHow to Improve your Vibe Coding