An Investigation of Metrics for the In Situ Detection of Software Expertise
summary
A logging plug-in installs into AutoCAD and records commands, start/end times, mouse and keyboard timestamps over two to four weeks (minimum eight hours) for 14 completed participants; lab work includes 500-command familiarity ratings, UI access tasks and four high-level tasks judged by two experts. Key findings: in-situ AutoCAD logging plug-in captures useful signals, command completion time correlates with expert ratings (strongest in lab; in-situ command times also correlate), while years of experience and self-rating do not, and menu access time and pause counts show weaker correlations.
AI-written summary · about
chapters (6)
- 0:00Intro
- 1:09Research Background
- 4:30Working Definition
- 6:17Classification
- 8:12Methods
- 16:04Institute Results
related talks
13mDeveloper Satisfaction = Better Code | Here's Why
01h 16mHow METR measures Long Tasks and Experienced Open Source Dev Productivity
58mWin with Metrics: How to Track and Show the Value of Your Program
01h 01mIntersection of Automation and Exploratory Testing
34mMeasuring Engineering Productivity from a Standing Start: A 320-Year-Old Bank’s Journey
05mFunctional Documentation Testing
10mAdvancing the Design of Visual Debugging Tools for Roboticists
26mFrom autocomplete to agents: AI coding assistance state of play