How METR measures Long Tasks and Experienced Open Source Dev Productivity

on YouTube ↗01h 16m

" summary

METR time-horizon scores and SWE-bench improvement claim models are getting faster and tackling longer tasks, but a field RCT with 16 experienced open-source devs on mature repos shows no average speedup — videos reveal heavy variance, frequent verification and cleanup, and unreliable self-reported times. Cursor experience split (including a 30–50 hour Cursor bin) shows no clear benefit, and an unpublished greenfield hackathon finds only ~4 percentile points higher judge scores with AI allowed. Data- and deployment-metric tasks routinely fail.

AI-written summary · about

" chapters (18)

1. 0:00The Compute-Time Horizon Argument
2. 1:43Potential Constraints on AI Scaling (Power & Dollars)
3. 4:23The Problem of Eclipsing Evaluation Time
4. 6:52Meta's "J-Curve" of Developer Productivity
5. 9:12Unreliability of Self-Reported Time Estimates
6. 11:43Personal Experiences with AI Tools (Cursor) & Learning Curves
7. 14:10METR Study Deep Dive: Scatter Plots & Variance
8. 16:48The Controversy of "Conservative" Usage Estimates
9. 21:41Unpublished Hackathon Results (AI Allowed vs. Disallowed)
10. 25:28Why AI Struggles with Data Science & Messy Enterprise Data
11. 30:35Example of AI Failure on Complex Deployment Metrics
12. 38:29Quantifying Speed-Up: The Methodological Challenges
13. 46:30Future Metrics: "Watched" vs. "Unwatched" Time Horizons
14. 52:52Moving Beyond Benchmarks: "In the Wild" Transcripts
15. 56:12The "Agent Village" & Fuzzy Goal Measurement
16. 58:53The "Neurodivergent AI" Hypothesis & Interface Mismatch
17. 1:06:31Software-Only Singularity vs. Hardware Constraints
18. 1:13:53AI Applications in Chip Fabrication & Yield Improvement

" related talks

• 45mThe Definitive Answer to Measuring Developer ProductivityRoy Wasse · DevDays Europe 2024
• 02h 31mDeveloper Productivity Engineering - Extended SessionJustin Reock · Devoxx France 2023
• 43mMeasuring Developer Productivity: The Good, the Bad and the DataMarco Pierbon · XConf Europe 2024
• 27mDevelopers Expose What's Breaking Productivity in Their CompaniesYanina Ledovaya, Olga Lvova · DPE Summit 2025
• 30mOptimizing for Time: Dark MatterKarim Nakad · DPE Summit 2025
• 10mAchieving engineering velocity by optimising developer experienceBen L. · LeadingEng London 2024
• 13mDeveloper Satisfaction = Better Code | Here's WhyAnjali Gugle · DPE Summit 2025
• 46mMeasuring the Impact of AI on Developer ProductivityPavel Avgustinov, Payam Shodjai · DPE Summit 2025