SoK: I Have the (Developer) Power! Sample Size Estimation for...
summary
Meta-analysis of 54 papers (64 studies) extracts 467 tests, 413 variables and shows many developer-centered usable-security studies are underpowered: even the strongest study has under 50% power to detect small effects. The work builds a structured database and a searchable web prototype (downloadable SQLite) that reports test-specific effect sizes and conversions and supports Fisher’s exact, chi-squared, McNemar’s, Wilcoxon rank-sum/signed-rank and t-tests, plus reporting recommendations and reproduction guidance.
AI-written summary · about
chapters (8)
- 0:00SoK paper introduction
- 1:12Developer studies power status
- 2:37Power analysis basics
- 4:23Challenges in effect sizes
- 6:32PowerDB database
- 7:35Using PowerDB tool
- 9:44PowerDB for end-user studies
- 10:25Feedback on effect sizes
related talks
23mSecurity Outgunned: Measuring Software Defined Attack Surface
30mTopics of Interest: Effective Usage Analysis: The Shortest Path Between a Developer ...
57mUsing developer-centric data to predict, prioritize, and improve App Security Outcomes
50mFix The Damned Software
43mDo Certain Types Of Developers Or Teams Write More Secure Code?
54mUsing developer-centric data to improve Application Security Outcomes
43mMeasuring Developer Productivity: The Good, the Bad and the Data
45mThe Definitive Answer to Measuring Developer Productivity