Estimating Security Risk Through Repository Mining
summary
Compares OpenSSF Scorecard scores to bugs found by running Scan-build, Facebook Infer and an experimental binary tool across ~2,000 C/C++ GitHub repos (>=400 stars) with a monthly GitHub Actions pipeline; finds Scorecard has a statistically significant but practically negligible correlation (very low R²), while number and percent of cognitively complex functions explain much more variance; removing 41 outliers raises R² to ~0.37; all data, scripts and results are published on GitHub.
AI-written summary · about
chapters (23)
- 0:00Security risk estimation intro
- 1:06Motivation and problem statement
- 2:52OpenSSF Scorecard as risk predictor
- 4:59Experimental design and tooling
- 7:07Data collection challenges
- 9:18Initial results and correlations
- 10:43Revisiting OpenSSF Scorecard metrics
- 12:28Metadata and bug correlation attempts
- 13:52Cognitive complexity as bug predictor
- 16:01Combining metrics for regression models
- 18:27Identifying and filtering outliers
- 19:52Cognitive complexity and bug distribution
- 20:56Limitations and validity threats
- 22:43Risk estimation and tool trustworthiness
- 24:31Recommendations for security tooling
- 25:36Q&A
- 25:56Selection bias in sampling
- 27:39OpenSSF Scorecard improvements
- 29:02Bug definition and categorization
- 31:06Cognitive complexity paper reference
- 32:08Outlier repository analysis
- 33:52OSS Fuzz integration and false positives
- 35:18Final remarks and resources
related talks
42mAssessing the Risk of Open-source Components Using OpenSSF's Scorecard
31mScorecard at Scale: Old and New Possibilities for Lifting Security on All Repositories
27mAn Open Book Test: Securing Open Source Software With OpenSSF’s Scorecard
25mHow do you trust your open source software?
50mHow Do You Trust Open Source Software?
27mHow Do You Trust Your Open Source Software?
16mFortify Your Code: Secure Your Supply Chain with Scorecard
49mQuantifying the Probability of Flaws in Open Source