Stack Overflow Meets Replication: Security Research Amid Evolving...
summary
Replication of six prior Stack Overflow studies finds that four produce significantly different conclusions when rerun on a 2022 dump, so Stack Overflow behaves like a time series not a static corpus. Repeating a C/C++ pipeline that originally reported ~15,000 vulnerable versions yields ~7,700 vulnerable latest versions and sees CWE758 rise from 482 to ~11,000 and the emergence of CWE476. Repeating another study finds insecure-post counts jump from ~12,500 to >30,000 and precision drops from 91% to 11%, prompting recommendations to version datasets, publish tooling, and analyze longitudinally.
AI-written summary · about
chapters (10)
- 0:00Stack Overflow security research
- 2:20Cross-sectional study limits
- 4:04Evolving Stack Overflow data
- 5:10Systematic review methodology
- 6:12Programming language trends
- 7:14Security edits over time
- 9:22C++ snippet weaknesses replication
- 12:57Security snippet revisions replication
- 15:07Longitudinal security research recommendations
- 16:56Conclusion and future work
related talks
30mThe Myth of the Meteoric Rise in Vulnerabilities
33m40,000 CVEs and Counting: What 60 Years of AppSec Tells Us About 2026
56mCVE Is The Worst Vulnerability Framework (Except For All The Others)
16mPerformance improvements: exposing hidden security fixes in Open Source
28mUnderstanding the Reproducibility of Crowd-reported Security Vulnerabilities
53mFlipping the Script on Application Security
01mEvolution of CVEs and Vulnerability Management
19mFlipping the script on application security