Machine Learning for source code analysis
summary
Takes the entire GitHub snapshot (Dec 2016), parses source files and identifiers into a bag‑of‑words, deduplicates forks with weighted minhash LSH to remove near‑duplicates, and reduces vocabulary to about 14M unique tokens. Applies regularized topic modeling to produce 256‑topic embeddings, training on a single machine in under a day after a 64‑node preprocessing phase; resulting clusters surface concrete communities (games, Bitcoin, kernels) and show the pipeline is scalable and yields interpretable repo‑level topics.
AI-written summary · about
chapters (9)
- 0:00Intro
- 2:47Machine Learning
- 5:20Get source code files
- 6:45Parse, get names
- 7:27Build Bow model
- 8:22Filter forks
- 11:46Topic Modelings recover N Topics
- 12:52Cluster by Topics
- 19:15Questions?
related talks
37mMachine Learning on Source Code
16mMachine Learning on Source Code
28mDelivering AI on Code: Live Demo of source{d}
07mInterview
29mTaming A Monster With Git Driven Data
43mMachine Learning on Git: Hercules and His Labours
48mDeveloper Insights: ML and analytics on src/
12mKeynote: Learnings from Studying 800 Generative AI Open Source Repos