Structural Language Models of Code
summary
Describes a production-ready method that models code as AST paths and trains a structural language model used for name prediction, caption generation and real-world code completion (live demo on codegen.org), achieving higher top-1/top-5 accuracy than sequence models and surprisingly beating graph neural networks. Shows GNNs fail on long-range program dependencies due to over-squashing (training accuracy already degrades at depth ~4, whereas program tasks need depth ≳8), and reports a cheap fix of dense last-layer connections while urging better solutions. AST path representations are the practical sweet spot.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:29Neural methods for code
- 9:36Code in neural networks
- 15:20Sequence models limits
- 21:43Structural models with AST paths
- 33:44Tree-based code completion
- 37:19Model approaches for code
- 41:08Graph neural networks
- 46:28GAIL and formal analysis
- 49:18Summary and future
- 50:00Q&A
related talks
43mMachine Learning for Developer Productivity
46mCode + AI: Will Robots Take Our Coding Jobs? ML Applied to Programming
26mLanguage Model Agents Enable Semi-Formal Programming
01h 50mChatGPT & LLMs: A Practical Starter's Guide to Language Chains
01h 22mBeyond Decoding: Meta-Generation Algorithms for Large Language Models
27mTeaching a Machine to Code
01h 01mC++hatGPT: How AI Tools Impact Your Life as a Systems Programmer
16mMulti-modal Program Inference: a Marriage of Pre-trained Language Models and Component-based Synthes