Building code analysis tools at Twitter
summary
Twitter builds a pipeline that extracts compiler facts into compact semanticdb files (names, denotations, compiler messages, inferred “sugars”), via a Scala compiler plugin that targets Scala 2.11 and 2.12, and stores one .semanticdb per source file. That data powers tooling: an IDE‑style go‑to, diagnostic ranges, and incremental transforms — e.g. Scala Fix removal of unused imports runs without recompiling. Work is integrated with a scalable graph index (Kite graph indexing) for multi‑language queries; the system is a technology preview but already works on large codebases.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:19State of Source
- 3:52Achieving the promise of a monorepo
- 4:21A day in the life of a core lib dev
- 5:32Avoid deprecations in the common case
- 9:53Code comprehension
- 11:01Code review
- 11:27Code evolution
- 12:26Executing the vision
- 13:48Nextgen metaprogramming library for Scala
- 17:45Old-school semantic tooling for Scala
- 20:38Second attempt (scalameta, 2014)
- 23:35Semantic database
- 25:02Live demo: semantic db for an example Scala file
- 33:49Early feedback
- 37:07Kythe: What is it?
- 37:45Kythe: A schema for a semantic graph...
- 38:24Kythe: Value proposition
- 39:38Kythe: Language-agnostic tooling?
- 41:12Kythe: Adding Scala support
- 45:50Kythe: Complexity / generality
- 48:25Future work
- 50:38Twitter is hiring
related talks
42mHow We Built Tools That Scale to Millions of Lines of Code
19mExploring Scala Tooling ecosystem
29mExploring Scala Tooling ecosystem
57mVariant analysis
50mMove fast and fix things
36mCode as Data: How data insights on legacy codebases fill the knowledge gap in modernization projects
46mSix steps from zero to IDE
33mHidden Gems of TypeScript compiler