Lessons from building GitHub code search
summary
GitHub builds a new engine called Blackbird that indexes 60M+ repos and 160 TB of content and achieves reindexing from scratch in ~24 hours by combining deduplication (shard by git blob oid), a novel geometric xor filter for delta compression, dynamic shard assignment to separate compaction, and a Rust cache server that yields a 99.9% reindex cache hit and cuts servers from 312 to 130 while still supporting low‑millisecond queries.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:08Why did it take so long
- 4:51Tokenization
- 8:37Indexing
- 12:37Improvements to indexing
- 16:07Delta compression
- 19:04Symmetric difference
- 22:54Managing cluster resources
- 27:38Cache server
- 30:30Conclusion
related talks
28mPushing Git repos over the speed limit.
08mSupercharging Developer Productivity With Advanced Code Search
28mBuilding a Cutting Edge Search Engine in Go
45mHow GitHub builds GitHub on GitHub
45mHow GitHub builds GitHub with GitHub
01h 00mIndexing, or How We Made Indexes Shared and Fast
48mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
01h 01mMastering Git’s index