Building a Cutting Edge Search Engine in Go
summary
searchcode.com indexes 75 billion lines in 40 million projects using Go and a custom index that combines bloom filters and trigrams. code turns into trigrams, each document maps to a sized bloom filter shard (sharded by document length, ~30% bit density), packed into uint64 blocks of 64 docs; queries hash trigrams twice (two hashes gives best false‑positive rate) and logical‑AND rows to skip large ranges. result: single‑machine end‑to‑end searches drop from tens of seconds to about 50 ms, with compact memory and simple for‑loop scanning.
AI-written summary · about
chapters (12)
- 0:00The problem with search
- 2:55Designing a custom index
- 4:48Trigrams and index basics
- 7:18Postings lists and limits
- 9:29Introduction to Bloom filters
- 12:21Bloom filter limitations
- 15:22Bit-sliced signatures
- 17:07Optimizing memory usage
- 18:44Document length sharding
- 20:02Frequency conscious tuning
- 21:55Concurrency and search loop
- 24:46Results and conclusion
