Building a WebScale Search Engine with a Data Driven Architecture
summary
Built a homegrown Sherlock reservation search serving Booking.com’s >1 billion bookings at ~2,000 req/s with a <50 ms goal, replacing off‑the‑shelf engines. Uses per‑shard isolation, RocksDB storing each reservation as a single compressed binary blob (2× size reduction), JNI C indexers, and streaming Kafka updates to keep replicas near real time. Indexes use Roaring bitmaps for compact fast intersections, grew from 6 to 60 indices without CPU blowup, and deployment practices (SSL, nginx, chaos experiments, client‑side monitoring) deliver reliable adoption.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:40What is Bookingcom
- 4:50The 5 phases of a product
- 6:22Phase 1 Preparation
- 12:17Documentation
- 13:47Language
- 14:27Data Structure
- 16:22Ownership
- 19:25Bitmaps
- 23:42Backup strategies
- 23:57Challenges
- 25:05Cap Theorem
- 25:21Impact Liability
- 25:57Client Monitoring
- 26:07Lazarus Clause
- 26:19Latency
- 27:49Adoption
related talks
30mData @Scale Seattle- Sundaram Narayanan
42mImproving Search @scale with efficient query experimentation
40mSmarter Search with Spark, Solr and Machine Learning
35mHighly Available Search at Shopify
20mBuilding a Personalised Home Feed #bbuzz
37mMeasuring 2.0 – Building the next generation webanalytics ...
46mCentralized logging and sleeping well at night
51mSpeed, scale, query: can NoSQL give us all three?