Evolution of Uber's Search Platform
summary
Search at Uber moves from Elasticsearch to a custom Lucene-based engine (called Sea) with a live concurrent index plus snapshot and batch base-index rebuilt by Spark (twice weekly), driven by streaming ingestion from Kafka and gRPC for RPC. The platform supports tens of millions of rides per day and 1M+ restaurants, needs per-second freshness for matching, and implements parent‑child joins, virtual shards and an HNSW vector port (months of engineering). The in-house system works for freshness but adds operational cost, so Uber joins and contributes features like Kafka pull ingestion and OpenSearch 3.0 contribution while pushing a cloud‑native, etcd-style cluster state design.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 3:07Search use cases at Uber
- 8:07Early Elasticsearch challenges
- 15:33Building Uber’s search engine
- 26:45Transition to open source
- 32:48Cloud-native OpenSearch vision
- 35:20Q&A session
related talks
31mUber’s Innovations in OpenSearch for a Cloud Native, Serverles...
43mDeep Dive Into OpenSearch Pull-Based Ingestion at Uber
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...
41mHow Uber Audits Real time Infrastructure of Trillion+ Messages
34mEvolution of Slack Search
35mHighly Available Search at Shopify
01h 48mInformation Retrieval from the Ground Up
30mSearch on Steroids — A Story of Semantics