Deep Dive Into OpenSearch Pull-Based Ingestion at Uber
summary
Uber contributes a pull-based ingestion engine into OpenSearch 3.0 that actively reads from Kafka and Kinesis via plugins, stores per-document offsets and a batch start pointer in the Lucene index, and maps each OpenSearch shard to a one-to-one shard-to-partition binding. It supports full-document upserts with external versioning, segment replication and remote store recovery, and offers 35–40k msgs/sec per shard with multiple writers today while the single-threaded consumer is the bottleneck; error policies are drop or block and roadmap items include partial updates and concurrent consumers.
AI-written summary · about
chapters (14)
- 0:00Pull-based ingestion intro
- 1:23Real-time data importance
- 6:06Scalable search requirements
- 7:12Uber’s active-active search
- 10:24Uber’s OpenSearch contributions
- 12:31Pull-based ingestion design
- 16:23Kafka to OpenSearch flow
- 20:13Blocking queue & multi-writer
- 23:00Shard recovery with pointers
- 26:36Versioning and error handling
- 28:44Scale of pull-based ingestion
- 30:10Future features roadmap
- 31:57Cloud-native & multi-writer
- 32:42Q&A
related talks
31mUber’s Innovations in OpenSearch for a Cloud Native, Serverles...
38mEvolution of Uber's Search Platform
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
25mPersistent and performant ingestion using Data Prepper
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
16mSponsored Keynote: OpenSearch at Atlassian Scale
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch