Platformizing OpenSearch at Uber
summary
Uber runs ~100 OpenSearch clusters across thousands of nodes and ships a custom OpenSearch fork via a build tool plus an Uber wrapper to produce deployable binaries, with a tiered rollout pipeline, pre/post hooks and a pingless health checker for zero‑downtime upgrades. Ingestion uses Spark + OpenSearch‑Hadoop and gRPC endpoints for batch and Kafka/Flink for streaming, and a billion‑vector job drops from ~12 to ingest a billion vectors in 3 hours after tuning. Security uses a secure proxy L4→L7 and a role plugin that enforces Uber charter tokens, and observability comes from a metrics plugin pushing to M3DB alongside Prometheus.
AI-written summary · about
chapters (10)
- 0:00OpenSearch at Uber
- 1:26Why platformize at scale
- 4:15Custom OpenSearch binaries
- 7:27Zero-downtime deployment
- 12:08Custom security plugin
- 19:58Data ingestion strategies
- 26:45Vector index injection
- 29:21Observability for clusters
- 31:51Migrating to OpenSearch
- 33:18Q&A on security
related talks
35mInnovating at Scale: Lessons from Pegasystems' OpenSearch Journey
09mIntroducing Opensearch in existing multisource enterprise App
31mMigrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
28mRunning OpenSearch at Scale in High-Traffic Gaming Systems
39mFlexible OpenSearch Data Management With Apache Iceberg: Data...
33mHow Uber Uses Cassandra to Power Core Applications at Scale?
38mMigrating from Self-Managed ElasticSearch to OpenSearch
05mKeynote: OpenSearch: From Code to Community