Using Solr unconventionally to serve 26bn+ documents
summary
Brandwatch rearchitects Solr on AWS to serve 26bn+ documents by splitting writes and reads: a compact write cluster snapshots indexes to S3, and multiple Kubernetes read clusters boot via the Solr Operator; a custom Argo (Rust) sidecar pulls segment byte ranges from S3, hard-links new indexes, and supports cloud repair. Added a Rust health daemon and circuit breaker, shadow-loaded real traffic for validation, and reaches near–datacenter performance while cutting provisioning time and cost.
AI-written summary · about
chapters (10)
- 0:00Introduction to Brandwatch
- 1:03Infrastructure challenges
- 4:22Project planning and design
- 7:24Implementing the architecture
- 11:00Introducing Argo
- 14:16Solr health management
- 17:58Hera Manager and tool logic
- 19:33Testing and performance
- 22:10Future plans and summary
- 23:06Q&A session
related talks
46mSearch in the Cloud: separation of compute and storage
34mEvolution of Slack Search
24mOpen heart surgery - Replacing our search index on the fly
19mYou Know, for Search. Querying 24 Billion Records in 900ms
39mOne Core to Rule Them All: Distributed Searches with Solr and Drupal
27mScalable by Design: How to Build an Elastic Cloud Infrastructure...
29mHow local businesses are rapidly achieving global scale on AWS
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...