Architecting Solr indexing pipelines in Google Cloud Platform
summary
Box moves its Solr pipeline off HBase into GCP using Bigtable, Dataproc, Pub/Sub and Cloud KMS, while keeping Solr on-prem and running dual writes to avoid data loss. It migrates ~200+ TB and an index of 100 billion docs with >10k updates/s, performs incremental MapReduce migrations with continuous sampling (1% and hashed checks) and removes optimistic concurrency during rebuilds. Outcome: rebuilds run much faster, autoscaling and DR work, and the change saves >$1M while achieving zero downtime zero data loss.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 2:55Indexing pipeline motivation
- 4:34Original indexing architecture
- 7:27Legacy infrastructure challenges
- 12:14Migration motivations
- 15:23Hybrid cloud architecture
- 19:45New GCP indexing pipeline
- 23:40Migration strategy
- 27:19Outcomes and benefits
- 31:36Future work and conclusion
related talks
20mMigrating a Large Scale Search Dataset in Production in a Highly Available...
24mOpen heart surgery - Replacing our search index on the fly
46mSearch in the Cloud: separation of compute and storage
56mLarge-Scale Distributed Processing in Google Cloud - 7 lessons...
46mHow Back Market Rebuilt Its Platform by Changing Cloud Provider
36mHow we finally migrated an eCommerce-Platform to GCP
31mGoogle Cloud Bigtable: petabyte-scale NoSQL database for scalable computation
31mReliability and Resilience in the Cloud