Cross Data Center Replication in Solr - A new approach
summary
New Solr XDC module implements scalable cross‑data‑center replication by pushing mirrored update requests into Kafka and consuming them into remote clusters, supporting N‑way replication and uses Kafka as the queuing backbone. It ships a producer plugin (an update request processor) and a configurable cross-dc consumer that handles retries, prioritization, multiple topics, and pausing for snapshot/restore. Only update requests are mirrored, external versioning is required for multi‑writer scenarios, delete‑by‑query is not officially supported, and the module runs in production at Apple but is not yet released.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:40Motivation and background
- 4:21History of the solution
- 7:20Architecture and setup
- 11:45Configuration details
- 15:38Operational strategies
- 23:56Consumer features
- 29:01Administration and recovery
- 31:53Delete by query challenges
- 35:06Future work and Q&A
related talks
25mCloud native data replication strategies
04mPREVIEW: Scaling Kafka Across Multiple Data Centers
40mDistributed Data on Distributed Infrastructure
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
20mLightning Talk: Multi-Datacenter Strategies for Data Services
01h 00mParallel Asynchronous Replication b/t YDB Database Instances
43mOn the path to a massive scale SolrCloud: removing Overseer
30mBuilding domain specific databases with a distributed commit log