No Mean Feat: Upgrading a Customized Solr to Upstream Solr
summary
Bloomberg migrates a heavily patched Apache Solr fork to upstream Solr while keeping 750M documents (126 TB), 22M queries/day (median 90 ms) and 2.2M new stories/day live. The team builds a parallel Solr cluster, moves custom logic into plugins/hooks or upstreams fixes (including learning‑to‑rank), mirrors traffic and logs hashes for traffic mirroring and hash comparison, uses feature‑flagged cutover and keeps a hot rollback “red button.” Outcome: keyword queries run ~3x faster, deep paging is slower, migration spans years but succeeds without user-visible regressions.
AI-written summary · about
chapters (14)
- 0:00Introduction and background
- 2:08Why upgrading Solr is challenging
- 3:11Why this upgrade was difficult
- 4:17Pre-migration planning and version selection
- 5:40Setting up parallel Solr systems
- 6:23Moving out custom code
- 8:29Upgrading services with feature flags
- 9:11Quality verification and testing
- 10:13Fixing quality differences
- 11:19Performance testing and trade-offs
- 12:46Incremental deployment and rollout
- 13:28Post-migration cleanup and retirement
- 14:09Key takeaways and lessons learned
- 15:16Q&A session
related talks
35mLate, Over Budget, & Happy: Our Service Extraction Story
46mMoving to GraphQL
33mKeeping Throughput High on Green Saturday
37mDenormalizing Rails
31mSo Long, Hoboken: Migrating a Huge Production Codebase from Sinatra to Rails
28mMoving a Monolith from AWS EC2 to K8
41mThe 30-Month Migration
21mRails Upgrade Mastery: Best Practices for Large-Scale Enterprise Applications