Evolution of Slack Search

on YouTube ↗34m

" summary

Slack runs search on Apache Solr with a hybrid pipeline: an offline MapReduce/EMR indexer that emits Solr cores to S3 plus seven daily live collections layered on top, routing users into 64 stripes and per-team partitions of 120–128 shards. Language handling indexes messages into locale-specific fields (special CJK detection) so queries scan multiple analyzers. A Java search proxy (SolrJ) does query rewriting and re‑ranking, and multitenant enterprise key management forces thousands of encrypted collections, creating cluster scaling and slow-node isolation challenges.

AI-written summary · about

" chapters (16)

1. 0:00Introduction to Slack Search
2. 1:07Slack Architecture Overview
3. 2:58User-Specific Search Results
4. 4:42Search Indexing Infrastructure
5. 6:10Offline Index Building Process
6. 9:05Composite Dimensions Job
7. 10:07Solar Input Documents
8. 11:10Live Collections and Stripes
9. 12:35Advantages of Offline Indexing
10. 13:43Enterprise Key Management Impact
11. 15:10Internationalization and Localization
12. 19:33Searching Across Language Fields
13. 23:29Query Architecture Overview
14. 24:54Search Proxy and Query Rewriting
15. 27:28Proxy Layer Challenges
16. 27:55Q&A Session

" related talks

• 40mSmarter Search with Spark, Solr and Machine LearningGrant Ingersoll · Data By the Bay 2016
• 30mUsing Solr unconventionally to serve 26bn+ documentsRichard Goodman · Berlin Buzzwords 2022
• 46mSearch in the Cloud: separation of compute and storageIlan Ginzburg · Berlin Buzzwords 2024
• 46mDiscovery & Analytics with Hadoop, Mahout & SolrGrant Ingersoll · Berlin Buzzwords 2012
• 22mSmarter Search With Spark-SolrJake Mannix · MLconf 2016 Seattle
• 38mEvolution of Uber's Search PlatformYupeng Fu · Berlin Buzzwords 2025
• 34mArchitecting Solr indexing pipelines in Google Cloud PlatformShubhro Jyoti Roy · Berlin Buzzwords 2022
• 25mApache Lucene: From Text Indexing to Artificial IntelligenceLucian Precup · Devoxx France 2024