Smarter Search with Spark, Solr and Machine Learning
summary
Shows how Apache Spark and Solr are glued to build ranking‑centric search: live Spark‑Solr integration (open source on GitHub), a commercial Fusion layer, and an example corpus called 20 newsgroups on steroids with ~100 sources. Claims: Solr supports SQL-like queries, faceting, spatial queries and scales (Bloomberg ~300 billion documents cited); Spark runs iterative ML (ALS, LDA, k‑means, word2vec) and tokenization via Lucene analyzers. Implements clickstream recommenders, multi‑armed bandit experiments and log reindexing; advises measuring, using autosuggest and starting simple.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:30Setting the stage
- 2:00What is search
- 6:05Distributed Computability
- 15:30Fusion
- 16:58Machine Learning
- 21:35Time Series Data
- 23:08Relevance
- 26:02Other common traps
- 33:17Questions
related talks
22mSmarter Search With Spark-Solr
46mDiscovery & Analytics with Hadoop, Mahout & Solr
34mEvolution of Slack Search
01h 22mBoot your search with Spring
35mThings I Learned the Hard Way Building a Search Engine
21mSmart Autocompl ...
50mApache Spark vs Machine Learning
01h 35mDeep Dive into the Big Data Landscape