Recommendations at XING: From theory to production
summary
Recommender pipeline matches 650,000 job postings to 20 million users using content and collaborative signals, an outlier filter that at a 2.5 threshold removes 86% of bad recommendations while dropping 18% of good ones, and clustering to combat extreme sparsity. Engineering uses Scala + Kafka + Cassandra + Elasticsearch, about 39 data sources, 13 filters and 11 re-rankers, moving from a 12s sequential design to a parallel system with p99 around 400–500ms and a mix of real‑time and offline precomputation.
AI-written summary · about
chapters (10)
- 0:00Start
- 1:05Job Recommendations
- 4:06Key Properties of a Job Posting
- 7:33Collaborative Filtering
- 11:34Career Level Filtering
- 15:09Willingness To Change Jobs
- 16:10Profile Updates
- 20:22Push Recommendations
- 23:15Hadoop Cluster
- 26:28Data Source
related talks
19mX1 Recommendations Platform: Started from the bottom..
26mFrom Spark to Elasticsearch and Back
37mBuild a Recommender System in Apache Spark and Integrate it Using Akka
41mRecommender Systems: Crafting Effective Personalised Experiences
58mRecommender Systems – A Walkthrough
32mResurrecting a Recommendations Platform
42mDeep Learning Recommender Architecture: Master Thesis to ...
20mDataEngBytes 2023 - PER-T1-06 - Kamal Hossain