Computing recommendations at extreme scale with Apache Flink

on YouTube ↗33m

" summary

Implementation uses Apache Flink to run three ALS variants and a blocked ALS partitioning that cuts redundant network IO and removes one shuffle by prepartitioning user and item blocks. The system factorizes matrices up to 28 billion nonzero entries (naive ALS times out past ~8 billion). Ten ALS iterations with 50 factors take about 14 hours on the base cluster and drop to 5.5 hours after doubling nodes and memory. ALS lands in the Flink ML library for direct use.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to recommendations
2. 10:25Why Apache Flink
3. 23:05Implementing ALS in Flink
4. 28:08Performance results and scaling
5. 30:16Using the Flink ML library
6. 31:42Conclusion and next steps

" related talks

• 43mTo Petascale and Beyond: Apache Flink in the CloudsGreg Hogan · Flink Forward Berlin 2016
• 47mBring Flink to Large Scale ProductionXiaowei Jiang · Flink Forward Berlin 2017
• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024
• 40mStreaming ML with FlinkMarton Balassi · Flink Forward Berlin 2016
• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 38mFaster and Furiouser: Flink DriftTed Dunning · Flink Forward Berlin 2016
• 29mBuilding Resilient AI Platforms: Financial Fraud DetectionRahul Ganti · Conf42 Platform Engineering 2025