Apache Spark and machine learning: dreams and realities
summary
A practical look at using Apache Spark ML for clustering, showing a demo that clusters nine pumpkins with Spark ML and revealing real-world issues: dependence on VectorAssembler, untyped DataFrames, and KMeans’ sensitivity to outliers (a 9,122 kg pumpkin skewing a 2.5 kg mean). The talk compares mean-based KMeans to median/mode alternatives, implements a custom K-medoids (medoid-style) and a weighted-iteration anomaly detection inside Spark, and reports improved anomaly discovery alongside performance and scaling tradeoffs.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:08ML data engineer role
- 5:15Clustering applications
- 11:58K-means demo
- 17:14Data prep for ML
- 23:11Algorithm limits
- 28:08K-medoids alternatives
- 33:45Custom Spark ML algorithms
- 40:06ML challenges
- 45:26Conclusion
related talks
50mApache Spark vs Machine Learning
17mMachine Learning at the Speed of Light with Spark
39mMachine Learning with Scala on Spark
19mReal Time Machine Learning Visualization with Spark
35mPySpark : Combining Machine Learning & Big Data
44mData Science at Scale with Apache Spark
44mApache Spark and ML workflows
30mAlice and Mad Hatter....