Getting Started with Spark and ML
summary
Clear, runnable recipe for building a Spark ML pipeline using Spark + Jupyter + Python, JDBC to read SQL Server, SingleStore/MemSQL connector for SQL pushdown, MLflow to load a saved Spark model, and Docker Compose for quick setup. Code trains a linear regression on a random 1,000,000-row sample from TPCH, evaluates with R² on an 80/20 split, saves the model to a folder, then transforms rows with null predictions and writes them back. Warns that most time goes to cleaning data and that models amplify bias.
AI-written summary · about
chapters (9)
- 0:00Introduction and setup
- 3:32Tools for machine learning
- 13:52Getting started with Spark
- 19:33Working with Spark DataFrames
- 26:00ETL with Spark
- 30:36Machine learning basics
- 42:15Building a model
- 51:06Operationalizing models
- 54:39Q&A and conclusion
related talks
44mApache Spark and ML workflows
48mMachine Learning in Azure Databricks
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers
23mRethinking scalable machine learning with Spark ecosystem
39mMachine Learning with Scala on Spark
33mDeploying and Scaling Spark ML and Tensorflow AI Models
50mApache Spark vs Machine Learning
35mPySpark : Combining Machine Learning & Big Data