Simplify Big Data & AI on Spark and Ray with MLSQL
summary
MLSQL fuses Spark and Ray so SQL can orchestrate ETL, feature engineering and model training with embedded Python, using existing Ray clusters and standard Python libraries (scikit‑learn, TensorFlow). A demo trains a linear regression to predict monthly AWS cost, saves the model to HDFS and calls it as a registered function, proving end‑to‑end execution. Engine uses socket‑based Arrow partition transfer between Spark executors and Ray actors to avoid intermediate disk shuffles, runs Ray isolated from Spark, and exposes a pi‑java bridge for Python workers.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:122 Steps to Combine Big Data & Al
- 2:47Is Python based on Spark + Ray enough?
- 5:08MLSOL: Open Source SOL Variant for Big Data & Al
- 7:10One Stop for All Data Users
- 15:33MLSQL Summary
- 17:36Ray on Spark vs. Spark on Ray Ray DP
- 20:12The New Way MLSQL on Spark + Ray
- 21:55Why do it this way?
- 25:03Deep Dive for Data Exchange in detail
related talks
24mBuilding Complex Data Analytics Pipelines with Ray
11mScaling R with Spark
23mRethinking scalable machine learning with Spark ecosystem
35mOperating a Production-Scale Physical AI Data Platform with Ray
50mApache Spark vs Machine Learning
03mFred Reiss Interview
44mApache Spark and ML workflows
24mSmart approach in development and deployment process for ML models