Making Spark ML Models Portable Know Your Options
summary
Compares ways to export Spark ML pipelines: use Spark runtime, DataBricks/local jars, the PFA standard, or build a custom runtime. Describes a custom library that wraps Spark stages, uses Java reflection to call internal methods, and scores 6 million rows at 0.03 ms per record, roughly two orders of magnitude faster than Spark for per-row scoring. Notes PFA has useful control flow but is buggy and lightly maintained and mentions Microsoft MMLSpark serving to hit executors directly.
AI-written summary · about
chapters (11)
- 0:00Start
- 1:33Learning Phase
- 7:06Train a Model on Spark
- 7:18Do You Really Need Spark
- 9:10Real Time Scoring
- 9:36Prerequisites
- 15:29Pmml
- 17:14Case Study on the Transmogrified Library
- 19:36What Have We Learned
- 21:06Decision Diagram
- 22:23Why Would You Even Stay with Spark
related talks
48mDeploying Machine Learning Models in Practice
50mApache Spark vs Machine Learning
39mMachine Learning with Scala on Spark
44mApache Spark and ML workflows
43mDeploying Large Spark Models to production #bbuzz
23mRethinking scalable machine learning with Spark ecosystem
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
34mOpen Standards For Machine Learning Model Deployment