Deploying Large Spark Models to production #bbuzz
summary
Shows how a PySpark pipeline becomes a portable, production scorer by serializing the Spark pipeline to an MLeap bundle (JSON + protobuf), packaging that into a Docker container, and deploying it as an AWS SageMaker endpoint. Real runs: training on ~23M events/day (~140 GB initial), notebook scoring takes ~215 ms, MLeap scoring drops to ~44 ms locally and ~50 ms in production; benchmarks can reach <11 ms. Achieves ~0.88 ROC with pseudonymized GDPR-compliant features.
AI-written summary · about
chapters (12)
- 0:00The challenge of big data
- 2:07Bridging the gap to prod
- 3:29Business use case and goals
- 6:41Architecting the pipeline
- 10:23Initial model hurdles
- 12:50Introducing MLeap
- 16:08MLeap serialization flow
- 18:55Demonstrating the pipeline
- 27:36Scaling with SageMaker
- 32:30Results and validation
- 34:20Lessons and takeaways
- 38:44Q&A session
related talks
11mScaling R with Spark
31mProductizing Machine Learning over Big Data with AWS tools
33mDeploying and Scaling Spark ML and Tensorflow AI Models
27mDeveloping and Deploying Models at Scale — without Spark!
30mDeploy end to end ML pipeline using Apache spark ...
24mMaking Spark ML Models Portable Know Your Options
32mLarge-scale machine learning pipelines using Luigi,...n
26mFrom Spark to Elasticsearch and Back