Deep Learning with Apache Spark
summary
Shows how to run Databricks’ Deep Learning Pipelines (spark-deep-learning) on Spark 2.3+, read images with the new image reader, and do transfer learning with Inception (feature extractor) then a Spark ML classifier; a Titanic binary model reports AUC ≈ 0.74. Demonstrates distributing a local Keras .h5 via a KerasImageFileTransformer or running predictions with the deep image predictor (transfer results can mislabel rare classes — daisies hit ~95% while tulips get confused). Presents a tiny helper library Optimus that wraps these steps into one-line calls for loading, training and SQL UDF deployment.
AI-written summary · about
chapters (35)
- 0:00Introduction
- 4:10What you will learn
- 4:42Github repository
- 5:00What would you do
- 6:35What is Apache Spark
- 10:15Apache Spark Components
- 10:50RDDs
- 12:17Spark Context
- 13:14Spark Session
- 13:33Context
- 15:00Deep Learning
- 16:45Emily
- 17:54Demo
- 18:20ML Live
- 23:00Onehole Encoder
- 25:00Pipeline Code
- 27:00Two Pandas
- 30:00Part Images
- 31:00Sample Code
- 33:00Models
- 36:00Pipelines
- 37:00Deep Image Predictor
- 39:00Transfer Learning
- 40:00Kiras Image Loader
- 41:00Kiras Transformer
- 42:00ML Lib
- 43:00Control the number of tasks
- 44:00Create a sequel function
- 45:00Read the data
- 46:00Map the indexes
- 47:00Optimus
- 49:00Image predictor
- 50:00Next steps
- 51:00Trim analytics
- 52:00Python course
related talks
22mBringing Deep Learning into the Big Data Analytics
23mRethinking scalable machine learning with Spark ecosystem
40mScala and Spark in the New Deep Learning Era
35mJVM based DeepLearing on IoT data with Apache Spark
39mMachine Learning with Scala on Spark
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
34mDeep Learning at Scale
51mDeep Learning with DL4J and Apache Spark?