Bringing Your Data Pipeline into The Machine Learning Era
summary
Shows a concrete pipeline that reads distributed data running on Apache Mesos and feeds it into TensorFlow running on Kubernetes via Kubeflow on Kubernetes. Uses Mesos-hosted HDFS and Kafka with Flink for streaming and Spark for batch cleansing, launches a Spark job and a Jupyter notebook, trains a model that writes checkpoints to HDFS, and monitors with Jupyter notebooks and TensorBoard. Demonstration runs on AWS, submits jobs successfully, and outlines model serving and routing with HDFS + Kafka + Flink integrations.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:34Chris Gaun
- 4:30Tensorflow
- 5:20Github Stars
- 6:16Developer Power
- 6:43Data Management
- 9:09Distributed Setup
- 10:25Failures
- 11:03Pipeline overview
- 13:06What does it include
- 13:47What is magic
- 14:28Problems
- 15:01Big data market
- 16:06Data preparation
- 16:47Model management
- 17:12Recap
- 17:30Mesosphere
- 19:16Twolevel scheduling
- 21:33Data cleansing
- 22:59Cueflow
- 23:36Demo
- 25:00Spark Job
- 28:18Jupiter Hop
- 29:52Tensor Flow
related talks
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
46mDeploying scalable Machine Learning models with Kubernetes
37mA Day in the Life of a Data Scientist. Conquer ML Lifecycle on Kubernetes
46mAccelerating Machine Learning DevOps with Kubeflow
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
54mScaling Machine Learning Pipelines in Cloud
22mThe Containerization of Machine Learning: TensorFlow: Kubernetes, and Kubeflow
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)