Machine Learning Data Pipelines
summary
Pipeline uses Kafka as the data backplane with Kafka Streams calling a dockerized TensorFlow Serving model; the Reuters demo trains on a 10,000-word input vocabulary, reduces to 64-node hidden layers and outputs 46 classes. Model serializes, uploads to a container registry, deploys on Kubernetes, and the stream consumer posts JSON to TF Serving then writes classifications back to Kafka. Demo classifications work; validation accuracy plateaus around ~80% while train hits ~95%.
AI-written summary · about
chapters (11)
- 0:00Introduction to ML pipelines
- 6:46Roles in ML pipelines
- 11:47Architecture with Kafka
- 18:33Demo setup
- 20:20Jupyter notebook walkthrough
- 33:43Model training and evaluation
- 40:08Loading data into Redis
- 44:23Dockerizing the model
- 48:45Deploying to Kubernetes
- 51:16Kafka streaming demo
- 55:16Conclusion and Q&A
related talks
52mMachine Learning Data Pipelines
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
42mHow to Leverage the Apache Kafka Ecosystem to...
22mThe Containerization of Machine Learning: TensorFlow: Kubernetes, and Kubeflow
46mAccelerating Machine Learning DevOps with Kubeflow
42mDeep learning beyond the learning
33mDeploying and Scaling Spark ML and Tensorflow AI Models
54mScaling Machine Learning Pipelines in Cloud