Machine Learning Data Pipelines
summary
A working real-time pipeline uses Kafka Streams to consume credit‑card or article messages, calls a versioned TensorFlow model served with TensorFlow Serving (Docker → Kubernetes, hosted at 8501), and writes classifications back to Kafka; Redis holds the word‑index and labels. The demo trains on the Reuters dataset with a 10,000‑word input vector, a 64‑unit hidden layer, softmax over 46 classes, ~20 epochs, saves timestamped model versions, and successfully classifies live articles (examples: acq, oilseed). KSQL DB or a UDF can replace the Java stream to invoke the model.
AI-written summary · about
chapters (16)
- 0:00Introduction to ML pipelines
- 1:06Machine learning process overview
- 8:16Roles in ML data pipelines
- 11:52Dean Wampler architecture explained
- 15:06Simplified Kafka streaming model
- 17:11Model deployment formats
- 19:22Neural networks basics
- 27:10Reuters dataset demo setup
- 33:11Training the neural network
- 36:44Saving and versioning models
- 37:48TensorFlow Serving demo
- 39:10Kafka Streams processing
- 43:01Deploying to Kubernetes
- 46:29Live demo walkthrough
- 49:18KSQL DB and UDFs
- 51:06MLOps cycle and conclusion
related talks
57mMachine Learning Data Pipelines
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
42mHow to Leverage the Apache Kafka Ecosystem to...
35mBringing Your Data Pipeline into The Machine Learning Era
42mDeep learning beyond the learning
46mAccelerating Machine Learning DevOps with Kubeflow
20mShip your Machine Learning Application #bbuzz
41mAdding Generative AI to Real Time Streaming Pipelines