Machine Learning with Apache Beam
summary
Apache Beam shows concrete ML work not promises: Beam DataFrames runs pandas‑style transforms distributed, the Run inference transform shares model instances, auto batches and picks up live model file updates, and a demo runs BLIP and CLIP to caption and rank three images. Model evaluation uses TFMA with TF records to compare a model trained on 40% versus 80% of data, co‑group‑by‑key merges image and caption streams, and pipelines scale to arbitrary datasets depending on the chosen runner and GPU support.
AI-written summary · about
chapters (6)
- 0:00Start
- 6:40Basic Beam Pipeline
- 7:22The ML lifecycle
- 11:59Challenges of Distributed Inference
- 12:58Distributed Inference with Beam
- 34:01Online Training
related talks
19mBeam Up Your GenAI Usage: Usability, Efficiency, Reliability
39mML Inference at scale, with Apache Beam and the Flink runner: Easy as learning your 5 times tables.
52mApache Beam, a unified model for Big Data processing
38mData processing with Apache Beam
26mA Gentle Introduction to Apache Beam
27mHow to make ML teams 80% more productive.
21mPredicting Hacker News with Beam and TensorFlow
38mDistributed Processing for Machine Learning Production Pipelines