Distributed Processing for Machine Learning Production Pipelines
summary
Claims that a production ML pipeline is practical end-to-end: they implement TFX components (ExampleGen→StatisticsGen→Transform→Trainer→Evaluator→Pusher) on top of Apache Beam, run those DAGs on runners like Beam portability + Flink runner (also Dataflow and Spark), and use a metadata store so unchanged components are cached instead of re-running hours or days of work. Cross‑language transforms use an expansion service with protobuf/grpc; portability reduces SDK‑runner cost and the setup is reported running in Google and by partners (e.g., Twitter), so the approach demonstrably works.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 4:35Apache Beam
- 9:48Beam Python SDK
- 19:25Portability Framework
- 23:17Components
- 24:32Orchestration
- 26:07Metadata
- 28:15Standard Components
- 28:30Ingest Data
- 29:00Statistics
- 29:43Data Validation
- 30:38Feature Engineering
- 32:32Deep Analysis
- 35:44Key Points
- 36:09Flink
- 36:25TFX
- 36:45TFX Integrated Patterns
- 37:12Outro
related talks
38mFrom Experimentation to Products: The Production Machine Learning Journey
38mTensorFlow Extended (TFX): Machine Learning Pipelines and Model Understanding
10mDemocratizing the pipeline with Apache Beam and Dataflow
27mHow to make ML teams 80% more productive.
36mMachine Learning with Apache Beam
31mMLOps In Practice – How To Run Your Machine Learning Models In Production At Enterprise Scale
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
18mAutomate the boring ML stuff with pipelines