Robust and End-to-End Cloud Native Machine Learning & Data Processing Platform
summary
Claims a production-ready open source orchestration: Flyte runs containerized, versioned tasks on Kubernetes and converts data between runtimes (PySpark, pandas, PyTorch) so users avoid glue code. The demo walks 11 incremental examples on the Palmer Penguins dataset, shows strongly typed tasks with enforced registration, runs a tiny grid search of 3 hyperparameter combos via dynamic workflows and map tasks, demonstrates caching to reuse intermediate results, and wires to Spark, Ray and SageMaker — examples execute and produce results.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:04Union AI and Flight
- 3:13ML pipeline challenges
- 9:12Flight addresses challenges
- 13:51Flight features and language support
- 20:24Tutorial setup with Penguins dataset
- 24:20Simple ML workflow in Flight
- 34:21Adding grid search
- 42:14Map tasks for parallel computations
- 48:57Flight plugins and integrations
- 59:43Type system and validation
- 1:09:07Caching for performance
- 1:15:15Flight Decks artifacts
- 1:20:35Deployment and resources
- 1:23:26Q&A
related talks
38mFlyte: Cloud Native Machine Learning & Data Processing Platform
39mFlyte: Production-Grade Data and Machine Learning Orchestration
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
25mMLOps with Flyte
41mEfficient Data Parallel Distributed Training with Flyte, Spark & Horovod
30mEmbracing Multi-Tenancy While Scaling MLOps
28mFuture of Data Platforms: making use-case driven...
51mSelf-serve Feature Engineering Platform Using Flyte and Feast