Flyte: Cloud Native Machine Learning & Data Processing Platform
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 17:20· Room 14AB - San Diego Convention Center, Mezzanine Level·
summary
Flyte is Kubernetes native, runs as Lyft’s orchestration fabric for ML and ETL, and executes 10M+ containers per month across ~3500 workflows. It provides a protobuf spec, Python Flytekit SDK, CRD operators, and UI; supports multi-cluster support out of the box and enforces multi tenancy, resource pooling, IAM per execution, and observability via Prometheus/Envoy. It offers caching with unique signatures so fixed steps are reused, and the demo shows composable, versioned workflows that run and reproduce successfully.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 3:25Data vs Machine Learning
- 4:05Machine Learning Models
- 5:15Flyte Workflows
- 7:43Flyte Goals
- 9:03Task
- 10:47Workflow
- 11:53Serverless
- 13:37Flyte Architecture
- 15:23Projects Domains
- 16:41Sharing
- 18:50Visibility
- 20:20Extensibility
- 21:38Platform Level Capabilities
- 22:21Demo
- 27:24Workflows
- 36:03Confusion Matrix
related talks
39mFlyte: Production-Grade Data and Machine Learning Orchestration
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
25mMLOps with Flyte
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
30mEmbracing Multi-Tenancy While Scaling MLOps
28mFuture of Data Platforms: making use-case driven...
26mEmbracing Multi-Tenancy While Scaling MLOps
41mHow Spotify Leverages Flyte To Coordinate Financial Analytics Company-Wide