Flyte: Production-Grade Data and Machine Learning Orchestration
summary
Platform claim: a Kubernetes-native workflow automation system (Flyte) that registers workflows to an admin, stores metadata in RDS/S3, and executes tasks as pods via Propeller. Uses Python decorators (Flytekit) and Flyte SDK / flytectl CLI, supports dynamic workflows, caching, resource specs including intratask checkpointing, Spark operator and FastAPI/unionML integrations, and 8‑bit/LORA LLM fine-tuning. Demo registers and runs a pipeline end‑to‑end (completed in ~31s); cross‑project task referencing is supported conceptually but the demo hit a reference error.
AI-written summary · about
chapters (23)
- 0:00Welcome to Flyte
- 1:11MLOps challenges
- 3:43What is Flyte
- 5:51Flyte key concepts
- 8:01Projects domains workflows
- 9:04Writing ML workflows Python
- 10:32Language independence resources
- 11:35Flyte CLI execution
- 13:24Platform engineering Flyte
- 14:07Flyte on Kubernetes
- 15:52ML checkpointing Union ML
- 17:40LLM fine-tuning ecosystem
- 18:22Flyte MLOps best practices
- 19:07Demo intro resources
- 20:12Installing Flyte
- 21:23Demo workflow registration
- 25:15Reviewing workflow results
- 26:40Kubernetes cleanup
- 27:23Cross-workflow tasks
- 29:44Q&A part one
- 31:20Flyte vs Airflow
- 34:29Q&A part two
- 38:39Closing remarks
related talks
38mFlyte: Cloud Native Machine Learning & Data Processing Platform
25mMLOps with Flyte
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
30mEmbracing Multi-Tenancy While Scaling MLOps
26mEmbracing Multi-Tenancy While Scaling MLOps
28mFuture of Data Platforms: making use-case driven...
41mEfficient Data Parallel Distributed Training with Flyte, Spark & Horovod