The Python Data Ecosystem: Navigating a fragmented landscape
summary
Claims a concrete orchestration: a strongly typed Python DSL that registers containerized tasks and workflows, builds images for you, and runs elastic multi‑GPU training. Demonstrations show a 1B‑parameter model fine‑tuned in ~13 minutes for under $4 and models published to Hugging Face. Integrations include Spark, Ray, Hugging Face and S3, plus caching, versioning and compile‑time checks; the system is open source and already runs in production for multi‑team workloads.
AI-written summary · about
chapters (15)
- 0:00Python data ecosystem intro
- 2:09Core problems in Python data
- 5:26First principles for ML workflows
- 8:56Declarative ML infrastructure
- 11:06Ideal data orchestration properties
- 12:49What is Flight?
- 14:36Flight’s tasks and workflows
- 20:16Flight workflow demo
- 27:38Flight dynamic tasks demo
- 29:48Flight for LLM fine-tuning
- 35:52Flight in modern pipelines
- 37:15Flight production benefits
- 39:01Flight vs Airflow
- 41:31Flight resource optimization
- 44:21Flight hyperparameter optimization
related talks
37mThe Python Ecosystem for Data Science: A Guided Tour
39mHow Python Drives Modern Data Engineering
40mScaling Python - Bank Edition
22mProductive Machine Learning and Deep Learning Projects
24mHuman Friendly, Production Ready Data Science Stack
37mPanel Discussion
38mState of Data Science & Machine Learning
43mKEYNOTE: Scaling Out PyData