End-to-End AutoML with Ludwig on Ray
summary
Ludwig on Ray runs the same Ludwig config locally or at cluster scale with one parameter on the command line, enabling out‑of‑core preprocessing via Dask on Ray, distributed SGD training via Horovod on Ray, and parallel hyperparameter search via Ray Tune. v0.4 adds Ray-based HPO and an experimental auto_train API. Proven production use at Uber, 7k+ GitHub stars and 60+ contributors, parity with TabNet results on benchmark tabular tasks, and nightly GPU Docker images; ingest from S3 or Snowflake and write transformed data as Parquet for training.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:37What are the problems today
- 2:38Trends in Modern Data Infrastructure
- 4:48Trends in Machine Learning Frameworks
- 7:21What is Ludwig?
- 8:19Ludwig Architecture
- 9:11Ludwig Task Flexibility
- 10:26Why declarative?
- 11:17Ludwig is SOTA on NLP & Tabular Tasks
- 12:21Ludwig before Ray
- 14:32Configuring Ray for Ludwig
- 15:23Running Ludwig
- 18:40A sneak peek into Predibase
related talks
28mEfficient AutoML with Ludwig, Ray, and Nodeless Kubernetes
28mEfficient Deep Learning Training with Ludwig AutoML, Ray, and Nodeless Kubernetes
33mOvercoming Distributed ML Challenges with Ray Train
28mRay on Public-Cloud k8s - experiments, lessons & best practices
29mScaling ML Pipelines for Autonomous Driving with Ray
26mAirbnb's LLM Evolution: Fine-Tuning with Ray
21mLeveraging Ray to Enhance Machine Learning Models in Finance
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT