Bridging Classic ML Pipelines with the World of LLMs
summary
Claims that standard MLOps patterns map directly onto LLM workflows and that a DAG-as-code approach is sufficient to unify them. Demonstrates the open source microframework Hamilton as a one-function-per-node system for feature engineering, RAG, embedding customization and model training, shows swapping batch/online logic with config_do_when, notes integrations with Spark Ray Dask and potential compilation to Airflow, and reports faster iteration plus retained observability and testing benefits.
AI-written summary · about
chapters (14)
- 0:00Bridging ML pipelines with LLMs
- 1:07Classic ML vs LLMs convergence
- 2:30Inference pipelines comparison
- 5:59Training and fine-tuning in LLMs
- 7:46ML pipeline engineering challenges
- 9:53DAGs for pipeline complexity
- 13:28Hamilton for unified pipelines
- 15:16Hamilton models DAGs
- 19:26Feature engineering with Hamilton
- 21:34LLM pipelines with Hamilton
- 23:39Combining ML and LLMs
- 24:42Hamilton pipeline benefits
- 25:45Summary and next steps
- 26:50Q&A on Hamilton
related talks
33mBuilding Resilient (ML) Pipelines for MLOps
36mLLMOps in Practice- Building Secure, Governed Pipelines 4 LLMs
29mBuilding the Bridge Between DevOps and MLOps in a GenAI world
10mML-based Log Analysis for Faster Debugging
53mExperience with an end-to-end MLOps pipeline: Practical insights and 3D architectural perspectives
33mDesigning LLM Applications for Massive Datasets
43mLarge Language Models for Devs: from zero to your first LLM application
22mIntegrating LLMs with Traditional Data Analysis