Making Happy Modelers: Build and Maintain...
summary
Practical recipe for running ML pipelines: use COPY from S3 into Redshift with distribution and sort keys, run ANALYZE COMPRESSION and EXPLAIN to avoid multi-day queries, and schedule jobs with Airflow DAGs with operators (AWS Athena/Redshift operators, credentials managed in Airflow). Training jobs pickle models to S3 (monthly) and prediction jobs run daily, with retrying, PagerDuty alerts, and streaming to avoid overloaded workers. Examples include identity auto-increment keys, single-transaction updates, and a real long-running query that revealed the need for explain/optimization.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:17Agenda
- 3:35Overview
- 4:48Redshift Architecture
- 8:11Using Redshift
- 11:35Redshift Limitations
- 12:58Table Example
- 14:44Dont Do This
- 16:26Airflow
- 17:54User Interface
- 19:32DAG Definition File
- 22:40What is an Operator
- 27:24Things to keep in mind
- 28:46Practical examples
- 31:21Wrapup
related talks
52mPreparing data for reporting, ML, and AI on AWS
49mOrchestrating data and ML workflows with Apache Airflow
43mData warehousing reinvented for today's needs
25mDesigning an AWS Well Architected Framework for AI/ML
34mLat Designing and Building Complex Machine Learning....
12mLeveraging AWS and Kubernetes for Scalable AI/ML Workloads in the Cloud
44mBest practices for analytics and generative AI on AWS(ANT302)
18mAutomate the boring ML stuff with pipelines