Smart approach in development and deployment process for ML models
summary
Claims that 80% of time is spent on data preparation and that a smart generic data mart (ADS) cuts go‑to‑production time by enabling fast prototyping and reuse. Implements Python/PySpark pipelines, uses SQL federation versus Spark drops for DWH ↔ Data Lake transfer and Kerberos handling, and auto‑generates huge SQL via templates to avoid 500+ manual lines. Adds five data‑quality checks (missing, duplicates, relative change, statistics, data drift), uses AutoML to shortlist models, and a model manager for orchestration and fallback models.
AI-written summary · about
chapters (11)
- 0:00ML development and deployment intro
- 1:03Why models fail in production
- 2:28Smart data marts for ML
- 4:58Data warehouse vs data lake
- 8:09Domain knowledge for ML
- 10:59Five data quality checks
- 14:52Automating SQL for ML
- 15:59AutoML to speed development
- 18:06Deploying ML models
- 21:18Handling model drift
- 22:01ML docs and teamwork
related talks
41mInfrastructure for ML Applications
17mProductionizing Data Science Workloads
29mML in Production on Databricks
39mMachine Learning with Scala on Spark
32mMaking Models Work: How an optimized ML lifecycle can drive better ROI
27mA Look Inside Applied Machine Learning
38mFantastic ML apps and how to build them
28mAI-driven query plan optimization in big data processing tools