Building Advance Analytics pipelines with Azure Databricks
summary
Walks through an end-to-end Azure Databricks pipeline that ingests KDD 1999 data (~1.7M rows), uses Spark ML Pipelines with one hot encoding, scaling and Gradient Boosted Trees and Random Forests, achieves ~99% AUC on this synthetic dataset, and persists models to Blob storage. Shows batch scoring jobs and Structured Streaming via Event Hubs, mounting Blob storage in DBFS, autoscaling clusters, attaching custom libraries, and a Dockerized demo that deploys the workspace and notebooks end to end.
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:27Apache Spark a unified computing engine
- 5:00Apache Spark: APIS
- 5:55Inside a Spark Application
- 8:13Azure Databricks Managed Apache Spark platform optimized for Azure First party service
- 14:55Hidden Technical Debt in ML Systems
- 16:03Azure Integration
- 16:36Databricks Core Concepts
- 20:35Anomaly Detection - Network Intrusion KOD Cup 1999 Data
- 21:20Demo Architecture
- 26:07Estimators and Transformers
- 27:03Custom Transformers and Estimators
- 33:40Productionizing Machine Learning Workloads
- 35:29Spark Structured Streaming
- 43:55Databricks Developer Tooling
- 44:23Try the demo!
related talks
48mMachine Learning in Azure Databricks
48mBuilding an Azure Data Analytics Platform End-to-End
33mBuild your own advanced AI solutions on Azure Databricks
52mElevate your data and AI governance with Databricks Data Intelligence Platform
25mFrom Data Exploration to Modeling to Deployment
49mDelta Live Tables: Building Reliable ETL Pipelines with Azure Databricks
51mHow to get started with MLOps in Azure Databricks
24mAzure ML, Azure Kubernetes Service, Model deployment