Getting started with SparkML in Microsoft Fabric

on YouTube ↗20m

" summary

Shows end-to-end SparkML work inside Microsoft Fabric notebooks: load a ~500,000-row flight dataset (day, departure time, estimated air time), assemble features with vector assembler, train a pipeline using a decision tree classifier, and evaluate on a 70/30 split yielding ~68% accuracy. Experiments and model artifacts live in Fabric (Lakehouse integration, experiments view), and an automated search via Flam ML tries many algorithms to improve results before exporting models for reporting.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to machine learning in Fabric
2. 3:59Data prep for ML
3. 8:54ML pipelines with SparkML
4. 15:13Training and evaluating
5. 17:45Improved models with FLAML
6. 19:07Summary and next steps

" related talks

• 22mModels to outcomes with end-to-end data science workflows in FabricNellie Gustafsson · Microsoft Build 2023
• 48mMachine Learning in Azure DatabricksTerry McCann · SQLBits 2020
• 55mGetting Started with Spark and MLRob Richardson · Developer Summit 2020
• 31mUsing Spark to accelerate your lakehouse architecture with FabricJustyna Lucznik · Microsoft Build 2023
• 50mApache Spark vs Machine LearningPatryk Pilarski · Confitura 2019
• 24mMaking Spark ML Models Portable Know Your OptionsMatthew Tovbin · Scale By The Bay 2018
• 39mMachine Learning with Scala on SparkJose Quesada · Scala Days Berlin 2016
• 26mTransform productivity with AI experiences in Microsoft FabricPatrick Baumgartner, Arun Ulagaratchagan · Microsoft Build 2023