Flying High with Data Engineering in Microsoft Fabric

on YouTube ↗44m

" summary

Demonstrations show Microsoft Fabric using notebooks (PySpark/Scala/Spark SQL with magic commands) plus Lakehouse storage on ADLS Gen2 and a SQL analytics endpoint per workspace; a live demo reads 76 million rows and warm Spark sessions start under 5 seconds usually. Environments let teams upload custom libraries (pip/Maven), set cluster size and Spark properties, and create high‑concurrency pools. Copilot for data engineering sends schema/column metadata (not raw data) to Azure OpenAI with tenant geo settings; F64 Copilot compute costs around $8k/month.

AI-written summary · about

" chapters (9)

1. 0:00Data engineering in Fabric
2. 3:54Key artifacts in Fabric
3. 7:11Notebook and Spark code
4. 12:00Lakehouse and SQL storage
5. 19:13SQL analytics endpoint
6. 24:57Fabric’s data co-pilot
7. 34:02VS Code integration
8. 37:40Spark session environments
9. 41:32Q&A

" related talks

• 47mData analytics for the era of AIPatrick Baumgartner, Amir Netz, Swetha Mannepalli, Miwa Monji, Arun Ulagaratchagan, Justyna Lucznik, Ron Chang, Thasmika Gokal, Nimrod Shalit, Nandini Srinivasan · Microsoft Build 2023
• 50mMicrosoft Fabric Data Factory Q&AWee Hyong Tok, Olivia Shea, Nellie Gustafsson, Justyna Lucznik, Shawndell Elfring, Miquella de Boer, Chris Finlan, Cristin Ford, Mark Kromer, Martin Lee, Piero Morano · Microsoft Build 2023
• 31mUsing Spark to accelerate your lakehouse architecture with FabricJustyna Lucznik · Microsoft Build 2023
• 27mModernize your data integration to enable petabyte scale analytics with Microsoft FabricWee Hyong Tok, Shireen Bahadur · Microsoft Build 2023
• 43mAnalyzing Big Data in Azure cloud using Spark and DatabricksMaria Zakourdaev · SQLBits 2023
• 46mMicrosoft Fabric Synapse data warehouse Q&AOlivia Shea, Joanna Podgoetsky, Priya Sathy, Shawndell Elfring, Bogdan Crivat, Steve Howard, Faisal Mohamood · Microsoft Build 2023
• 27mEmpowering Innovation through DevOps & AI IntegrationHossam Gaber · DevOpsDays Cairo 2025
• 19mAccelerate your data potential with FabricSwetha Mannepalli · Microsoft Build 2023