Azure Data Factory, Azure Databricks, or Azure Synapse Analytics? When to use what.

on YouTube ↗40m

" summary

Starts from a 6M‑row parquet on ADLS Gen2 and compares ingestion, transformation, cost and runtime across the three Azure options. Azure Data Factory is a GUI ETL with 90 built-in data sources and an integration runtime; data flows use Spark under the hood but offer limited flexibility and no caching. Azure Databricks runs notebooks (Python/Scala/Java/SQL), caches results and in the presented benchmark is fastest and cheapest. Azure Synapse bundles ADF‑like pipelines, Spark and SQL pools and favors tight Azure integration.

AI-written summary · about

" chapters (41)

1. 0:00Intro
2. 0:43Agenda
3. 1:54Household announcements
4. 4:33Who is Alice
5. 5:44What does Alice need
6. 6:19What does the data look like
7. 6:40What do we know
8. 7:12What do you think
9. 7:44What is Azure Data Factory
10. 8:58Azure Data Factory Canvas
11. 9:37Why you should not use Azure Data Factory
12. 11:06What is Databricks
13. 11:43Who can use Databricks
14. 12:14Why use Databricks
15. 13:12Notebooks
16. 13:53Technical Knowledge
17. 16:58Synapse Analytics
18. 17:46Multiple disciplines
19. 18:14Comprehensive platform
20. 19:14Languages
21. 19:50Notebook
22. 21:34Error in Databricks
23. 22:27Error in Synapse Analytics
24. 24:13Lets Compare
25. 25:50What should Alice choose
26. 26:10Comparing tools
27. 26:55Comparing Spark clusters
28. 27:44Optimized Spark
29. 28:48Implementation Costs
30. 29:21Performance
31. 29:43Performance comparison
32. 30:53Cost comparison
33. 31:20Conclusion
34. 32:29Benchmarks
35. 32:43Summary
36. 34:21Maturity vs integration
37. 35:23Which should you choose
38. 35:47Its not that easy
39. 36:49When to use Databricks
40. 37:25When to use Synapse
41. 38:46Chart to make a decision

" related talks

• 55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?Dinesh Kumar · Global Azure Virtual 2021
• 20mChoosing between Microsoft Fabric, Azure Synapse Analytics and Azure Data FactoryCathrine Wilhelmsen · SQLBits 2024
• 01h 00mBuilding cloud-native data pipelines using Azure Data FactoryAbhishek Narain · AzConf 2020
• 50mAzure Data Factory patterns and best practicesShirley Wang · SQLBits 2020
• 51mIntroduction to Azure Data PlatformJavier Villegas · AzConf 2020
• 53mAzure Synapse Analytics: Evolving Azure SQL Data WarehouseMark Pryce-Maher, Artur Vieira · SQLBits 2020
• 54mAn introduction to Azure Synapse Analytics!Mahesh Balija · SQLBits 2020
• 49mAzure Synapse AnalyticsJames Serra · SQLBits 2020