An Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric

on YouTube ↗48m

" summary

Claims that bringing ACID to data lakes solved scale and consistency problems by letting storage and compute cooperate, with Delta Lake open-sourced April 2019 and Databricks/Spark-based implementations on Azure (Synapse, Data Factory, Databricks, HDInsight). Kappa simplifies Lambda by unifying batch and streaming into Delta tables; Fabric (May 2023) packages OneLake, Delta tables, SQL endpoints, KQL, workspaces and event streams as a unified stack. Practical outcome: time-to-insight drops from months to days but data mesh requires heavy org change and federated governance.

AI-written summary · about

" chapters (7)

1. 0:00Introduction and evolution
2. 4:27Data lakehouse and Delta
3. 6:48Implementing Delta Lake
4. 16:12Lambda and Kappa architectures
5. 22:09Introduction to Fabric
6. 23:28Data mesh concepts
7. 30:56Practical implementation

" related talks

• 21mWhy the Lakehouse?Craig Porteous · SQLBits 2023
• 28mLakehouse Architecture with Delta Lake and DatabricksDragan Beric · DSC Europe 2022
• 44mWhat's the Best Big Data Architecture for You?Christoph Windheuser · GOTO Amsterdam 2024
• 24mData Mesh and LakehouseMatei Zaharia · State of Data Mesh 2022
• 31mDelta Lake: Open Source Reliability and Quality for Data LakesMichael Paul Armbrust · Scale By The Bay 2019
• 44mUnify your data with OneLake and Microsoft FabricAbhishek Narain, Joshua Caplan, Harmeet Gill, Salil Kanade, Swetha Mannepalli, Aaron Merrill, Miwa Monji, Trevor Olson, Joanna Podgoetsky, Mark Pryce-Maher, Charles Webb · Microsoft Build 2024
• 39mData Observability and Reliability Engineering in a Real-Time WorldNilanjan Chatterjee · Data Summit 2025
• 31mUsing Spark to accelerate your lakehouse architecture with FabricJustyna Lucznik · Microsoft Build 2023