Modern Data Architecture

on YouTube ↗45m

" summary

Argues that enterprises move from slow nightly ETL data warehouses to a mix of a data lake on Hadoop, a lambda-style batch+speed path and a streaming pipelined replication layer, using CDC change data capture + queues so 4 TB/day sources replicate to Hadoop and an MPP like Greenplum, cutting time‑to‑insight from days (7 days for DR, backups taking days on 10–15 TB) to near real time. Describes concrete stack options: Spring Cloud Data Flow, Kafka/RabbitMQ, Spark, GemFire; pipelining with replayable queues solves DR and ordering limits CDC coverage.

AI-written summary · about

" chapters (9)

1. 0:00Introduction and speaker background
2. 2:35Evolution of data processing systems
3. 8:53Traditional data warehouse challenges
4. 13:29Moving from ETL to ELT and CDC
5. 17:49Introducing data lake and lambda architecture
6. 22:29How lambda architecture works
7. 27:33Pipelining for real-time replication
8. 33:18Technologies for modern data architecture
9. 35:51Summary and Q&A start

" related talks

• 34mTales of Data Architecture EvolutionJosef Goldstein · Build Stuff 2022
• 41mModern Data Architectures – What They Are and How to Implement?Matthias Niehoff · Software Architecture Gathering – Digital 2023
• 25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant AnalyticsMark Lyons, Jonny Dixon · Big Data LDN 2022
• 01h 03mReinvente su estrategia de datos con Lagos de Datos TransaccionalesJavier Quiroz, Santiago Abarca · AWS Summit Mexico City 2022
• 10mData Rivers — The New Analytics ArchitectureRachel Pedreschi · OSA Con 2021
• 53mBig Data: from mammoth to elephantRoman Nikitchenko · JavaDay Lviv 2015
• 49mData Platform Architecture, Evolution, and PhilosophyJustin Becker, Kurt Brown · @Scale 2014
• 29mData Mesh Streaming Patterns: A Real-time PerspectiveAlok Pareek · Big Data LDN 2022