Big Data Architectures and Ten Typical Pitfalls Along the Way
summary
Covers how to build production-ready data architectures using Big Data technologies such as NoSQL, Spark, Hadoop and Kafka, and how to integrate them with existing Oracle databases and data warehouses. Describes possible integration scenarios between data warehouses and Hadoop, presents streaming architectures, and lists ten common architectural pitfalls in Big Data applications.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:08Data warehouse architecture
- 3:16Data lake basics
- 5:02Data sources and ETL
- 7:34Hadoop and scaling
- 9:21HDFS limits and ETL
- 11:27Data lake patterns
- 14:20Streaming and real-time
- 16:05Lambda and Kappa
- 22:28Spark clusters
- 26:03NoSQL modeling
- 27:49Open source vs commercial
- 29:37Big data analytics
- 31:00Developer tools
- 33:08Team roles
- 35:36Organizational pitfalls
- 38:03Automation
- 39:26Storage-compute coupling
- 41:56Data locality
- 44:51Storage trade-offs
- 45:36Hardware use cases
- 47:00Cloud vs hardware
- 47:21Tech diversity
- 48:24Summary
related talks
33mDesigning Architectures: Big Data + Real Time
34mAvoiding Big Data Antipatterns
28mYour data lakes need architecture too!
05mThe Future of Data Warehousing
34mTales of Data Architecture Evolution
29mBig Data Analysis
56mResistance is Futile: What You Need to Know about Big Data
31mData Warehouses Meet Data Lakes