Unifying Analytics
summary
Claim: unify data lake and warehouse so BI and ML run on the same platform and avoid duplicate ETL. Implementation examples: Uber and The Trade Desk move raw data into S3/parquet then spin analytic clusters, The Trade Desk runs 15 petabytes on a 320-node cluster and spins up two more for BI (5,000 reports/day). Tech used: Vertica with Vertica Python, Spark, parquet, S3, distributed R, PMML/TensorFlow import. Results: isolated workloads, ML training and SQL analytics co-locate and go into production.
AI-written summary · about
chapters (13)
- 0:00Introduction to unified analytics
- 1:18Everyday predictive analytics
- 2:45Traditional data warehouse architecture
- 9:59Limitations of data warehouses
- 11:03Rise of the data lake
- 16:51Weaknesses of data lakes
- 17:35Evolution of data warehouses and lakes
- 21:54Complexity of combined systems
- 24:02Uber’s unified analytics approach
- 26:30Unified analytics warehouse concept
- 29:28Capabilities of unified analytics
- 32:38Future of unified analytics
- 33:20Resources and closing
related talks
34mData Lakehouse: Who, Why and How
17mDemocratising data analytics
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
44mWhat's the Best Big Data Architecture for You?
52mElevate your data and AI governance with Databricks Data Intelligence Platform
29mData foundation for analytics and AI
43mData warehousing reinvented for today's needs
26mModernize your Enterprise Data Warehouse & generate value from data