The MODERN Modern Data Stack: Building an Open Distributed Data Warehouse Beyond...
summary
Argues that bringing compute to data cuts egress, latency and compliance risk and shows a working stack: deploys Bacalhau agents on VMs across AWS GCP and Azure, runs Python parsing jobs that push schematized rows into BigQuery, and converts noisy logs into query-ready Parquet/Iceberg. Demo moves 838,000 rows from one node and 27 million rows uploaded when run across nodes. Uses Docker/WASM-compatible executors, supports low‑resource edge devices, sanitizes PII by stripping the IP last octet (sanitize IP addresses), and runs early ETL steps at the edge while keeping heavy processing central.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:04Centralized data storage problems
- 4:13Traditional pipelines failing
- 8:27Modern data stack intro
- 10:55Distributed warehouse limits
- 13:05Raw data challenges
- 16:55Near compute processing
- 18:18Live demo setup
- 21:10Multi-cloud job execution
- 22:33Schematized data processing
- 25:24Data compliance sanitization
- 27:48Efficient data aggregation
- 28:53Real-world use cases
- 30:58Benefits and open source
- 32:26Q&A
related talks
42mBuilding Tomorrow's Warehouse Today: Modern Data Warehouse Modelling Insights
41mModern Data Stack: Optimising and Scaling Data in a Tech Company
11mBreaking Data Silos: Enterprise Data Management
36mMigrating Our Data Warehouse to the Cloud
37mBuild a Modern Data Platform on AWS
41mModern Data Architectures – What They Are and How to Implement?
19mDemystifying Modern Data Pipeline Architecture
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale