Taming the Data Tsunami
summary
Claims a concrete open‑source playbook that shifts filtering and governance upstream and policy-as-code for distributed pipelines that reduce dataset size by 50–70% and cut model onboarding time in half. Shows an implemented stack (near‑compute agents, local Iceberg/Parquet tables, Arrow/DuckDB/SQLite, Open Policy Agent, Benthos/vector) that ships answers not firehoses, enforces PII and sovereignty, and reports multi‑order cost savings in real deployments.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:05Ingest-it-all tax costs
- 3:53Ingest-it-all breaks ML pipelines
- 7:24Modern data challenges
- 9:30Bottleneck in clean data
- 10:32Three data strategy truths
- 11:56Playbook: Data is everywhere
- 18:36Playbook: Speed of light constant
- 25:38Playbook: Regulations and gravity
- 32:19Ingest-it-all red flag
- 34:47Adoption strategies
- 35:08Q&A
related talks
27mData team strategies: unleashing your team
47mDataOps for AI-Ready Platforms
31mBuilding a Data Infrastructure for AI/ML
35mHigh-performance data foundation for AI success
14mTony Baer, dbinsight & Doug Henschen, Constellation Research | Databricks Data + AI Summit 2023
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)
19mA world of data
18mUnleash your Private Datasets to Power up AI