AI Should Not Replace Well-Built Data Fundamentals
summary
Claim: AI speeds construction but does not replace a robust foundation; reliable systems require explicit storage, movement, and transformation design. Concrete tools and results: Apache Beam (Dataflow) for streaming and decades‑of‑satellite‑imagery batch retraining, Apache Kafka and NiFi for low‑latency movement, Apache Iceberg with BigQuery managed tables for interoperable storage, Apache Spark and Spark ML for feature engineering, plus Gemini code assist and BigQuery recommendations to accelerate work. Real builds include fraud‑blocking pipelines and retrained weather models; humans remain in the loop for policy and storage rolloffs.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:53Data fundamentals defined
- 6:03AI with Kafka and Beam
- 9:35Tooling lock-in risks
- 12:41Transformation tools for AI
- 15:12AI's role in data fundamentals
- 16:59Summary
related talks
31mBuilding a Data Infrastructure for AI/ML
21mOvercoming Data Barriers for Successful AI Projects
38mPanel Discussion: AI Needs a Foundation That Never Fails
20mAI
36mDatabricks as the Foundation for GenAI: What’s New and What Works
11mBreaking Data Silos: Enterprise Data Management
50mDesigning a scalable data platform based on Apache Spark/Iceberg
43mKey lessons from building Data and AI systems over the last 26 years…