Behind the Hype - Architecture Trends in Data
summary
Argues that treating datasets as data products—publishing SLAs, refresh guarantees and measurable business KPIs—actually drives ownership and value; distinguishes casual prompt tweaking from engineering by namechecking LangChain and Azure Prompt Flow for templated, testable prompts; describes Databricks/Delta adding vector indexes and vector databases + RAG to stop LLM hallucinations; flags medallion+Data Vault for exploding joins and poor analytic performance, and warns platform engineering bottlenecks (reported 3–5 month infra lead times) versus reusable infra as code with Terraform.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:10Hype and early adoption
- 4:39Gartner hype cycle for data
- 9:392022 hype curve trends
- 12:54Centralization vs decentralization
- 18:27Today’s data topics
- 19:49Data products and business
- 27:27Prompt engineering
- 35:00Data intelligence platforms
- 40:17Vector databases in AI
- 47:36Data storytelling and stacks
- 50:45Platform engineering pitfalls
- 52:19Data Vault and medallion pitfalls
- 53:45Q&A
related talks
50mBehind the Hype - Architecture Trends in Data
41mOn Data: Sky above, sand below, peace within
38mKeeping pace with data infrastructure so it doesn't outrun you
44mWhat's the Best Big Data Architecture for You?
41mModern Data Architectures – What They Are and How to Implement?
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
55mThe Great Data Debate 2022
38mThe Rise & Downfall of the Data Engineer REVISITED