Gotta Feed the Beast - Data Engineering at Scale
summary
Data engineering means building and operating the pipes that feed humans and AI agents with usable data at scale not theory but concrete tradeoffs and costs. The workflow is ETL/ELT with monitoring, security, governance and lineage; tool examples include SQL, Python, Snowflake plus SQL Server, Informatica, Qlik, Azure, Git and bespoke pub/sub. Cadence choices matter — real-time vs 5–10 minute cadences — and small hourly loads of a few thousand records can still overwhelm a target; pipelines run indefinitely and incur real token and bandwidth costs.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:09Data engineer role
- 3:17Data plumbing analogy
- 5:01Data role differences
- 7:08ETL process core
- 8:52Pipeline robustness challenges
- 9:55Monitoring pipelines
- 11:41Pipeline design for consumers
- 13:06Data cadence and volume
- 14:52Storage duration costs
- 16:17Pipeline cost impact
- 17:21AI complicates engineering
- 19:47Data engineering toolkits
- 20:50AI building pipelines
- 21:34Data governance decisions
- 23:19Conclusion and Q&A
related talks
40mData Engineering in a Rapidly Changing World
25m"Data Engineering: The World Between Worlds"
30mData Engineering building your BI infrastructure from scratch
38mKeeping pace with data infrastructure so it doesn't outrun you
51mPanel Debate: Rise Of The Data Engineer
34mTales of Data Architecture Evolution
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
22mDataEngBytes 2023 - BRI-T1-06 - Yanir Seroussi