From On-Ground to On-Cloud Data Engineering
summary
Covers how cloud data engineering turns messy sources into usable outputs with concrete tradeoffs and patterns including scale, freshness, correctness, completeness as design levers and examples that change those choices: city traffic, medical imaging datasets, weekly leadership dashboards, and live stock trading. Explains ingestion options and costs with batch vs real-time distinctions, storage choices (buckets, databases, queues), transformation tactics, DAG orchestration, and monitoring metrics like time to detect fix and recover.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:05Speaker background
- 3:54Data engineering role
- 7:27Data engineering vs science
- 9:56Expectations from engineers
- 21:55Data engineering pillars
- 31:30Raw data challenges
- 39:12Data ingestion strategies
- 44:12Data transformation modeling
- 51:13Storage systems design
- 1:01:32Data quality monitoring
- 1:07:31Starting data engineering career
- 1:11:24Q&A
related talks
40mData Engineering in a Rapidly Changing World
25m"Data Engineering: The World Between Worlds"
33mCloud & data - The Engine & Fuel of Business
24mGotta Feed the Beast - Data Engineering at Scale
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
41mModern Data Architectures – What They Are and How to Implement?
34mTales of Data Architecture Evolution
51mPanel Debate: Rise Of The Data Engineer