The Art of Data Engineering
summary
Data engineering separates data-prep pipelines from modeling and must run at production scale with automated checks. The talk describes pipelines built with Airflow (Docker executor), containerized training on Amazon Batch/EMR, streaming with Kafka and Spark Streaming, and production practices like automated quality checks, zero-downtime table swaps, and a Kubernetes executor deployed successfully. It provides concrete scale examples (12 million users, billions of events) and industry numbers for context.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:51Internet as imperfect system
- 3:16Data engineer's role
- 8:51Big data in practice
- 13:07Airflow pipelines
- 16:08ML production problems
- 19:49ML container services
- 21:39Spark apps
- 23:12Model integration in business
- 25:05Kubernetes in Airflow
- 27:21Best practices
- 30:09Data quality checks
- 33:33Learning resources
- 35:46Summary of the profession
- 37:35Q&A
related talks
41mData engineering in 10 years
30mDevOps For Data Teams
41mData engineering without walls
26mDataOps: Accelerating Digital Transformation
32mOrchestrating Your Data Pipelines with Apache Airflow
19mDemystifying Modern Data Pipeline Architecture
25mWhy Software Engineering Best Practices Fail in Data Engineering
53mThe State of Data Engineering