Tutorial - Intro to Data Engineering
summary
Hands-on intro that actually builds things: attendees import a VM image datacon la2020 into VirtualBox, use Python + Spark labs to run pip installs (requests), exercise RDD and DataFrame code, run spark-submit locally, and load a parquet file. Labs 1–6 include printing “hello world”, counting ~58,118 lines vs 58,109 unique words (one surprising duplicate), filtering for “aardvark”, and inspecting partitions and repartition/coalesce behavior. Covers batch vs streaming, SLAs, file formats and serialization. Practical troubleshooting (pip/pyenv issues) and working pipelines are demonstrated end-to-end.
AI-written summary · about
chapters (7)
- 0:00Introduction and setup
- 1:01:23Python basics overview
- 1:52:51Lab 1 walkthrough
- 2:00:31Data engineering fundamentals
- 2:34:28Spark fundamentals
- 3:00:14Building Spark pipelines
- 3:07:24Q&A and closing
related talks
01h 22mIntroduction to Big Data Processing using Spark and Python
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
04h 02mHands-on Session on Data Science for Beginners
40mEveryone can Data! - A Practical Data Science Intro
25m"Data Engineering: The World Between Worlds"
50mIntroduction to Real-Time Big Data with Apache Spark
28mSoftware Engineering Skills for Data Engineers
43mPySpark in Practice