Sparking Success: Unveiling the Journey of Apache Spark Application Development
summary
Builds a Spark pipeline that reads par files from the mov lens export, extracts year and explodes pipe‑delimited genres, groups and counts movies per year, and finds drama as the overall most common genre through 2018 with comedy leading early decades. Uses Apache Spark on EMR, prototypes in IDE, converts CSV to parquet, runs spark‑submit, recommends ADRs, profiles data and adds assertions with Great Expectations, and notes heavy backfill costs (example: a week for two years).
AI-written summary · about
chapters (16)
- 0:00Welcome
- 1:05Speaker intro
- 4:16Data engineering lifecycle
- 6:25Lifecycle wheel
- 7:08Understanding task
- 10:23Data acquisition
- 12:59Architecture design
- 14:47Architectural decision records explained
- 15:53Data quality
- 18:19Development
- 31:59Pipeline testing
- 34:27Deployment
- 38:19Backfilling
- 40:48Monitoring pipelines
- 43:19Iteration
- 44:23Key takeaways
related talks
36mData pipelines engineering made simple with Scala
44mProcessing large data with Apache Spark
44mUsing Python with Hadoop to create production ready big data applications
39m10 things I learned about writing data pipelines in Python and Spark.
43mPySpark in Practice
42mScala: The Unpredicted Lingua Franca for Data Science
39mHow Python Drives Modern Data Engineering
18mScalable, Code-Free ETL