DataEngBytes 2023 - BRI-T1-03 - Yousry Mohamed
summary
Explains concrete Spark fixes: don't collect to the driver because pulling a billion-row DataFrame crashes memory; cache or persist intermediate results to avoid repeated lazy re-execution (examples drop runtime from ~27s to ~3s); beware task retries causing duplicate DB appends and use idempotent writers or vendor connectors; prefer JDBC partitioning increases parallel reads from 1 to 8 tasks over a single connection; and avoid Python UDFs — Python UDFs are very slow; prefer built-in SQL functions or pandas UDFs and use broadcast joins when one table is small.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:45Spark anatomy and pipelines
- 8:31Task retry and input data
- 14:58Query patterns and pitfalls
- 25:27Advanced Spark optimizations
- 27:13Q&A
related talks
18mDataEngBytes 2023 - MEL-T3-02 - Alexei de Lauw
28mSoftware Engineering Skills for Data Engineers
19mDataEngBytes 2023 - MEL-T1-04 - Michael Hyatt
36mNo more struggles with Apache Spark (PySpark) workloads in production
04mDBTB INT Sandy Ryza r
30mImproving PySpark Performance: Spark performance beyond the JVM
24mImproving interactive querying experience on Spark SQL
21mDataEngBytes 2023 - MEL-T1-05 - Fabio Ramos