DataFrames in Apache Spark - the analysts perspective
summary
Practical debugging of Spark DataFrame failures with concrete fixes and numbers: shows partitionBy can produce thousands of tiny HDFS files when 8,000 input blocks and 90 days are written, while repartition writes shuffled buckets to local disk then commits one file per target partition to avoid hammering the NameNode. Explains shuffle cost, default spark.sql.shuffle.partitions = 200 so 10 TB / 200 = ~50 GB per task, boosting to 2,000 tasks lowers per task size, and using salted joins that duplicate small tables across salt values mitigates skewed keys.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 1:51Data analysts perspective
- 5:42Spark execution model
- 11:00Partitioning pitfalls
- 16:39Repartitioning explained
- 20:11Join operations deep dive
- 24:49Handling skewed joins
- 29:10Key takeaways summary
- 31:15Q&A session
related talks
40mProductionizing Big Data - Stories from the Trenches
29mBig Data Analysis
55mSpark Architecture
52mSolving Analytical Problems using Apache Spark
16mPetabytes of data: How to use R at scale
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
24mTaming Data Skew in Production ML Pipelines
36mNo more struggles with Apache Spark (PySpark) workloads in production