Challenges that everyone struggles while productionizing Apache Spark workloads
summary
Migrating a 25,000‑line SQL ETL to Spark produces concrete wins: parallel JDBC with partition bounds boosts read concurrency, using lower/upper bounds and partition counts instead of a single connection; bulk load API for MS SQL Server writes in batches and avoids row‑by‑row JDBC. Dynamic join selection (measure size, choose BroadcastHashJoin or SortMergeJoin) cuts shuffles, coalesce/repartition tuning limits partitions, workarounds for inconsistent dropDuplicates avoid repartition pain, and replacing Scala Future with Monix IO yields stable, production behavior.
AI-written summary · about
chapters (20)
- 0:00Start
- 2:16What is Apache Spark?
- 3:07Distributed Data Abstraction
- 3:31Resilient & Immutable
- 4:07Compile-time Type Safe / Strongly type inference
- 4:26Lazy evaluation
- 5:03Apache Spark Operations
- 5:23Essential Spark Operations
- 12:20DataFrame API Code
- 14:05Catalyst in Spark
- 15:02Example: DataFrame Optimization
- 18:01Spark Internals terminology
- 19:19Spark Internals: Jobs
- 19:50Spark Internals: Tasks
- 20:20Spark on Yarn Internals terminology
- 22:27Resource Manager (Yarn) Tuning
- 23:22Spark Scheduler FIFO to FAIR
- 24:27Case Story and Discovery
- 33:47Spark Parallel JDBC Read
- 35:52Bulk Load API vs JDBC write
related talks
41mOptimising Apache Spark and SQL for Improved Performance
40mScaling with Apache Spark (or a lesson in unintended consequences)
40mBest Practices and Optimization with the Spark Engine
04mKuassi Mensah Interview
43mBeyond Shuffling: Scaling Apache Spark
28mAn Apache Spark query's journey through the stack
36mData pipelines engineering made simple with Scala
15mOptimizing Query Performance in Big Data Systems