PySpark in Practice
summary
Shows concrete PySpark choices and outcomes: configuration recipes for a 6‑node 64 GB cluster with 1.5× executor overcommit and per‑executor 512 MB Python process defaults, how that yields 24 concurrent Python procs and why Python memory sits outside the JVM; benchmarks showing DataFrames compile to JVM bytecode and beat Python RDDs; unit testing with mocked data and Luigi pipelines to avoid shipping code with spark-submit; streaming online learners (streaming LR and k‑means) working at sub‑second limits; API‑first deployment with Flask and Cloud Foundry.
AI-written summary · about
chapters (9)
- 0:00PySpark introduction
- 1:04PySpark and JVM/Python
- 2:51RDDs and DataFrames
- 6:51PySpark performance config
- 18:33Unit testing pipelines
- 22:06Luigi for pipelines
- 26:50Streaming online learning
- 29:22Operationalizing models
- 31:29Q&A
related talks
44mProcessing large data with Apache Spark
24mAn Introduction to PySpark
40mProblem Solving Recipes Learned from Supporting Spark
37mPixieDust make Jupyter Notebooks with Apache Spark Faster, Flexible, and Easier to use
59mSpark 2.0
01h 18mScalable Data Science with Spark and R
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers
36mScaling data workloads using the best of both worlds: pandas and Spark