PySpark in Practice
summary
Clear, actionable takeaways: configuration recipes (enable compression and speculative execution, reduce Python concurrency to avoid OOMs, use Anaconda on nodes), concrete Yarn sizing example with multiple executors, spark 1.6 dynamic storage/execution memory and partition on read to avoid shuffles to stop groupBy blowups, prefer DataFrames and native functions over RDDs for Java bytecode speed, unit tests run on a local single-node Spark with mocked data under a TDD workflow, use Luigi + Spark tasks for pipelines, Flask + pickled models and Cloud Foundry deployment, streaming supports streaming linear regression and streaming k‑means.
AI-written summary · about
chapters (9)
- 0:00PySpark introduction
- 1:08What PySpark is
- 2:55PySpark data structures
- 6:13Configuring PySpark
- 16:14Unit testing PySpark
- 22:34Managing pipelines with Luigi
- 26:34Online learning with PySpark Streaming
- 29:56Operationalizing models
- 33:14Q&A
related talks
41mImproving PySpark Performance Spark performance beyond the JVM
01h 22mIntroduction to Big Data Processing using Spark and Python
43mImproving PySpark Performance: Spark performance beyond the JVM
24mPySpark - Data processing in Python on top of Apache Spark.
41mKeynote: Making the Big Data ecosystem work together with Python
50mBig Data Processing and Data Science Environment in Python
26mDebugging PySpark -- Or trying to make sense of a JVM stack trace when you were minding your own bus
31mBringing the Jewels of the Python World to Scala with Spark