PySpark and Warcraft Data
summary
Uses PySpark with distributed DataFrame to process a 22Gb json blob of World of Warcraft auction snapshots, provisioning short‑lived EC2 clusters and comparing runtimes to numpy/pandas/scikit with in‑memory Spark claims up to ~100× speedup. Demonstrates partitioning/caching, converts results to pandas for plotting, and finds that the top one percent owns about 25% of auction listings; linear regressions fail to show expected negative price–quantity elasticity. Cluster setup, demo queries and cost estimates (~$15/day) all work.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:51Overview
- 1:49Warcraft
- 7:08Big Data
- 8:03Many Small Bits
- 13:59Getting Started
- 17:43Simple Queries
- 21:44Cost
- 23:45Results
- 30:33Conclusion
related talks
05mBig Data, No Big Deal: PySpark
30mSpeed Up Your Spark Data Processing with a GPU
36mScaling data workloads using the best of both worlds: pandas and Spark
30mImproving PySpark Performance: Spark performance beyond the JVM
45mIndroduction to the PySpark DataFrame API
25mDistributed data workflows: PySpark vs Dask
27mHow to spend ¾ of your yearly budget in 3 weeks: a PySpark cautionary tale
50mBig Data Processing and Data Science Environment in Python