Parallelising pandas with
summary
Koalas implements the pandas API on Apache Spark, so one line change import databricks.koalas as ks gives pandas-like code but runs distributed. Project claims ~75% pandas coverage, >30,000 daily downloads and ~2,000 GitHub stars, targets Spark 3.0 and ships on Databricks 7.0. Architecture uses an immutable internal frame and stores an index as an extra column; beware the default distributed sequence vs sequence index OOM tradeoff. Convert with .to_spark / .to_pandas and use pandas UDFs for Python apply.
AI-written summary · about
chapters (9)
- 0:00Parallelising pandas with Spark
- 1:53Pandas versus Spark origins
- 6:54Introducing Koalas
- 9:05Koalas API coverage
- 10:50Koalas vs pandas differences
- 13:38Koalas architecture
- 23:20Demo with Koalas
- 41:10Koalas roadmap
- 42:12Q&A
related talks
36mScaling data workloads using the best of both worlds: pandas and Spark
24mPolyglot data with python: Introducing Pandas and Apache Arrow
30m"Better dataframes"
24mAn Introduction to PySpark
53mEnterprise-Ready Python for High-Performance Data Teams
26mLarge Scale Data Validation (with Spark and Dask)
30mSpeed is Not All You Need for Data Processing
45mUsing Kotlin for Data Science