Koalas
summary
Koalas is an open source Python library that implements the pandas API on Spark and translates pandas calls into lazy Spark operations, using an immutable internal frame plus a public API that mimics in-place pandas semantics. It supports plotting, Parquet IO and SQL integration, reports about 11,000 daily downloads and ~20 contributors, and currently covers roughly 70% of the DataFrame/Series API. A customer migrates with <1% code changes and shows large speedups on UDFs and group operations.
AI-written summary · about
chapters (9)
- 0:00Introduction to Koalas
- 1:06Pandas vs Spark differences
- 5:25Introducing Koalas project
- 7:36How Koalas works under the hood
- 11:58Koalas demo with Jupyter
- 18:05Koalas API and features
- 20:43Real-world Koalas performance
- 22:08Technical details of Koalas
- 27:31Future of Koalas and Q&A
related talks
34mPandas vs Koalas: The Ultimate Showdown!
47mThe future of data: Apache Spark 3.1, Delta Lake and Koalas
08mQuokka-Rewriting SparkSQL in Python
25mScale Data Science by Pandas API on Spark
31mBringing the Jewels of the Python World to Scala with Spark
37mpandas at a Crossroads, the Past, Present, and Future
41mKeynote: Making the Big Data ecosystem work together with Python
01h 13mFrom Pandas to PySpark