Quokka-Rewriting SparkSQL in Python
summary
Quokka is a distributed Python SQL engine that executes a DataFrame/SQL flavor API and recovers from worker loss so queries survive spot instance interruptions. It composes existing projects — built on Ray Apache Arrow and polars with Redis for coordination — shards plans to Ray actors, shuffles via Arrow, and runs Polars locally. TPC-H on Parquet from AWS S3 shows Quokka a bit slower than Fino/Trino but noticeably faster than spark SQL, and the codebase is 1–2k lines of code.
AI-written summary · about
related talks
30mKoalas
31mBringing the Jewels of the Python World to Scala with Spark
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
36mKafka Streaming, the Pythonic Way
24mPolyglot data with python: Introducing Pandas and Apache Arrow
23mQuery like a Pro: ES|QL meets Apache Arrow and Pandas
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
30mIs the great dataframe showdown finally over? Enter: polars