Data Refinement: Unleashing the Power of SQL in Python using Polars
summary
Polars runs on Rust with Apache Arrow memory, exposes SQLContext and an easy Python API, and uses lazy frames to query very large files (example: a Pokemon CSV ~200 million rows) without loading them into memory. The demo registers tables, runs SQL that streams output to result.csv, and uses a CTE to join trainers with Pokemon, LIMIT 6, returning one champion (Alex Bell). Eager vs lazy control and streaming collect actually write results and avoid memory spikes.
AI-written summary · about
chapters (8)
- 0:00intro
- 0:20preamble
- 0:32agenda
- 0:54why polars?
- 3:09introduction to sql content
- 5:05utilize lazyframes for big datasets
- 10:16common table expressions (ctes)
- 13:47thank you!
related talks
30mIs the great dataframe showdown finally over? Enter: polars
32mPandas, Polars and the DataFrame Consortium.
58mFrom Pandas to Polars Upgrading Your Data Workflow
31mPolars, the End of Spark?
58mData Wrangling [for Python or R] Like a Boss With DuckDB
34mHow you can write a dataframe-agnostic library
21mPosit and the modern data stack: Using Posit, Snowflake, and Databricks
23mQuery like a Pro: ES|QL meets Apache Arrow and Pandas