Unified dataframe API for different data backend
summary
A unified dataframe API called Ibis compiles Python dataframe code into relational expressions that target Snowflake, DuckDB, Spark, RAPIDS/GPUs and others, aiming for native performance and cross-backend consistency. It supports batch and streaming, user-defined functions, and lazy schemas so it avoids pulling full tables locally. Benchmarks claim orders-of-magnitude cost and speed wins when shifting certain ETL from CPU Spark to GPU engines (examples at 10–30 TB and a 100 TB GPU run), and teams use Ibis to benchmark and port workloads.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:46Alex Net
- 2:02How has AI changed data landscape
- 7:11Divergence
- 9:47Data Systems
- 12:17Challenges
- 13:53Composable Data Systems
- 14:37IBIS
- 15:44Requirements
- 16:43extensible
- 17:12neighhood
- 17:45magic underscore
- 18:17SQL
- 20:24consistency
- 20:46pandas
- 22:39standardization
- 23:26benefits
- 24:34example
- 24:54aacho
related talks
25mIncrease productivity and simplify data operations in Python
41mAccelerating Data Science with RAPIDS
30m"Better dataframes"
39mAccelerating Data Science with RAPIDS
31mcuDF: RAPIDS GPU-Accelerated Dataframe Library
28mAnalyze data easily with duckdb - and the implications on data architectures
39mStreaming doesn’t have to be hard
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas