Building Python Binding for the Delta Lake Library in Rust
Talkat Open Source Summit Europe 2023 · on Wednesday, 2023-09-20· at 14:40· Room 0D-2-0D-3 (Floor 0), Abandoibarra Etorb., 4, 48011 Bilbao, Bizkaia, Spain·
summary
Delta-rs is a native Rust reimplementation of the Delta Lake protocol that reads Parquet, converts to zero-copy Arrow columns, runs on Tokio for async, and plugs different clouds via an object-store unified API. Python bindings use pyo3 and maturin to produce wheels published to PyPI (multi-OS CI recommended); adoption jumps after release and the project has 100+ contributors. Reported performance example: 1B rows: pandas 3 min → delta-rs 2 s. Works in small runtimes like AWS Lambda.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:54Why Rust for bindings
- 7:01Rust vs C/C++
- 9:51Delta RS architecture
- 12:45Python bindings with Maturin
- 17:20Project workflow
- 19:50Python vs Rust performance
- 21:18Memory management
- 23:24Community insights
- 24:27Q&A
related talks
26mBuilding Python Binding for the Delta Lake Library in Rust
29mBuild simple & scalable data pipelines with Polars & DeltaLake
25mIntegrating Rust with Python...
34mRust for the Python dev
29mAWS Lambda Officially Rust: From Scratch to Progressive Bindings
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
30mPython and the AI Lakehouse
39mPython and SQL: Better Together, Powered by DuckDB