Async Rust Meets Async Python: How Daft runs user defined Python coroutines from Tokio
summary
Daft runs user async Python coroutines from a Tokio scheduler by keeping a real asyncio loop on the Python main thread, exposing coroutines via PyO3 + Tokio, converting each coroutine to a Rust future that signals completion over a oneshot channel, and recycling results with a shared broadcast to reduce churn. The system splits work into separate IO and compute runtimes, scales to billions of events in practice, and interoperates without blocking Tokio threads.
AI-written summary · about
chapters (11)
- 0:00Daft introduction
- 1:18When to use Daft
- 2:41Video frame captioning example
- 4:27Query plans in Rust
- 5:51Tokio for IO and compute
- 7:58Tokio simplifies async
- 10:26Python coroutines from Rust
- 13:39Dual runtime for Python UDFs
- 16:10Scaling Python async UDFs
- 18:43Async Rust meets Python
- 19:46Summary
related talks
29mAsychronous programming in Rust
01h 29mStandardised Coroutine Concurrency in Python 3
18mA better way of doing Async Rust
44mDo Pythons Rust? How we used PyO3 to build a Python Stream Processor w/ a Rust Heart
45mAsynchrony and Coroutines
57mFear and Awaiting in Async: A Savage Journey to the Heart of the Coroutine Dream
29mI/O in Rust: the whole story
35mHow Do Python Coroutines Work?